Solved

Custom Operands with Python

  • 27 September 2023
  • 3 replies
  • 186 views

Hello. I have been learning how to use ZOS-API in Standalone mode with Python for a while now, but it seems that custom operands for optimization are only available using C# or C++. I have started working with C# (despite not knowing the language) and I have managed to get custom operands to work using this. Currently, I am working on connecting C# to Python so I can run my previously written scripts and then optimize the system based on some values calculated in Python. My current pipeline looks something like

  1. UDOC operand runs Python script
  2. Python script runs an instance of Zemax in Standalone mode and calculates some value
  3. Value is sent back to UDOC operand
  4. Zemax retrieves value

This setup somehow manages to work with just getting the value from Python into Zemax in the Merit Function Editor, but actually using the UDOC operand for optimization doesn’t seem to work and is also incredibly slow. A single cycle in the optimization loop can take over two minutes while retrieving the UDOC operand value in the Merit Function Editor only takes a couple of seconds, so I assume the main problem with this setup has something to do with the interaction between Zemax and Standalone Mode being looped due to the nature of optimization. My main question regarding this topic is whether or not there are other methods to creating custom operands using Python. Based on what I have done so far, I currently think that it is probably best to just convert most or even all of my current Python code into C#, which will be a daunting task as my code is quite complex. I have also considered making my own optimization algorithm in Python which would completely circumvent this issue. I want to hear some opinions regarding this matter before doing anything.

icon

Best answer by MichaelH 27 September 2023, 20:09

View original

3 replies

Userlevel 3

Hi Cameron,

Kudos for implementing that in C# 😀

In principle there should be no difference in the parts of Zemax that you can set via python or C# (they use the same  API). In ZOSPy we have included some basic functionality with the Merit Functions, but probably not something as sophisticated as you require.  This is something that we aim to expand in the future (also with some examples), but as it is currently not critical to our scientific work, I don’t have a timeline on that.

On the other hand, if you want/can share some of your efforts, we are happy to include it in a future release of ZOSPy. In that case, it is probably easier to start an discussion or pull-request on the ZOSPy’s GitHub page, so we can jointly work on it.

@Luc.van Vught or @chaasjes : do we have any of our merit function examples/code that we can easily share?

Userlevel 6
Badge +2

Hey Cameron,

There are a few things to consider with your workflow:

  • For the UDOC, make sure you have the EXE compiled as a Windows Application and not a Console Application.  Each time the MF updates, the UDOC is executed and GUI I/O such as displaying the command window is expensive.
  • Launching a Standalone Application via Python is probably taking the most time for each cycle.  The typical connection between a Standalone Application and the ZOS-API is 10’s of seconds.  
  • Depending on how you’re communicating between C# & Python, you could run into a handful of issues:
    • If you’re calling python via exec(), then Python is launched on a new thread and there is no communication between your C# UDOC and Python.  You’ll have to “guess” at when Python is done, most likely by querying an output file from Python (this is an expensive operation)
    • If you’re using IronPython, then you’re limited in the set of Python modules you can use within your script.  No modules with compiled code (such as numpy or matplotlib) can be used with PythonNet (I suggest MathNET for numeric calculations and ScottPlot for graphing, both available via nuget).  The advantage of IronPython is you don’t need Python installed on the target machine.
    • If you’re using PythonNET and with using (Py.GIL()), you lose all multi-threading capability.  Although the MF sequentially updates each operand individually, a single operand can be evaluated with multiple threads.

So, although it is a “cool experiment” to get Python running from C#, you will never see the performance you’re expecting with a UDOC.  I would suggest converting all your code to C#.  Another benefit of the UDOC is if it’s compiled correctly, you only need 1 EXE file and this 1 file is stored in any ZAR if the UDOC is loaded in the MF.  This makes a pure C# UDOC extremely transferable between users.

Also, this is a great time to remind Zemax Engineers and Developers that UDOCs are still called an a executable and not on the same process as OpticStudio itself.  This makes UDOCs much slower than built-in operands.  Zemax should implement in-process UDOCs to really help expand the ZOS-API’s capabilities. 

Userlevel 3

In principle there should be no difference in the parts of Zemax that you can set via python or C# (they use the same  API). In ZOSPy we have included some basic functionality with the Merit Functions, but probably not something as sophisticated as you require.  This is something that we aim to expand in the future (also with some examples), but as it is currently not critical to our scientific work, I don’t have a timeline on that.

Some correction after I had some further discussion with Luc and Corne.

In line with the post of @MichaelH , this is a bit more complex than I originally anticipated. The merit function operands can be set via python/ZOSPy, but for the UDOC you indeed need a compiled executable, so for that part python is less optimal. 

Reply