I don’t think its a Windows 11 issue. I use it on my laptop without any problems.
As always, check for updates both for Windows and your graphic card driver.
Hello!
Once other suggestion I can offer - The windows may actually be opening, but they are attempting to open on a secondary monitor that isn’t working at the moment, so they are trying to open outside the main screen.
I would unplug any extra HDMI / Displayport or VGA cables that are connected to your machine so you only have 1 monitor attached, or if a laptop, only use the built in display on the laptop. Then try Opticstudio again. If all the popup windows work that way, you should be able to plug any extra displays back in and be back in business.