The first thing you can try is to reduce the number of Layout Rays in the system. The Layout Rays are only used for showing rays on the layouts and will not have an effect on the Detector Viewer.

Next, if you have CAD object in the system, I would recommending decreasing the ‘Chord Tolerance’ and ‘Surface Tolerance’.

Other faceted objects in the system can have the resolution turned down as well by using the ‘Draw’ tab in the Object Properties. This may free up some memory.

Finally, in the Non-Sequential tab of the System Explorer, there are settings for the ‘Maximum Segments per Ray’ and ‘Maximum Source File Rays in Memory’. Decreasing the ‘Maximum Segments per Ray’ could free up enough memory to render the system. Increasing the ‘Maximum Source File Rays in Memory’ could also work (A recommended value would be 1,000,000 rays). I would also suggest checking that the ‘Minimum Relative Intensity’ option is on the order of 1e-3 or 1e-4 and making sure your ‘Maximum Segments per Ray’ is high.

If these do not work, feel free to post your file here!