Floating point number representation; issues in OpticStudio

  • 29 May 2020
  • 2 replies

Userlevel 4
Badge +1

OpticStudio is no different from any other software in using floating-point representation for storing numbers other than integers. Although the accuracy is very high, sometimes a number like 0.7 can be stored internally as a very close approximation, like 0.700000000001712 or 0.6999999999998394 for example. For calculations these differences are usually acceptable. The differences are generally confined to the digits near the limits of double precision storage anyway.

Be very careful, however, using numbers like these in a comparative, like checking if a variable is less than another. In Zemax Programming Language (ZPL), OS operands, and some types of displays (bar charts), particularly with discrete data divisions, you might see unexpected results. For instance, in the following code, there is a big difference between A being 7.1 and A being 7.099999999981728.

IF A >= 7.1

  PRINT 'Success'


  PRINT 'Fail'

For similar reasons, the use of non-integers as indices in a FOR-loop is discouraged, as the counter may not increment precisely as expected.

Filter strings are another area where an implicit comparison takes place. Consider selecting out, for example, a spectrum subset using a filter such as X_WAVERANGE(3, .5, .599) to select out rays hitting object 3 between 500 nm up to but not including 600 nm, with some spectrum provided in discrete steps. If the wavelength of a particular ray or group of rays is represented as .499999999991827, they will not be included. And if another filter like X_WAVERANGE(3, .4, .499) is included, it will still fail. So even in the event the spectrum is defined at a coarser scale, such as every .05, a reasonable boundary like .499 on the low side and .500 on the upper side will both miss the data.

It is not always easy to examine the data points directly to full precision, but the effects of these roundings can be intuited fairly readily if you remain aware of them and are careful about selecting your boundary points for any given calculation or display. What is not generally possible, however, is to find a setting or property that blocks the problem from occuring. These rounding issues take place at the microprocessor level, not the software level for OpticStudio. The solution is good practices in coding and data management.



2 replies

There is this issue in the component editor that everything is rounding to 3 digits. Can I change to 4 somehow? My system is in μm.

Userlevel 5
Badge +1

Hi Elisavet,

You can modify the reported digits in any Editor in your Project Preferences:

Please let us know if I misunderstood your needs or if you have any other questions here -- thanks!

~ Angel