A virtual propagation means the rays are propagated in a direction opposite to the direction energy would actually flow. Virtual propagations are often useful for placing virtual sources or pupils. Consider this negative lens, which diverges an input beam:

If the diverging beam is traced backwards, along the negative z direction, it forms a 'virtual image'. These are the blue rays in the image above. This image cannot be measured directly in real life, but OpticStudio can compute spot size, wavefront error, MTF etc. just as if it were a real image. What's more, the virtual image can be relayed by subsequent optics just as if it was a real image.

Virtual propagation requires that the ray be traced in the opposite direction to real rays. By default, real rays propagate along the positive z-direction, as indicated by the coordinate axes in the layout above.

A complication arises when mirrors are used. Mirrors change the direction of propagation of a ray, and so after a mirror, real rays propagate in the -z direction, and virtual rays propagate in the +z direction. After a second reflection, real rays are again propagating in positive z, and virtual rays in negative.