For a pancake type of VR, most of the straylight issue comes from the imperfect polarizer model. Here we have provided an example that users can play with for investigation.
The components in the system are as below.
In the polarizer (object #6), users can control the light leakage at reflection and transmission side. The polarizer is based on this paper but with some modifications: Non-paraxial idealized polarizer model
Note the input light is with circular polarization.