Hi Shaojie,

Thank you for your post!

From this screenshot, it's kind of difficult to see what's going on inside the Diffraction grating. But the NSC Diffraction Grating object is basically a Standard Lens with a diffraction grating on one face, and this diffraction surface can diffract both in reflection and in transmission. If you assign a material on the Diff Grating object, it's possible for rays being diffracted back in reflection from one face of the grating, and then bounce inside of the Diff Grating object, something as shown below. 

Best regards,

Hui