P.S: I have intentionally set the splitter coating on back surface. 

Hi Saurabh,

Thanks for your question here on the Community forums!

I have taken a look at your file and the difference between the 3D Layout and the Shaded Model was caused by the different Settings, i.e. Scattering was enabled only in the 3D Layout Settings, but not in the Shaded Model.
Once Scattering is enabled, the Minimum Relative Ray Intensity has to be decreased under the System Explorer > Non-Sequential settings to make low-energy ray segments, such as those reflected from the front face of the splitter, visible in the System Viewer plots. 
For more details about coatings and scattering in NSC mode, please check out this knowledgebase article:
How to add coatings and scattering functions to Non-Sequential objects – Knowledgebase (zemax.com)

Regarding your question about how to mask specific rays or beam parts, you need to save the Ray Trace results to a ZRD file, and then apply Filter Strings.
You may find the available Filter Strings in the Help system at:
The Setup Tab > Editors Group (Setup Tab) > Non-sequential Component Editor > Non-sequential Overview > The Filter String
The process of applying Filter Strings is discussed in detail in this article:
Identifying specific rays using filter strings – Knowledgebase (zemax.com)

Finally, you may define the Polarization of the beam under the Object Properties > Sources tab of the source object. Make sure to untick Random Polarization to enable the settings:

If you have any further questions, please ask!

Best,
Csilla