To model a diffuse surface in OpticStudio, you need to add a Lambertian scatter profile to the face of the object. Lambertian scattering will give the rays an equal probability of scattering anywhere in the unit circle - most diffuse surfaces are nearly Lambertian.
The Lambertian scatter profile can be entered under the 'Coat/Scatter' tab in the 'Object Properties' of the object you would like to make diffuse.
Make sure you choose the correct 'Face' that you would like the scatter profile applied to. The 'Number of Rays' effects how many scatter rays will be generated. The scatter fraction will be the amount of power from an incident ray gets transmitted into scattered refelction/transmission vs. specular reflection/transmission. For example, if the scatter fraction is 1, the specular ray will receive zero energy and will no longer be traced; and all the energy will be divided equally among the scattered rays. If the fraction is set to 0, no scattered rays will be traced, and the specular ray retains all the original energy.
Make sure when running the ray trace to have 'Split NSC Rays' and 'Scatter NSC Rays' checked on in order to see the effects of your scatter profiles.