Take a look at Sections 11.8 and 11.9 in Modern Optical Engineering 3rd Ed., by Warren Smith.  I’ve attached a couple of pages that explain how the geometric Line Spread Function (LSF) is calculated from a ray-tracing spot diagram.  This is obviously an approach based on geometrical optics, so diffraction is ignored.  It should provide a good approximation when the optical system is not diffraction limited (i.e., when aberrations dominate the point spread function).  The ESF is simply the integral of the LSF.

Regards,

Jeff

To add to Jeff’s answer, the Geometric ESF uses the spot diagram to calculate the edge spread function while the FFT ESF uses the FFT PSF to calculate the edge spread function.  Mathematically, the calculation is the same as explained in Jeff’s excerpt “the line spread function is determined by integrating the spot diagram in one direction” (where the spot diagram is either the Geometric Spot Diagram or the FFT PSF).   

The FFT ESF accounts for diffraction and should be used if the Spot Diagram is smaller than the Airy Disk or the wavefront is less than 1/4 waves; otherwise the Geometric ESF should be used.