Now that we know from the introductory article that the spatial frequency response of a typical perfect digital camera and lens (its Modulation Transfer Function) can be modeled simply as the product of the Fourier Transform of the Point Spread Function of the lens and pixel aperture, convolved with a Dirac delta grid at cycles-per-pixel pitch spacing

(1)  

we can take a closer look at each of those components ( here indicating normalization to one at the origin).   I used Matlab to generate the examples below but you can easily do the same with a spreadsheet.  

Diffraction’s MTF

The lens in our simple example is assumed to have circular aperture and to be diffraction limited with no aberrations, therefore just producing an Airy pattern on the sensing plane as defined in the last article.  At f/16 the pattern would look as shown in Figure 1 of the last post, a tiny white dot at the center of a 1024×1024 linear unit black image.  It can be modeled according to the following continuous equation in the frequency domain:

(2)  

with the linear spatial frequency normalized so that it is unity at extinction: .  The MTF is zero when  is greater than 1.  This is what it looks like as a solid and in profile.

The graphs show MTF on the Z axis and the spatial resolution in cycles per pixel pitch (c/p) on the  X and Y axes.  As mentioned earlier the perfect pixel here is considered to be a square 5 linear units on the side.  It doesn’t really matter but think of a unit as one micron if it helps understanding (more on this later).

If we perform a Discrete Fourier Transform of the f/16 Airy pattern image in Figure 1 of the previous article we get the same result as applying Equation (2), indicating that the formula is correct and that Matlab’s FFT routine works accurately.  Below is the horizontal   central radial slice of the resulting two dimensional MTF: the ‘Measured’ dots are from the DFT while the ‘Modeled’ blue line is from Equation (2).  The plot is similar to the graph in Figure 1b but showing positive frequencies only, zoomed into the area of detail that we are normally used to seeing when evaluating MTF curves, the 0->1 c/p range.  Thanks to the Projection Slice Theorem we know that it represents the one dimensional MTF of the projection of the corresponding two dimensional PSF in one direction only:

They are identical.  Because the Airy disc is rotationally symmetric, the one dimensional MTF profile will be the same no matter what radial slice out of the 2D MTF we look at.  This graph shows numerically that the ideal circular aperture lens MTF matches that obtained by taking the Discrete Fourier Transform of the image of its Point Spread Function on the sensing plane.

This for a perfect lens of circular aperture and no aberrations.  For the MTF of a lens of circular aperture with simple aberrations see this article, with complex aberrations see this one.

Pixel Aperture’s MTF

We can perform the same exercise by comparing the FT and DFT of a pixel’s photodiode active area, aka the pixel’s effective aperture or footprint.   Starting simple, if we assume the aperture to be a perfectly uniform square, the relative frequency response can be modeled in the frequency domain as

(3)  

with and the horizontal and vertical spatial frequency components and the width of the  square effective aperture.  Equation (3) is shown below as a solid on the left; and on the right its X-Z axis profile, as before.

A horizontal radial slice from the origin with no vertical frequency component can be modeled in one dimension as follows:

(4)  

with ** the linear spatial frequency in the horizontal direction.  It would of course look the same in the vertical direction, with no horizontal component.

We want to compare this analytical model to the computed Discrete Fourier Transform of the 5×5 linear unit square footprint of the perfect pixel under investigation.  To do so we generate an image of the same size as the Airy pattern  image (1024×1024 linear units), fill it with zeros everywhere except in the center where a 5×5 square area of ones is inserted, and take the two dimensional DFT of that.  The plot below shows a horizontal radial slice from the analytical model in Equation (4) (‘Modeled’) and from the numerical result (‘Measured’) after correction for the fact that the latter is not bandwidth limited (to learn more about why this correction is needed see the Appendix at the bottom of the article)

We can’t see the blue modeled line hidden under the orange measured dots because the curves are identical.  Good, it works.

Modeling Lens and Pixel Aperture Together

The model predicts the ‘measurements’ obtained from each component in isolation.   We can check how well it works on the components together by applying equation (1).  We can generate a modeled 2D MTF by multiplying together equations (2) and (3) and compare the result to the 2D Discrete Fourier Transform of the image of the Airy disc multiplied point by point by the 2D DFT of the image of the 5×5 linear unit pixel aperture square.

Because diffraction at f/16 is dominant in this example the result appears to be rotationally symmetric, but it is not exactly so as can be gleaned by looking at the minimal differences below extinction (about 0.55 c/p) in Figure 5.  At larger relative apertures the differences would be more marked.

We can check theory against practice as before by plotting a radial slice of the MTF of the Lens times the MTF of the Pixel Aperture solid shown in Figure 6.  We can work in one dimension by multiplying diffraction from equation (2) by a single sinc(f*w) function for the horizontal slice ‘Model’; and since we are assuming a perfect square aperture with 100% Fill Factor, by sinc(f*w)^2 for the diagonal slice, as shown below.

The curves are all on top of each other, indicating success, the simple model works.  Of course real lenses are not unaberrated and real pixels are not  perfectly square.   You can read more about basic aberrations in the center of the lens starting here and some nuances of practical pixels below.

Asymmetric Squares

In this simple model series we have assumed pixels with perfectly square, uniform aperture per Equations (2) and (3) .  It is a subjective assumption, with some basis in reality since I understand that these days sensor designers strive for square active areas and gapless microlenses [EDIT: no longer necessarily true today as shown by Figure 10].

However the response of a square is not rotationally symmetric.  Its edgy pixels affect captured image spatial frequencies differently depending on the orientation of the detail, compared to that of the pixels, on the sensor.  For instance if we had taken an MTF radial slice 45 degrees from the horizontal or vertical axis alone this would have been the result with a perfect 100% fill factor square pixel:

The blue line of the model is once again invisible under the measured dots, indicating that the analytical and numerical results match perfectly.  A 45 degree diagonal radial slice from the origin means that horizontal and vertical frequencies are the same . Applying Equation (3) then gives

(5)  

This is the MTF of a triangle because that’s the intensity profile that arises when a square at 45 degrees to an axis is projected onto it.  The linear spatial frequency in cycles per original horizontal pixel pitch here is the same as earlier but care must be taken when plotting these values that diagonal is longer than in the horizontal (or vertical) direction.

Slices intermediate angles will yield curves somewhere in between these two extremes per equation (3):

Note that slices of less than 8 degree inclination are virtually identical to the horizontal’s in the typical region of interest below 1 c/p used to evaluate imaging systems, while changes become noticeable above that.  This is one of the reasons why the edge is ideally set up 4 to 6 degrees off the horizontal/vertical in the slanted edge method.

Pixel Aperture and Fill Factor

Effective pixel aperture width () and pixel spacing (horizontal/vertical sampling pitch ) are separate variables.  We sometime take them to have the same value in classic photographic digital sensors – therefore implicitly assuming a perfect square pixel footprint resulting in an effective 100% fill factor.  But that’s definitely not a given, especially in specialized imaging applications.

Fill Factor is the active area of a pixel, the photosite or photodiode, expressed as a percentage of the pixel’s overall area suggested by pitch, including all support electronics.

Effective Fill Factor (e) takes into consideration the capability of the pixel design – especially the microlenses, if present – to concentrate onto the photodiode photons that would otherwise fall onto non active portions of the pixel and be lost.  A well designed pixel with a good microlens can have an effective Fill Factor that approaches 100% of the ideal, uniform, square area suggested by pitch.  This is only true near the center of the Field of View in practice.

With square pixel layouts, Effective Fill Factor e is related to pixel pitch and effective pixel aperture width  as follows:

(6)  

For example one could have effective aperture width = 4um and pixel pitch = 8um to simulate an 8um pixel with 25% Fill Factor (sometimes also referred to as 50% linear Fill Factor because each side of an effective square would then be 50% of the pitch); or alternatively to simulate using every other pixel in a sensor with perfect contiguous 4um pixels (say for a single color channel of a Bayer CFA sensor).

A smaller effective pixel aperture area all else equal means a ‘sharper’ image  because it reduces the low-pass effect of the pixel on system MTF by shifting the relative sinc’s first null (1/ ) to a higher spatial frequency than what would be suggested by pitch (1/).  Of course that also means more aliasing and fewer interacting photons, hence lower sensitivity and a noisier picture – but that is a compromise that some manufacturers are willing to make for their use cases.  Medium Format cameras like some Phase Ones or the Fuji GFX 50S are known to have reduced or absent microlenses to achieve this objective.

[More recently other specialized cameras appear to be following in those footsteps, see for instance this analysis of the Nikon Z7]

Pixel Aperture Shape

Nor do the microlenses and the active area – therefore effective pixel aperture – need to necessarily be uniform or square, although I understand sensor designers strive for a square active area in digital cameras.   Taking those into consideration a more appropriate model for the shape and intensity of the Point Spread Function of effective pixel aperture would typically be a pillow, for what would normally be called a ‘square’ pixel.  Even then, the pillow would be distorted near the corners of the Field of View.

On the other hand one could instead desire to have round Back Side Illuminated effectie pixels of diameter at spacing.  In that case the sinc function used in Formula 1, the Fourier Transform of a square pixel aperture. would have to be replaced by the equivalent for a disk.  The Fourier Transform of a disk is the  Sombrero/jinc/besinc function***

(7)  

If the disk’s diameter is equal to pixel pitch , the function has a first zero at about 1.22 c/p**** and the disk’s area is 78% () of our reference uniform square with sides equal to .   Assuming a square reference in this case would cause us to underestimate effective fill factor, because is equal to 67%.

Sampling and aliasing are next.

Notes

* Code for Figure 1 is shown as an answer to Kenny’s question below.
** Spatial frequency is explained in this article.
*** Note the resemblance of the Sombrero Function to Equation (1) in the previous article for the Airy Function: the latter is also the Fourier Transform (albeit squared) of a perfect circular aperture, in that case of the exit pupil of the lens.
****  The first zero of = 3.8317 divided by is 1.2197, see code for Figure 11.

 

Appendix: Reconciling Analytical and Numerical Results for non-band-limited signals

When we compare the DFT of a 5×5 square at the center of a 1024×1024 image (‘measured’ Figure 12 below) with results modeled by analytical Equation (4) it looks like there is a pretty good fit up to about 1 c/p but after that the model and measured curves part ways.

That’s because, contrary to the Fourier Transform shown as the ‘Modeled’ blue line above, which works on unbounded continuous curves resulting in only a baseband signal, the DFT works on a finite set of discrete input and output values, which produce an infinite set of replicas of the baseband signal in the frequency domain.  If the signal is not band limited to start with, energy from these replicas seeps back into baseband in the form of aliasing.

A rect function is definitely not band limited (it has infinite support), so the aliased energy from the replicas adds up quickly.  Its DFT is therefore not the sinc function shown in Equation (4) of the continuous model, but an aliased sinc (also known as asinc, periodic sinc, Dirichlet or diric function), which is separable and – when the rect is symmetric – can be written as follows for each axis:

(8)  

with * in cycles per sample and the width of the square aperture as before.  This can be thought of as the ratio of two sincs, equation (4) divided by :

(9)  

So in this case the output of the DFT differs from the continuous transform by this additional factor in the denominator (its effect shown as the yellow dashed curve above).  To make its MTF curve match the continuous transform model we therefore need to multiply it by , which brings us back to having ‘measured’ results accurately match theory for a square aperture, as shown below just in the horizontal direction:

The DFT’s output was not apparently different in the earlier Airy plots because of diffraction extinction, which produces an already bounded, band limited signal.   The way the effect of the finite size of the sampling interval on the signal is often mitigated in practice when using numerical methods for these calculations is to increase the size of the sampling interval relative to the signal, effectively raising the ‘ROI’ curve in Figure 12.  For more on this you can read the discussion on Q at the bottom of this article.