Over the last couple of years I’ve been using Frans van den Bergh‘s excellent open source MTF Mapper to measure the Modulation Transfer Function of imaging systems off a slanted edge target, as you may have seen in these pages.  As long as one understands how to get the most out of it I find it a solid product that gives reliable results, with MTF50 typically well within 2% of actual in less than ideal real-world situations (see below).  I had little to compare it to other than to tests published by gear testing sites:  they apparently mostly use a commercial package called Imatest for their slanted edge readings – and it seemed to correlate well with those.

Then recently Jim Kasson pointed out sfrmat3, the matlab program written by Peter Burns who is a slanted edge method expert who worked at Kodak and was a member of the committee responsible for ISO12233, the resolution and spatial frequency response standard for photography.  sfrmat3 is considered to be a solid implementation of the standard and many, including Imatest, benchmark against it – so I was curious to see how MTF Mapper 0.4.1.6 would compare.  It did well.

MTF Mapper comes with a sleeper little executable called mtf_generate_rectangle that allows one to create files of synthetic slanted edges with known properties – such as angles, contrast, MTF50 and noise – for testing purposes.  It is actually much more sophisticated than that, permitting the creation of advanced custom test charts  including objects like Siemens stars, USAF1951, pinch charts and more – you can read all about it at Frans’ blog here.  Today however we will just use it to generate square rectangles the top edge of which will be fed to MTF Mapper and sfrmat3 in order to get their estimates of a Modulation Transfer Function.  This is what a typical mtf_generate_rectangle output file looks like:

Below is a first easy try of the procedure.  The edges were produced to represent conditions slightly worse than I normally see in captures of studio charts: a 186 pixel edge (horizontally), slanted 9 degrees, 1% gaussian noise, 90% contrast, gaussian blur with a MTF50 of 0.15 cycles/pixel (cy/px).  You are looking at three curves: the almost invisible black line is the actual MTF used to generate the edge, the purple one is the result of sfrmat3’s edge analysis and the blue one is MTF Mapper’s.

Both packages perform well in the lower frequencies hitting MTF50’s 0.15 cy/px spot on, although in the upper frequencies sfrmat3 appears to bounce around a little, hovering above the real value of zero.  This situation by the way is very similar to the 64MP E-M5II real studio capture seen in a previous post: MTF50 of 0.15 cy/px, contrast 92%.

But the lower frequencies are no test because, as Frans mentions in his blog, accuracy decreases as frequency increases: “MTF Mapper’s relative measurement error will remain below 2% at real-world noise levels” at 0.3 cy/px (currently typical of a sharp system) and below 5% at 0.5 cy/px (currently non-existent system).  He has a nice chart there to show how the error appears to grow exponentially  (?) with frequency.

So here is a more typical situation, MTF50 = 0.3 cy/px, normal noise:

Still both nailing MTF50 but you can see that the two analyzers do not quite hug the curve as much from 0.4 cy/px on.  We can make things harder by adding more noise, which can actually be done two ways: more actual noise and/or fewer pixels to evaluate.  Here is MTF50 at 0.35 cy/px, twice the gaussian noise and only 50 pixels on the side instead of 186, brutal:

Note how the kink present in both systems around MTF50 would cause us to overestimate it by a fair margin because of the short edge/noisier image.  If we were to put through new edges with the same parameters the result would be slightly different each time (probably an underestimation the second time) with a mean equal to the reference and a standard deviation from it related to noise levels/edge length.  So to smooth the MTF curve out we can simply reduce Edge Spread Function noise by making the edge longer, back to 186 pixels (at a still noisy 2%)

Better, and MTF50 looks like it is now back within range.  So the longer the edge, the better as far as minimizing the effect of noise is concerned.  On the other hand one needs to be careful not to go so long that lens distortion starts impacting the formation of a tight Edge Spread Function.  MTF Mapper only ever looks at a maximum of 400 pixels on the edge, in part I think for this reason.

Looking at these charts I get the impression that both programs do a  similarly good job, although maybe MTF Mapper 0.4.1.6 stays a touch closer to the reference curve, especially in the higher frequencies.

One key parameter that needs to be estimated very accurately in order for the programs to provide meaningful results is the angle of the slanted edge with respect to the sensing plane.  To check their performance at this task I bumped up noise to an unrealistic 4% and put 100 different edges through both.  The actual angle was 9 degrees

The average over 100 runs was pretty well spot on for both, but MTF Mapper had a substantially tighter standard deviation:

To check sfrmat3’s apparent hovering bias in the upper frequencies I ran 16 edges in normal noise conditions, subtracted the reference from each MTF curve and summed them all up.  This is the resulting plot at a target MTF50 of 30 cy/px:

The vertical axis represents 16 times the absolute MTF error at each spatial frequency.  If there were no bias the figure should be very near zero, and it virtually is up to about 0.65 cy/px.

Above 0.65 cy/px sfrmat3 appears to show some bias,  perhaps unsurprisingly because of what we saw earlier: values in the upper frequencies tend to zero and there are no negative values in MTF to average the sum down, so they can only go up.   Nothing to worry too much about as we know that above 0.5 cycles per pixel MTF curve reliability goes down quickly as frequency is increased further so we typically do not put much weight on results there.  I also understand that sfrmat3 does not apodize while MTF Mapper does so it may be related to that.  Apodization is not without its drawbacks but for our (photographic) purposes I think it tends to produce a smoother output which takes a little guessing out of the game.  What we don’t know will not hurt us.

In the end I believe that both programs will give very similar MTF curves and MTF50 results in many conditions.  I will feel comfortable using both interchangeably with perhaps a leaning towards MTF Mapper because of its slightly more controlled output. I will definitely continue using sfrmat3 to double check MTF Mapper’s results in difficult situations, as I’ve been doing recently.  Thank you Peter Burns and Frans van den Bergh for two excellent, professional quality, free tools.

 

PS MTF Mapper has since gone through a few major revisions, improving apodization and edge angle robustness.  Here is a comparison of its performance vs Imatest.