Olympus just announced the E-M5 Mark II, an updated version of its popular micro Four Thirds E-M5 model, with an interesting new feature: its 16MegaPixel sensor, presumably similar to the one in other E-Mx bodies, has a high resolution mode where it gets shifted around by the image stabilization servos during exposure to capture, as they say in their press release
‘resolution that goes beyond full-frame DSLR cameras. 8 images are captured with 16-megapixel image information while moving the sensor by 0.5 pixel steps between each shot. The data from the 8 shots are then combined to produce a single, super-high resolution image, equivalent to the one captured with a 40-megapixel image sensor.’
A great idea that could give a welcome boost to the ‘sharpness’ of this handy system. Preliminary tests show that the E-M5 mk II 64MP High-Res mode gives some advantage in MTF50 linear spatial resolution compared to the Standard Shot 16MP mode with the captures in this post. Plus it apparently virtually eliminates the possibility of aliasing and moiré. Great stuff, Olympus.
[Edit: I wrote this article early in my understanding of the Modulation Transfer Function framework, note how Ken takes me to task on the issue of increased resolution in the comments. He is right, even though there may be some resolution improvement in the pixel-shifted image with some scene detail in some directions, the slanted edge method should measure none and any slight improvement shown here is due to external factors like differences in variables at the time of capture or processing before image data was written to the raw files. See here for the reason why.]
Shake it Baby
Moving the sensor around to good effect is not new. The Pentax K-3 for instance uses its in-camera image stabilizer to jiggle the sensor during exposure in order to provide anti aliasing action, therefore giving the photographer the ability to turn the AA on or off at will depending on the scene, trading off aliasing for resolution. The difference in the E-M5mkII implementation is most likely additional control and half-pixel precision in sensor positioning, resulting in AA action AND additional resolution. Clever, and the first time in a DSC I believe.
Since in High-Res Shot mode spatial information is captured at every half (original 16MP) pixel position the resulting raw file represents twice as many pixels horizontally and vertically, resulting in a 64MP file. In order to accomplish this feat 8 images are captured and assembled sequentially. It is therefore imperative that the subject be static, shot with good technique and with a totally vibration-free setup. The cost is an overhead of 1 second or so per capture, 3 second delay between captures and almost 8 times the raw file size: 103MB vs 14MB at base ISO.
It turns out that somewhat dated dcraw 9.22 is able to open the High-Res .ORF raw files in grayscale document mode, so these results are based on the raw, unprocessed data provided by the camera according to the procedure described here. Hopefully ol’ dcraw opens them properly with the -d -4 -T -w switches, if not ignore the rest of the post. [Edit: I get similar results by opening the HR raw files with the latest version of RawDigger which does support them, so we are most likely good to go.]
One High-Res EM5II raw image is twice the linear size of one captured in Standard Shot mode: 6938×9280 pixels, that’s 64MP, weighing in at 103MB at base ISO – versus 3472*4640 and 14MB or so, the same 16MP resolution as earlier E-Mx models.
About 10% MTF50 Advantage
Open Source MTF Mapper produced the following MTF50 results from the slanted edges in the center of DPR’s New Studio Scene raw files at f/5.6. The E-M5II is wearing a Zuiko 45mm:1.8 at f/5.6. A Sony a6000+55:1.8ZA (APS-C) and Nikon D810+85:1.8G (Full Frame) were thrown in for reference only, both also at f/5.6:
The blue curves show the relative performance of the Olympus E-M5 II in its 64MP High-Res Shot mode (dotted line) and 16MP Standard Shot mode (solid line). The right side of the diagram shows results from DPR’s ‘daylight’ Studio Scene raw files, the left side from the ‘Low light’ raw files. The first capture of each Low light or Daylight series is at base ISO, the last one of each series is at ISO 1600 (MTF curves start becoming unreliable at the noise level of this lofty ISO).
DPR’s E-M5 mark II’s ‘Daylight’ shots appear to suffer from some form of vibration. As mentioned in earlier articles shutter speeds in the 1/10 to 1/320s range are sometimes susceptible to vibration because of the multitude of small masses and mechanisms that start and stop motion to accomplish a capture. It could also be poor technique, or simply bad luck (heavy traffic nearby for instance) – spatial resolution measurements are famously finicky.
But the first three EM5II ‘Low Light’ shots with shutter speeds slower than 1/10s I think give us an idea of how well the new High-Res feature performs. In Standard 16MP shot mode, with it off, the E-M5II achieves on average MTF50 readings of around 900 lp/ph in the center of the frame, as would be expected of an AA-less sensor of this pixel size; in 64MP High Res Shot mode, about 1000 lp/ph. That’s quite an improvement, perhaps in this preliminary test not quite generically ‘beyond full-frame DSLR cameras’ as the marketing talk goes – but it’s an impressive, perceivable change nonetheless, besting the 16MP D4+85mm:1.8G at f/5.6 for example. Mileage may vary based on technique.
Keep in mind that although a 5% change in MTF50 readings may be hard to spot by eye in two images set side by side, 10% is definitely noticeable.
Excellent Bonus: Reduced Aliasing and Moiré
Perhaps more important than a potential resolution improvement, though, is the positive effect of the High-Res Shot on aliasing and moiré: it looks like it minimizes frequencies above Nyquist, effectively eliminating both. That’s brilliant if you need to shoot non-movable man-made subjects. This is how the MTF curves from the raw green channels alone look like according to MTF Mapper at base ISO:
Note how in Standard 16MP Shot mode the Modulation Transfer Function (black line) crosses Nyquist’s frequency with a lot of energy still, indicating that the sensor has a weak or (most likely) non existent Anti Aliasing filter. All that energy past Nyquist can and does show up in images under the guise of aliasing and moiré.
But in the 64MP High Res Shot the MTF curve (green line) stays well away from Nyquist, virtually leaving no spatial frequency information above it. The raw files should therefore be free of aliasing and moirè in this mode, truly excellent news.
Of course if you were to pixel peep the two images side by side, the one from the 16MP raw file with an MTF50 of 0.27 cy/px would look significantly ‘sharper’ than the 64MP image at 0.15 cy/px (see here for an explanation of the units used to measure spatial resolution). But when comparing different systems we know better than to look at images at 100% – so we quickly and figuratively fit both of them to our monitor and evaluate them based on image height instead. This is what the performance of the two modes looks like in comparable units of line pairs per image height:
Looking just at the green channels as in the previous figure, 64MP mode produces a slightly better MTF than 16MP (MTF50 1041 lp/ph vs 937 lp/ph ) – that’s noticeable on a displayed image. Plus it gets the no aliasing and moiré advantage. Minus the 1 second capture time, 3 second recycle time and 103MB raw file – not necessarily an issue in studio conditions.
A Little MTF Theory
So how do you model a sensor that captures and assembles ‘8 images […] with 16-megapixel image information while moving the sensor by 0.5 pixel steps between each shot’, as far as grayscale spatial resolution is concerned?
The key variables in the simple frequency domain model of spatial resolution (see this series of articles) are sampling pitch, pixel aperture (pixel size), f-number, filters and aberrations.
I think we start by recognizing that these are 8 separate captures of a neutral subject with the same lens at the same f-number, therefore the imaging plane is sampled 8 times with the same pixel aperture, filter and aberrations as in 16MP Standard Shot mode. So if the lens was not ‘outresolved’ at good ol’ 16MP, it will still not be outresolved when in 64MP High-Res Shot mode.
Assuming that two pixels of data are recorded every physical pixel-width as suggested by current literature, linear sampling pitch for the 64MP ORF neutral raw files is effectively doubled. The combination of an unchanged pixel aperture and doubled pitch is a relatively wider integrating function. The top hat hangs over neighbouring pixels, mixing spatial information from adjacent photosites as if the lens were moderately out of focus. And in fact the 64MP High Res MTF green curve can be modeled fairly accurately by a defocus function with about a 0.7 wavelength Optical Path Difference:
The green solid line is the measured MTF curve, the black solid line is the prediction of the model and the product of the three main components: f/5.6 diffraction, 3.74 micro pixel aperture/pitch and 0.7 wavelength OPD lens blur/defocus (dashed lines). Recall that Lord Rayleigh’s in-focus criterion was an optical path difference of less than 1/4 wavelength.
So if the working assumption holds, as far as modeling grayscale spatial resolution of the HR ORF file goes, we can actually think of High Res mode as producing a moderately out of focus 64MP capture.
Main Advantage = No Aliasing and Moirè
The no aliasing and moiré bonus is simply due to the fact that twice the sampling frequency means that the Nyquist frequency is shifted twice as far out, near diffraction extinction, where there is virtually no incoming energy. Since the Slanted Edge Method of measuring MTF takes Nyquist out of the equation by supersampling the data by a large factor (8x with MTF Mapper), ideally the pixel-shifted vs not shifted MTF curves should be exactly equal. Here they are not equal most likely because of non-idealities in the capture taking/shifting process, resulting in a slightly better shifted MTF.
All in all a very interesting feature. Good job Olympus.
Thanks Jack, I also thought that it is not possible to make a 64 MP from 16MP sensor!
But, I did not thought it would improve aliasing and moiré.
I thought that the noise is improved. What about the noise?
Hi Jean Pierre. These images are not suitable to estimate noise performance. For noise evaluation we’ll have to wait for DxO or someone who actually owns the camera to perform proper tests.
OK and thanks
I’m confused. What do you mean, “…it is not possible to make a 64 MP from 16MP sensor.” It is a 64MP image from a 16MP sensor. The number of pixel do give a product of 64MP, do they not?
6938*9280=64,384,640
Using the Hubble Space Telescope, NASA created a 1.5GP (gigapixel) image of Andromeda. (69,536 x 22,230 pixels) The Hubble’s sensor array uses a 16MP sensor.
It is not the endsum of the pixel, which makes better image quality. It is the pixel from the sensor and sensorsize. Why middleformat sensors give better image quality? And does a 20MP from 1″ sensor gives better image quality as an MFT-Senor with 16MP?
MP is not that, what it is important. Sensor size, photo diode, digicam built in processor and RAW storage of 12, 14 or 16 bit!
And the test of Jack shows us, what in real the 16MP sensor of the EM-5II can give us.
I agree. But your statement didn’t suggest state anything about the quality. “Thanks Jack, I also thought that it is not possible to make a 64 MP from 16MP sensor!” Those are your words.
I never suggested more pixels makes better quality.
Thanks, now it is clear for all readers.
Just to be clear, it is possible, however, to get 64MP from a 16MP sensor. And it does appear to increase quality overall.
Yes, a little bit, because the shift-movement is only 1/2 pixel of the sensor. Link.
Another nice post!
HF
Thanks!
I appreciate all the effort that you put in, but I’m not sure that your conclusions are valid. Or perhaps it simply shows the limitations of MTF50 as a methodology.
Since you based it on the DPReview test shots, one need only look with your eyes, rather than a specialized tool, to see that the difference in real resolution with the Olympus High Res mode is significant.
http://www.dpreview.com/previews/olympus-om-d-e-m5-ii/7
The best place to see this is in the text that gets progressively smaller directly above the center of the frame.
With the E-M5 II in High Res mode (RAW or JPEG) it is possible to read the very last sentence in the absolute smallest text:
“Altogether we have no so much difficulty as might be expected in determining our bearings.”
While it is also possible, with some significant struggle (and with the important foreknowledge of what the text really says!) to make out those lines with the A6000, it’s clear that the real-world resolution advantage lies with the E-M5, whereas your conclusion suggests that the A6000 has a 15% percent MTF advantage, while the D810 has a 35% advantage while looking nearly equivalent (slightly smaller and sharper when comparing the D810 RAW to the E-M5 JPEG) however with the added problem of moire and false colour from demosaicing.
You might be better off re-running your testing with the E-M5 High Res 40MP JPEGs rather than the 64MP RAWs to see if you get different results from MTF, but otherwise this appears to be a case of “lies, damn lies, and statistics” where test charts clearly belie what an ordinary set of eyes can plainly see.
Hi Turbofrog,
I only work off the unprocessed raw data to concentrate on the spatial resolution information produced by the hardware/firmware, unencumbered by subjective (post) processing: you can read more about it here.
The beauty of MTF50 measured this way in the context of ‘sharpness’ measurements is that it is quantitative and objective, so there is no danger of entering into an endless qualitative discussion of ‘mine is sharper than yours’. When done properly it is quite accurate. There is always the chance of user error but I doubt that the folks at DPR use worse technique than your average photographer.
On the other hand they unfortunately use LR to render the images in their New Studio Scene. And it is well known that LR’s processing varies from camera to camera.
In any case I would be more than happy to perform the test again if you or anyone else can point me to HR shot mode ORF captures of a slanted edge near the center of the field of view.
Thanks for your feedback.
Jack
PS A more in-depth answer here.
Maybe this is of interest to you, as it is a further analysis of the Hires-mode: http://philservice.typepad.com/Limits_of_Resolution/Limits_of_Resolution_5_Sensor_Shift.pdf
Best, HF
Nice find, HF. Interesting estimate of how much demosaicing degrades linear resolution. When we measure MTF off raw files with the slanted edge method we are effectively measuring the sensor undemosaiced, as if it did not have a CFA (monochrome). My experience is that current advanced demosaicing algorithms come very, very close to monochrome performance when measuring the luminosity channel of slanted edges in Bayer files.
You haven’t actually measured anything meaningful here, you have forgotten a fundamental aspect of slanted edge MTF measurements – they are *already* a super resolution technique.
The slanted edge is based on a known target (the edge) crossing at multiple sub-pixel phases so that the MTF can be computed at well above Nyquist. In other words, while High Res mode shifts the pixel array by 1/2 pixel dimension the slanted edge test is already shifting the edge at much finer gradations that 1/2. Turning on High Res does nothing at all to recover more information from a slanted edge test.
Now, what you have measured is the effects of the interpolation kernel the camera uses internally to convert its full color 32MP diagonally sampled image into a Bayer 64MP rectangularly sampled image. Slanted edge MTF results are of course dramatically affected by any sharpening or any difference in a interpolation kernel.
So the claim of a MTF50 boost of any percentage at all is really just an artifact – you haven’t actually measured anything about high-res mode in that regard. You just measured a bit about the interpolation kernel. Of course in any final image there will be a demosaicing kernel which would further alter the results.
Now, the graphs about the aliasing are a nice illustration of the whole point to High Res mode – it is an in camera super resolution technique and when applied to real world unknown targets it yields significantly more scene information to work with. But for an MTF slanted edge test it brings nothing to the table – the MTF test was already a super-resolution technique to begin with, all High Res did was put a sharpening kernel in the way.
You need to understand the underlying image processing of slanted edge MTF tests to get meaningful results. Otherwise you end up with erroneous conclusions like these.
Hi Ken, that’s assuming a lot and I mostly agree with you 🙂
I have seen two differing descriptions of how the images are collected (one by dpr and the other by Olympus itself), none on how they are combined into the raw data. Either way the slanted edge method as used above provides a directional radial slice of the 2D MTF of the 2D PSF of the resulting image(s) as stored in the raw data. The data depends on the imaging system that took the image, its physical setup and processing – all of which are reflected in the relative MTF curves. There are lots of provisos discussed elsewhere on this site.
If you are interested in pursuing this further look up Radon Transform and Fourier Slice Theorem and/or send me an email via the ‘about’ form top right.
“the slanted edge method as used above provides a directional radial slice of the 2D MTF of the 2D PSF of the resulting image(s) as stored in the raw data.”
Which in the 16MP capture does not include a sharpening or resampling kernel.
Which in the 64MP capture DOES include a sharpening/resampling kernel. It has to because the original actual RAW sensor data was never 64MP on a horizontal/vertical grid. It was 32MP on a diagonal grid.
Moving the sensor around changes nothing about the MTF of the system because the MTF is determined by the pixel size, microlens, filter stack and so forth. Moving it around changes nothing in that regard (as the other articles on your blog show). It *only* changes the sampling of that MTF.
But the slanted edge test by design already oversamples the target. The extra sampling of the 64MP brings no new MTF information to measurement compared to the 16MP.
So again, all you have measured here by comparing the 16MP and 64MP modes and seeing a 10% increase in MTF50 is the resampling/sharpening kernel that camera used to take its 32MP diagonal capture to a 64MP horz/vert pseudo-RAW.
It is an apples to oranges comparison akin to testing JPEGs from a camera with different sharpening settings set. Which you know of course invalidates any slanted edge comparisons.
The slanted edge method quantifies the spatial resolution information available in that one direction in the raw data, independently of how it got there (sampling, resampling or whatever): that’s the information the photographer has to work with. You can check in the next post whether the measurement correlates well with perception. I otherwise agree with your comments on how linear MTF is affected by the shifting process. If you want we can talk about it some more off line, just send me a message via the ‘about’ form top right.
Cheers,
Jack
Hello Jack,
Some measurements I made using an E-M5 II with an Olympus 60mm lens support your summary that the high-resolution mode provides a ‘moderately out of focus 64MP capture’.
My results, obtained by observing the edge-spread of an inclined (as opposed to slanted or tilted) edge, suggest that the high-resolution mode resolves 1.45x better than the normal resolution mode — less than the 2x one might expect going from 16MP->64MP but still quite an improvement.
I wrote up the results and method at http://web.ncf.ca/jim/photomicrography/loCAte/gallery/olympus60mm/
Your posting goes deeper than I am able into what’s probably going on in that camera so it was a pleasure to read, thanks.
Jim
Hi Jim, nice site you have there and nice work! I don’t know loCAte and it looks interesting so I’ll take a look when I have some time.
Jack