Now that we know how to open 12-bit raw files captured with the new Raspberry Pi High Quality Camera, we can learn a bit more about the capabilities of its 1/2.3″ Sony IMX477 sensor from a keen photographer’s perspective.  The subject is a bit dry, so I will give you the summary upfront.  These figures were obtained with my HQ module at room temperature and the raspistill – -raw (-r) command:

The IMX477 was introduced in December 2016 for “DSC Cam” and “consumer use camcorder” applications.  It is a 1/2.3″ Back Side illuminated sensor with 3040×4056 pixels of 1.55um pitch, related to Full Frame by about a 5.5x multiplier.  It is based on Exmor RS BSI stacked technology and capable of capturing full resolution 12-bit images at about 11 fps via on-chip column parallel ADCs.^[1]

Black and White Levels

In order to capture image information fully, most sensor designers today do not set up the Analog to Digital Converters (ADC),  that convert the output of the pixels to raw Data Numbers (DN), to produce zero DN  with zero light.  They in fact typically have them start a little above that, with an arbitrary offset  known in raw conversion circles as the Black Level, which allows the capture of some or all noise that might spill below it.  A raw converter can later subtract the Black Level from raw data as needed to align zero DN with no light on sensor.

The Black Level figure is usually dependent on the environment and setup of the sensor. I could not find it in the HQ Camera Maker Notes or in optical black pixels but it is easy to estimate by taking a raw capture with the lens cap on in a dark environment.  Below is the histogram of the center crop of a dark field taken at room temperature with analog gain equal to 1, the minimum available.  The histogram’s shape should look approximately Gaussian, because the dominant component of Read Noise comes with a Normal distribution and, with no light, there should be no Shot Noise and PRNU.

The Black Level is the mean value of the black field crop, which in this case computes to 256.3 DN.  The histogram doesn’t look perfectly Gaussian because of quantization and possibly some Dark Signal Non Uniformities (DSNU).  Here it looks typical for base gain, obviously the Black Level offset was not exactly centered on a DN.  It almost never is.

As the gain is raised the Black Level creeps up but you can tell that the sensor designers are able to compensate, about a quarter of an LSB per step, attempting to keep it within a limited range.  With my module at room temperature that range was about 256.0 – 257.3 DN.

At the other end of the range, the White Level is the highest value produced by the ADC.  It appears to stay constant at 2^12 – 1 = 4095 DN for all gains as demonstrated by obvious clipping in the relative histograms, which I won’t bore you with.

Analog Gain Linearity

Analog Gain is the active ingredient of what is typically  referred to as ‘ISO’ in photography.  Increasing it reduces Dynamic Range but it also has the potential to reduce electronic noise relative to signal, an important determinant of Image Quality in the deepest shadows.  It is typically accomplished via Programmable Gain Amplifiers, which these days tend to be quite linear.  I was however curious to check whether the analog gain switch in raspistill (- ag) had one-to-one correspondence with actual sensor output.   Taking captures of an LED at varying shutter speeds verified that this is indeed the case with the IMX477 as you can see below:

This shows that when we tell raspistill to increase gain from a value of 1 to anywhere up to 16, its maximum working range, we physically get what we ask for.  Note that the linear fit suggests that if one expected to use gains uniformly throughout the range, a good generic Black Level for the purpose would unsurprisingly be 256.6 DN with this setup.

By the way, with the HQ camera we are not constrained to whole numbers for gain, but we can also use intermediate values like 2.43.  The driver rounds it to the nearest the PGA can do and reports the actual analog gain used in the MakerNotes as an integer that we then need to divide by 256.  The range is 256 to 4096.  In fact I set up raspistill with integer gains from 1 to 16 but only the ones that were powers of 2 came back exactly as requested, the others had some rounding.  For instance -ag 14 came back as 3543, which corresponds to an analog gain of 13.8398.  The plots on this page use reported gains.

Read Noise

When capturing a black field with the lens cap on one can obtain an estimate of the random Read Noise of the sensor combined with any fixed pattern noise and DSNU by measuring the standard deviation of the resulting data (see Figure 1).  These are the standard deviation readings of center crops of all CFA  pixels at working gains:

If this plot looks a little different than how it is usually seen elsewhere as a function of (ISO) stops, it is because I left it linear for ease of reading values off the graph.  The fact that the standard deviation of the captures does not double when gain is doubled means that raising analog amplification favors cleaner electronics in the earlier part of the chain, thus reducing input-referred Read Noise in physical units of photoelectrons rms (e-).  This is sometimes desirable, as described below.

Engineering Dynamic Range

Increasing analog gain may result in decreased input-referred Read Noise but it definitely results in a decrease of clipping headroom, stop for stop, thus affecting another major determinant of Image Quality, Dynamic Range.  It appears that clipping stays constant at 4095 DN at all gains, so plotting the ratio of highest output signal (4095-BlackLevel) to the relative dark frame standard deviation shown above we can obtain a rough estimate of the Dynamic Range of the HQ sensor:

Simplified eDR starts at 11.3 stops at base gain/ISO and declines as shown as the gain is raised.  This is typical performance for recent small Back Side Illuminated sensors out of Sony.

The simplified plot above assumes the lower end of the range to have no signal.  In practice however the ‘Range’ in DR refers to a signal swing, from minimum acceptable signal to clipping.  Minimum acceptable signal for engineering purposes is often taken to occur when SNR is equal to 1.  With that assumption engineering DR is 11.15 stops at base gain with my setup.

Photon Transfer Curve at Base Gain

We can plot a Photon Transfer Curve of mean Signal versus Signal to Noise ratio (SNR) by capturing uniformly lit raw images at increasing light levels.  The following one is based on frame pair subtraction and was obtained from central crops of the green raw channels at base gain/ISO^[*]

The beauty of such a curve is that it allows us to peer into the depths of the sensor without prying it open – thus estimating key parameters like average pixel Full Well Count (Clipping), input-referred random Read Noise and System Gain in physical units of photoelectrons (e-).  For my module at base gain and room temperature those turn out to be

• FWC = 8180 e-
• Read Noise = 3.065 e-
• System Gain = 0.469 DN/e-

With a 1.55um pixel, FWC per unit area is about 3400 e-/um^2 – at the higher end of current technology, assuming no filtering (see the next section below).  An input referred read noise of about 3 e- is respectable at base gain today.

The fit was repeated for the other CFA channels and then with all channels together obtaining virtually identical results so I would think there might be a couple of percentage points of play in these figures but not much more than that, for my unit as setup.

Note that by using frame pair subtraction we are effectively minimizing any non-random pattern noise and non-uniformities, including PRNU and 1/sqrt(12) quantization noise.  Read Noise so determined is in fact 3.065*0.469 = 1.44 DN in output-referred units, indeed lower than the 1.53 DN that we obtained earlier from just reading the standard deviation of the center crop of a single dark frame.  The difference is due to the aforementioned non-random noise sources and fairly typical for this type of sensor at base gain and room temperature.

Twinkle Twinkle Little Star

Having captured a number of dark fields it was then easy to generate Spectrograms in order to detect any potential processing performed on sensor data before it is written to the raw file.  Spectrograms are especially sensitive to periodic changes in the horizontal and vertical directions and that’s how we like them since the pixels are in a rectangular grid so aligned.  Here they are for the four CFA channel at base gain/ISO, top row B and G, bottom row G and R.

They all look about the same.  The good news is that there appears to be no obvious pattern noise indicating potential ‘banding’ or ‘striping’ issues in the deepest shadows.  Then again there is little excuse for banding these days unless the sensor features optical phase detect pixels to aid in focusing, which the IMX477 clearly doesn’t.

The bad news is that those should look like squiggly horizontal lines and instead show attenuation of higher frequencies, suggesting noise suppression before the data is written to file.  In fact the patterns have a signature similar to Sony’s Star Eater algorithm^[2], loathed by astro photographers and raw data purists.  Perhaps this speaks to the sensor’s video pedigree, though I read that it is intended for smartphones and other devices as well – and Sony shows its Application as ‘DSC Cam’^[3].  DSC stands for Digital Still Camera.

The fact that the pattern is evident in all four CFA channels already with a shutter speed of 1/8000s at base gain is somewhat disappointing though by no means show-stopping: with its 5.5x focal length multiplier, the HQ cam shows off its astro credentials in these two hours of exposure time on the M27 planetary nebula coupled to a William Optics SpaceCat 51 APO 250mm f/4.9.^[4]

Ask and Ye Shall Be Ostracized

Interestingly the low pass filtering seems to be more aggressive in the horizontal direction, a fact that prompted your humble correspondent to ask on the Pi Camera Board whether the HQ pipeline performed noise reduction before writing data to the raw file.   He was told that the Pi drivers didn’t but that the sensor module itself could perform Defective Pixel (DPC) and other Corrections subliminally.  Continued probing on how to turn filtering off was not appreciated by the local Jabba the Hutt  who promptly jettisoned him out of the pool by performing a backward tripped cannonball.  Touchy, touchy.  Should anyone know more on this subject please let me know.

[Update: it seems that Jabba eventually got around to sending to Sony the email the request of which got me exiled.^[5]  The result is a new firmware switch that kills all or part of Defective Pixel Correction, a first for a Sony sensor as far as I know.  After updating to the latest and greatest rpi software you can switch DPC off step by step as follows:

sudo vcdbg set imx477_dpc <value>

where <value> can be one of:
0 – All DPC disabled.
1 – Enable mapped on-sensor DPC.
2 – Enable dynamic on-sensor DPC.
3 – Enable mapped and dynamic on-sensor DPC.

The default is 3.

Astrophotographer tmorley on the board was kind enough to share images of a before and after test,^[6] so here is a preview of what turning off varying amounts of DPC did to captures from his HQ sensor, courtesy of David Plowman:

With DPC fully off images are both noisier and with a number of hot pixels (bottom right).

I have since sudo apt update and full-upgrade not to mention rpi-update to latest firmware rpi-5.4.y that incorporates the new switch.  Spectrograms of raw images obtained via

raspiraw -g 0 -eus 2000

are virtually identical to those in Figure 7 above for all DPC modes.  Read Noise also remained unchanged throughout.  This indicates that either the new firmware switch for some reason did not work, or Sony is still doing some other form of low-pass noise reduction.  ]

Summary and Conclusions

All in all a three year old 1/2.3″ 12MP sensor with decent performance for its size, similar to what one could find in a good smartphone – with the notable exception of on-sensor Phase Detect Autofocus Pixels.  I am quite happy to have an almost^[7] open-source camera to play with for the price of a dinner out.  Coupling it to a good lens and driving it with a Raspberry Pi creates limitless possibilities  in terms of control and applications.   What we discovered about it in this post is summarized below.

Notes and References

<sub>1. The Sony IMX477 sensor information brief can be viewed here.
2. This DPR thread has an in-depth discussion of the make up of the Sony Star Eater algorithm, which however in that case presents only at shutter speeds of 4s or greater.  If you are an Astrophotographer you can read a bit more about the issue here.
3. Scroll down to the middle of this page to find the IMX477’s Application listed as DSC Cam.
4. For more excellent astrophotography images captured with a Raspberry Pi and an HQ cam see here.
5. Link, where our hero gets banished from the Pi Camera Planet for having the temerity of suggesting that salaried workers should stop hiding behind NDAs and simply send a two line email to Sony asking how to turn off subliminal noise reduction. In the happy ending he gets part of his wish while still in exile.
6. The captures used in Figure 8 can be downloaded from here.
7. ‘Almost’ Open Source because, as I discovered, there are some things about the IMX477 that we are not meant to know.</sub>