# Value compression of raw data in the Leica M8

#### Doug Kerr

##### Well-known member
It is a usual article of faith that the digitized outputs of the various sensor channels are linear with respect to the photometric exposure on the sensor. The proportionality will depend on the ISO setting (which affects the analog gain from the actual photodetector outputs to the analog-to-digital converters) and in general there is an "offset" (the digital output for "dark" is often non-zero).

And to date my examination of the details of this had been focused on the Canon EOS (and G-series) arrangements.

But my study of the detailed description of typical raw data processing in the DNG specification introduced me to a new possibility: that the digitized sensor output data might be "compressed" (in the value sense) before being delivered in a "raw" file. The advantage (just as when we compress the r, g, and b coordinates of an RGB representation into the nonlinear R, G, and B form that is actually "stated" for an RGB color space) is that fewer bits can be used per "sample" while still maintaining the same overall "perceptual" resolution (again in the value sense).

In such a case, a table (held in the raw file) is used during raw processing to transform the "compressed" raw data back to "linear" form for further attention.

I found that a raw file from a Leica M8 (which is in fact in the DNG form to begin with) in fact held such a table. I extracted it with Exiftool and examined it. (We note that apparently the delivered raw - DNG - file for this camera has the raw data in 8-bit precision.)

The table had 256 entries, presumably for delivered raw data values of 0-255. The outputs spanned a range just short of 0 to 2^14 (that is, can be considered to be to a precision of 14 bits).

The function of this "decompression" table is this:

y = x^2​

where x is the delivered raw data value (on a scale of 0-1, scaled from its actual range of 0-255 for use in this equation) and y is the "decompressed" value (in this equation, on the basis of a range of 0-1, but in the actual table on a range of 0-16,383 - 14 bit precision).

This means that the "compression" done to the raw data before delivery is identical to a "gamma precompensation" function for gamma = 2.

By the way, for the Leica M8, the "black level" for the DNG file is stated at zero and the "maximum level" (often, but ill-advisedly, called "white level) is in fact stated as the maximum possible 14-bit value. Thus presumably the digitized data from the sensor has been "normalized" with regard to its range before applying the compression function. So for a "dark" photodetector, the delivered, compressed raw value would be 0 (8-bit representation). In processing, this would be converted by the table to the 14-bit value 0.

Very interesting.

Best regards,

Doug

#### Asher Kelman

##### OPF Owner/Editor-in-Chief
Doug I thought the linear function was primarily the distinction of the format primarily used in video, otherwise there is a roll off approaching "black" and again "approaching white" in the form of an S curve with asymptotic extremes.

Asher

#### Doug Kerr

##### Well-known member
Hi, Asher,

Doug I thought the linear function was primarily the distinction of the format primarily used in video, otherwise there is a roll off approaching "black" and again "approaching white" in the form of an S curve with asymptotic extremes.

I didn't think (in the case of still cameras) that happened at the sensor itself, but is rather applied in subsequent processing (typically subsequent to the delivery of the raw data from the camera).

The language of the DNG specification (which of course has an "indirect" relationship to these matters) seems to suggest that the behavior of the sensor itself (and the attending analog-to-digital converter) are expected to be at least nominally expected to be linear with photometric exposure ("linear light" is a common term for this).

Of course we know that in a typical digital still camera, there is a "tone curve" imposed as part of the processing of the sensor data into an RGB image (perhaps on the way to a JPEG file).

I had always assumed that one of the many advantages of taking the raw output from a camera is that we can modify such tone curves to suit our preferences for the particular shot.

But my knowledge of all this is very spotty and erratic.

Is it your thought that typically such an "S-curve" occurs by the time the sensor data is delivered in the raw file (when that is the form of output that is "taken"?

Thanks.

Best regards,

Doug

#### Doug Kerr

##### Well-known member
The above was one of several interesting things I learned about raw data handling in the Leica M8 camera.

An earlier one, brought to my attention by Asher (and I first I wasn't sure I believed there was any such thing!) was that in the Leica M8, correction for "vignetting" is applied in-camera to the raw data before it is delivered in a raw file (assuming that a "recognized" lens is aboard and that lens recognition is enabled).

The correction is defined separately for each of the camera's channels, thus allowing us to correct for "vignetting" that involves a chromaticity shift (as well as a luminance decline".

Because the mathematical definition of the radial variation in the luminance/chromaticity has several terms, we can by way of this "vignetting" correction process apply a chromaticity shift that is in fact uniform over the frame, thus allowing compensation for a "global color cast" imposed by a certain lens.

Again, all this is applied to the raw data before it is delivered from the camera.

I have not yet learned with certainty of other cameras in which this occurs.

Best regards,

Doug

#### Asher Kelman

##### OPF Owner/Editor-in-Chief
Hi, Asher,

I didn't think (in the case of still cameras) that happened at the sensor itself, but is rather applied in subsequent processing (typically subsequent to the delivery of the raw data from the camera).

The language of the DNG specification (which of course has an "indirect" relationship to these matters) seems to suggest that the behavior of the sensor itself (and the attending analog-to-digital converter) are expected to be at least nominally expected to be linear with photometric exposure ("linear light" is a common term for this).

Of course we know that in a typical digital still camera, there is a "tone curve" imposed as part of the processing of the sensor data into an RGB image (perhaps on the way to a JPEG file).

I had always assumed that one of the many advantages of taking the raw output from a camera is that we can modify such tone curves to suit our preferences for the particular shot.

But my knowledge of all this is very spotty and erratic.

Is it your thought that typically such an "S-curve" occurs by the time the sensor data is delivered in the raw file (when that is the form of output that is "

Yes, that's my current understanding. I need to find a reference to delineate my sense of it!

Asher

#### Doug Kerr

##### Well-known member
Hi, Asher,

Yes, that's my current understanding. I need to find a reference to delineate my sense of it!