advice requested on sequence of steps in raw conversion

[Alan Rew]

When RAW data from a Bayer sensor is converted to a usable RGB image, there are various steps that are carried out regardless of which RAW converter is used.

I realise that different RAW converters may vary the order of steps used, or omit some steps altogether, and I'm _ignoring_ the special case of DXO converters that may deal with chromatic aberration in the RAW values somehow.

Can somebody with some detailed knowledge of this please confirm whether I've got the basic sequence of events right in the following list?

A Bayer sensor with R, G and B filters is assumed, also an IR filter and anti-aliasing filter in front of the sensor.

1) De-mosaicing, to fill in the 'missing' values for each colour at each photosite.

2) Despite the presence of an anti-aliasing filter, additional steps could perhaps be taken here to deal with fine details like mesh patterns?

3) Colorimetric interpretation of the R, G and B values obtained, based on the characteristics of the colour filters used in the sensor.

4) White balance - maybe an estimate can be done at this point, or the value specified in the camera settings used?

5) Gamma adjustment of the data, to allow for the difference in response between the sensor & the human eye for example.

6) Noise reduction - could this be done earlier?

7) Sharpening - although perhaps the inherent 'sharpness' of the image depends on the earlier steps including the demosaicing algorithm used.

8) Conversion from the 'camera colour space' (if that's the correct term) to an RGB colour space such as ProPhoto RGB, for editing purposes.

I've derived this initial list from reading various documents, including Bruce Fraser's document 'Understanding Digital Raw Capture', which is downloadable from the Adobe web site.

Can any of the digital gurus in this forum help me out? Is there a consensus on a typical order for the above steps? Have I missed any steps out?

Thanks in anticipation for any comments or suggestions,

Alan

 

[Colleen Vermillion]

What is your goal in figuring this out exactly? Just curiosity or do you have a specific application of the knowledge in mind?

Here's how I would list the steps. A rendered image like a JPEG or TIFF would start the process at the noise reduction step.

• Extract and decompress the raw data from the file which usually has a depth of 12 or 14 bits and normalize it to a consistent range. Bibble (probably like most other converters) uses a 14 bit range at this point so we have room to adjust the image without clipping. Some other programs might use floating point or a different strategy.
• White balance the data using the values in the file, subtracting off the black level. Done before demosaicing to improve the quality of the interpolation. Some applications may not do this.
• Demosaic (any demosaicing artifact reduction step is incorporated in here)
• Perform highlight recovery and expand the range of the data to 16 bits
• Noise Reduction (This is highly variable - Bibble does Noise Ninja here)
• Exposure corrections
• White balance adjustment (if the user has changed it, back out the original and apply the new which is lossless unless the as-shot exposure and white balance is way off)
• Apply the tone curve if the color management doesn't do it
• Transform from the camera space into a working space (color management)

After that point, the processing is the same given a rendered or raw image. Noise reduction might apply a sharpening to compensate for the softening introduced by the correction. Typically raw converters default to some amount of sharpening to compensate for the demosaicing, but where and how it's applied is variable.

-Colleen

 

[Sean DeMerchant]

I've derived this initial list from reading various documents, including Bruce Fraser's document 'Understanding Digital Raw Capture', which is downloadable from the Adobe web site.

Can any of the digital gurus in this forum help me out? Is there a consensus on a typical order for the above steps? Have I missed any steps out?Is there a consensus on a typical order for the above steps? Have I missed any steps out?

Nothing on the Adobe site will teach you about the whys and hows of RAW conversion as Adobe does not publish such technical treatises. You have missed some steps here while creating artificial steps that have photographic reality, but little or no meaning in the underlying numbers. Steps 3 thru 5 are just one thing, color space transformation. Demosaicing is simply a specialized subset of interpolation algorithms.

As to order, you have simply listed some possible RAW conversion steps without any discernable order to them except the numbering. Without actual numbers/theoretical mathematical constructs one cannot understand and make determinations on order and uniqueness of operations.

You should use consumer documents (like Adobe publishes) to guide your workflow and improve your images.

You should use technical documents to help expand your understanding of the mathematics of what happens. You will needs years of full time study to learn the physics, mathematics, and computer science needed to understand this process.

For instance 1 & 2 above are the same thing. Demosaicing is simply a subset of interpolation methods. For instance one can accomplish 2 by simply minimizing the creation of artifacts in your interpolation algorithm rather than correcting them after the fact. Super-resolution is an equivalent problem to demosaicing with some additional degrees of precision and some additional degrees of freedom. To understand RAW conversion one should first understand the basics of super resolution. Understanding the basics of optics and image formation by lenses is needed to understand the basics of super resolution for photographic images. And the basics of super resolution also apply to other areas like accoustics and many other phenomena. For optical sensors, you will also need some understanding of the sensor (CCD, CMOS, ...) and how it handles incident light (microlenses, energy storage, ...).

You might find the following a good place to start:
Image mosaicing and super-resolution By David Capel


Online Dissertation

Then, once you understand the physics and optics you can begin to understand image reconstrusction means and begin the photographic software process.

The best order of steps depends upon the image at hand. The in practice order within a RAW converter is more likely bound by software architecture constraints, round off error and overflow, and good old fashioned time to market reducing business schedules (third rate hacks that work).

You should also be aware that this is the stuff of doctoral dissertations and it is a non-trivial subject. A college education in photography will do nothing (unless you took freshman physics, chemistry, and calculus) to even begin to comprehend the science behind photography. I myself am trained as a mathematician and I know enough to know that I do not know enough to be an expert in this area.

some thoughts,

Sean

 

[Colleen Vermillion]

You should also be aware that this is the stuff of doctoral dissertations and it is a non-trivial subject. A college education in photography will do nothing (unless you took freshman physics, chemistry, and calculus) to even begin to comprehend the science behind photography. I myself am trained as a mathematician and I know enough to know that I do not know enough to be an expert in this area.

There is a difference, I think, between understanding the process and understanding the science in enough depth to design improvements to the process. I think whys and wherefores of the process is accessible to anyone technical enough to want to discuss the list of things Alan posted. For example, someone could understand why he might want to do his noise correction before applying the tone curve without necessarily understanding signal processing in enough detail to create his own noise correction algorithm.

Conceptually white balancing, tone correction, and the color space transformation might all be lumped together as a color space transformation, but in practice it is useful to separate them in to independent steps. Both the tone and white balancing can be adjusted for artistic effect, so lumping them in with the conversion from the camera's color space to a more convenient working space like ERIMM or Lab is unnecessarily limiting.

One of my favorite adages - "The difference between theory and practice is that in theory, there is none and in practice there is" You can understand the practice without necessarily understanding all the theory and occasionally, understanding the theory can hurt your understanding of the practice (as demonstrated by a physicist friend of mine that tried to do my electrical circuits homework one night by deriving everything from his knowledge of particle physics).

-Colleen

 

[Alan Rew]

What is your goal in figuring this out exactly? Just curiosity or do you have a specific application of the knowledge in mind?
-Colleen

This is just for my own intellectual curiosity, although I'm thinking about producing a diagram in Visio which outlines all the steps, either to put up on the web or just to help my digital photography friends.

I have no ambitions whatsoever to write my own RAW converter. I do have a degree in Mathematics, and a lot of experience in software programming, but neither are very applicable to producing such products. Besides, my degree is about 30 years old, and has faded somewhat

I'm a digital photography enthusiast who's been shooting RAW with digital SLRs since 2004, tried a variety of different RAW converters over the years, and been mad enough to buy my own Gretag EyeOne for printer profiling. To ensure that I have even less of a normal life, I've spent untold hours using forums about colour management, digital photography, printing and Photoshop.

I'll reply to the various points that have been made so far later on, when I've had more time to digest them.

In the mean time, thanks to both of you, Colleen and Sean, for taking the time to offer advice - it's really helpful. This is the sort of knowledge that's very difficult for the 'ordinary' digital enthusiast to find out.

If anybody has any more suggestions I'd be very grateful.

Kind Regards,

Alan