Combing artifacts in 30D RAW histograms - what casues them? (Chuck Westfall?)

Hi all,

It has been two and a half months since I bought a 30D (after 14 years with two film SLRs) - it is a wonderful camera!

Being a programmer however I just had to look into the RAW data that the camera generates. One of the reasons I did it is because I had read that the intermediate ISO sensitivities (e.g. ISO 125) on the 30D are achieved by scaling the already digitized signal in the firmware. It indeed turned out to be the case. As a result the RAW histograms of photos taken at intermediate ISOs have quite heavy (dense) combing artifacts – which is not at all surprising.

However I also discovered that the RAW histograms of some photos taken at full stop ISO sensitivities are also combed! Here is an example of a good histogram:

ISO 100, EC 0, 1/80 f/22


And here is and example of a combed one:

ISO 100, EC +1/3, 1/500 f2


What the above shows is that even at full ISOs the firmware of the 30D sometimes manipulates the RAW data coming from the A/D converters before writing it to the CR2 file – so the “RAW file” is not really RAW in these cases.

How often does it happen? Well I wrote a program (and a batch file) that can go through any number of CR2 files and check whether their histograms are combed. Using that found that from the 1304 CR2 files that I have – all of them shot at full ISOs - 216 have combed histograms. The period of the combing pattern varies from one photo to the other. I have put together a web page that has many more examples, the software used and some examples from other Canon dSLRs.
http://www.pages.drexel.edu/~par24/rawhistogram/rawhistogram.html

Even though I tried – I was unable to find any correlation between the shooting conditions and the combing appearing in the histograms. I find all of the above very interesting and would really like to hear any ideas or explanations of why it is happening. You can also try it on your own RAW files using the software I wrote.

Best regards,
Peter Ruevski

Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Chuck Westfall(Canon USA) said:

Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Click to expand...

Chuck,

Again, we appreciate your following threads here for us!

Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?
Asher

Hi Chuck,

Thank you very much for your quick reply and for following user forums.
As for the combing artifacts in round ISO speed RAW histograms - I do not think that they visibly affect image quality. In some almost identical photos where one had the combing and the other did not - I could not see a difference in the image quality. In fact I am very satisfied with the image quality of my 30D! What is more, most post-processing operations do "much worse" things to the data.

So this is not about image quality at all - and I was very careful not to imply such a thing - it is more of an "academic" interest for me - an unsolved mystery in a way. Canon's "no comment" policy is perfectly understandable, I just though no harm can come from asking

As for the intermediate ISOs - I do not mind them at all - and they are not a "mystery". What is more they would certainly be useful for people shooting JPEG - and when shooting RAW one can achieve the effect "consciously" in RAW conversion and just avoid them.

Thank you again for your reply.

Best regards,
Peter Ruevski

P.S. Everyone else - the mystery remains

>>Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?<<

Hi, Asher:

Those questions are precisely the ones that would lead to 'further comments' on image quality that Canon Inc. is not willing to discuss. Sorry!

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

It appears to me that you are analysing dpraw interpretation of the files, not the actual raw data.

Stan Jirman said:

Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

Click to expand...

Stan, I was thinking the same thing. I do check my histiograms and I don't recall seeing combs on my 5D images. I'll look out for them in my current processing.

Peter, how many files would you need? Which ones?

Asher

My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?

Nill
~~
www.toulme.net

@Harvey Moore

Indeed I am analyzing the output of dcraw, however I am using it in the mode in which it outputs unmodified RAW data (not even Bayer demosaicing is done) to a 16bit PGM file. I posted a similar thread on dpreview (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19009766) and a similar question came up. I have traced through (debugged) the dcraw code path that outputs the RAW data and I believe that it is indeed untouched 12bit raw data as it came from the ADCs stored in 16bit values inside the PGM.

Bottom line - I am 99.9% certain that the combing is not introduced by dcraw.

@Asher Kelman

I would love to analyze some 5D files. Ideally it would be the same scene shot under different ISO settings - including 1/3 steps - and preferably one that you would not mind me posing the pictures of on the page I mentioned in the original post.

Keep in mind however that the combing may be a rare occurrence - I have a friend with a 350D, he ran the search on about 500 of his CR2 files and only two showed combing.

If you are using a PC (Windows) you can PM me and I can send you the required files and instructions so you can run the search on a big collection of files.

As for seeing the combing in the histograms - I am talking about histograms of the RAW data - even processing the RAW file to a 16 bit linear TIFF would already hide the effect - the bayer demosaicing is in effect an interpolation. The only other program that I am aware of (apart from mine) that can show the real RAW histogram is Iris http://www.astrosurf.com/buil/us/iris/iris.htm - and it shows the combing too. See this (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19131963) for an explanation how to use it - it has a GUI so it is easier to use (and immensely more powerful) than my program. The only advantage of mine is that it can be used to do an automated search through many files.

Peter,

Impressive work. I'm embedded in Mac computers! It would be great if XP users could volunteer to test.

Asher

Nill Toulme said:

My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?

Click to expand...

You mean "Italian Mafia style" hairdo - lots of gel and combed backwards?

No, we are talking about those equally spaced vertical lines (in fact they are single values that are zero) in the second histogram above. It is colloquially called combing because it looks like a comb. It happens for example when you “stretch the histogram” of an image (essentially multiplying each pixel value by a number). Imagine a simple monochrome image, which only has pixels with five shades of gray 0, 1, 2, 3 and 4. If you multiply every pixel by 2 now your image will be "twice as bright" (in fact it will have twice the dynamic range). But it will only have pixels with values 0, 2, 4, 6 and 8 – there will be no pixels whatsoever with values 1, 3, 5 and 7. If you build the histogram every odd value will be 0.

But who knows - maybe the Mafia has something to do with this too ;-)

If you got a pre-emptive "here's our official position" from Canon, you must be on to something here

I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.

Guy

Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!

Hi Doug,

I knew you'd arrive! Here's the challenge. What are the limits if any to dcRAW combed files?

A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.

Asher

Asher Kelman said:

I have a feeling that the file might be less robust.

Click to expand...

Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.

Hi all,

I will try to answer in a single post so that the thread does not get unnecessarily littered with my posts.

First let me say that I would love to look at any sample RAW files that anyone is willing to send me. For those that do not have any web space to host big files you can use http://www.yousendit.com/ - it is a service for sending big files. Just put your own email as the recipient, upload the file and when the upload is done it will give you a unique link for downloading the file. Email me that link and let me know if you just want me to analyze it and let the forum know how it looks or I can also post it as an example.

Guy Tal said:

If you got a pre-emptive "here's our official position" from Canon, you must be on to something here

Click to expand...

Well I am "on to something" but it may not be serious at all – they may simply be worried that people get the wrong message – e.g. that the image quality is compromised – which I actually do not think it is – at least not visibly.

Guy Tal said:

I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.

Click to expand...

The 30D does not provide ISO 50 when the ISO expansion is enabled – only ISO 3200 – and that is just ISO 1600 pushed a stop – so not interesting. If you send me an ISO 50 file I would certainly look at it.

Doug Kerr said:

Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!

Click to expand...

Absolutely! I spent quite a bit of time trying to, but so far I do not have any even remotely defensible conjecture to present.

Asher Kelman said:

A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.

Click to expand...

BobSmith said:

Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.

Click to expand...

On this I tend to agree with Bob – combing is a "fact of life" in post processing – the first time you do for example Levels, Curves, Brightness/Contrast in Photoshop – you rare doing "much worse" things to the data in the image. So the combing in the raw will hardly matter for image quality – in fact it is gone as soon as the interpolation to an RGB image from the Bayer (CFA) image is done during RAW processing.
The above does not make it any less interesting though - the the combing certainly appears because the firmware did something to the RAW data. Unless this "something" is just an obscure firmware bug then they (Canon engineers) must have felt it is necessary - and they have a proven track record of not touching the RAW data for nothing - so it may be compensating for some deficiency/effect in the image pipeline.

And finally (as a consolation and one more reason to like Canon all Nikon RAW files are like that - much worse in fact. Look at the histograms halfway down this page:
http://www.astrosurf.com/buil/d70v10d/eval.htm
in the "6. CODAGE NUMERIQUE / NUMERICAL CODING" section. And more information here:
http://www.majid.info/mylos/weblog/2004/05/02-1.html

Best regards,
Peter

I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Richard McNeil said:

I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Click to expand...

Should have added using the EOS 5D.

Fascinating stuff! I'll certainly download the source, compile, and see what my photo repository looks like.

I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.

If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done Unix provides much better scripting facilities... O.O

Daniel_Hyams said:

If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done Unix provides much better scripting facilities... O.O

Click to expand...

Hi Daniel and everyone else,

This is almost exacly what I did. And the DOS batch language is indeed quite limited. Here is my batch file:

=====cut here

@echo off

for /R .\ %%f in (*.cr2) do (
dcraw.exe -D -4 -t 0 -o 0 -v %%f )

for /R .\ %%f in (*.pgm) do (
rawhistogram.exe %%f & (if ERRORLEVEL 1 echo Combed>%%f.flg) & del /q %%f
)

echo List of combed RAW files:> ComedRAWList.txt
echo ------------------------->> ComedRAWList.txt
dir /s /b *.flg >> ComedRAWList.txt

=====cut here

You can use it to easily check many raw files for combing at once. Let us presume that your raw files are stored in a directory "C:\rawfiles" and in many sub-directories.

1. Place "findcombedraws.bat", "dcraw.exe" and "rawhistogram.exe" in the C:\rawfiles directory.
2. Start a command prompt – under Windows XP choose "Start | Run", type "cmd" an press OK.
3. At the command prompt type:
cd c:\rawfiles
and press enter.
4. Type:
findcomedraws.bat
and press enter.
5. The batch file will go through the current directory and all its sub-directories, find all CR2 files, convert them to PGM and then check whether their histograms are combed. This will take a while if you have many RAW files.
6. A text file "ComedRAWList.txt" that lists all of the combed RAW files found will be created in C:\rawfiles.
7. In addition for each *.cr2 file there will be one *.CSV file created that contains the RAW histogram data - you can plot this file in Excel. For all combed *.cr2 files there will be a small text file called *.pgm.flg that simply marks that the corresponding RAW file is combed. The *.PGM files will be deleted to save space.

Since the batch file may be useful and make it easier for peole to search their raw file collections I made a package that contains everything needed and placed it here:
http://www.pages.drexel.edu/~par24/rawhistogram/findcombedraws.zip
I also updated the page to offer a download of this file.

Hopefully this will stimulate more people to search (completely automatically) through their raw files.

Best regards,
Peter

Richard McNeil said:

I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.

Click to expand...

Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:

The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:


Best regards,
Peter

Peter Ruevsky said:

Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:

The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:

Best regards,
Peter

Click to expand...

Peter,

Yes i used all ISO settings including iso 125. Let me run your batch file and see what it uncovers.

Richard

None were flagged as combed!

Richard McNeil said:

None were flagged as combed!

Click to expand...

This is great! How many files did you analyze? It confirms once again that the 5D imaging pipeline is indeed better than the 20D/30D one.

So, Peter (or anyone else, really), do you think that the combed histograms in the intermediate ISOs on the 30D have a detrimental effect on image quality? Also, what percentage of intermediate ISO 30D files have combed histograms (I am under the impression that it is 100%).

Peter Ruevsky said:

This is great! How many files did you analyze? It confirms once again that the 5D imaging pipeline is indeed better than the 20D/30D one.

Click to expand...

I only did a quick 200 image test - not exactly sure of the ISO distribution. Unfortunately my test files are home and will have to wait till tonight; however, I don't expect to see any combing in those either. Will post results tonight.

Richard

I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

Richard