• Please use real names.

    Greetings to all who have registered to OPF and those guests taking a look around. Please use real names. Registrations with fictitious names will not be processed. REAL NAMES ONLY will be processed

    Firstname Lastname

    Register

    We are a courteous and supportive community. No need to hide behind an alia. If you have a genuine need for privacy/secrecy then let me know!
  • Welcome to the new site. Here's a thread about the update where you can post your feedback, ask questions or spot those nasty bugs!

Combing artifacts in 30D RAW histograms - what casues them? (Chuck Westfall?)

Peter Ruevski

New member
Hi all,

It has been two and a half months since I bought a 30D (after 14 years with two film SLRs) - it is a wonderful camera!

Being a programmer however I just had to look into the RAW data that the camera generates. One of the reasons I did it is because I had read that the intermediate ISO sensitivities (e.g. ISO 125) on the 30D are achieved by scaling the already digitized signal in the firmware. It indeed turned out to be the case. As a result the RAW histograms of photos taken at intermediate ISOs have quite heavy (dense) combing artifacts – which is not at all surprising.

However I also discovered that the RAW histograms of some photos taken at full stop ISO sensitivities are also combed! Here is an example of a good histogram:

ISO 100, EC 0, 1/80 f/22


And here is and example of a combed one:

ISO 100, EC +1/3, 1/500 f2


What the above shows is that even at full ISOs the firmware of the 30D sometimes manipulates the RAW data coming from the A/D converters before writing it to the CR2 file – so the “RAW file” is not really RAW in these cases.

How often does it happen? Well I wrote a program (and a batch file) that can go through any number of CR2 files and check whether their histograms are combed. Using that found that from the 1304 CR2 files that I have – all of them shot at full ISOs - 216 have combed histograms. The period of the combing pattern varies from one photo to the other. I have put together a web page that has many more examples, the software used and some examples from other Canon dSLRs.
http://www.pages.drexel.edu/~par24/rawhistogram/rawhistogram.html

Even though I tried – I was unable to find any correlation between the shooting conditions and the combing appearing in the histograms. I find all of the above very interesting and would really like to hear any ideas or explanations of why it is happening. You can also try it on your own RAW files using the software I wrote.

Best regards,
Peter Ruevski
 
Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.
 

Asher Kelman

OPF Owner/Editor-in-Chief
Chuck Westfall(Canon USA) said:
Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.
Chuck,

Again, we appreciate your following threads here for us!

Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?
Asher
 

Peter Ruevski

New member
Hi Chuck,

Thank you very much for your quick reply and for following user forums.
As for the combing artifacts in round ISO speed RAW histograms - I do not think that they visibly affect image quality. In some almost identical photos where one had the combing and the other did not - I could not see a difference in the image quality. In fact I am very satisfied with the image quality of my 30D! What is more, most post-processing operations do "much worse" things to the data.

So this is not about image quality at all - and I was very careful not to imply such a thing - it is more of an "academic" interest for me - an unsolved mystery in a way. Canon's "no comment" policy is perfectly understandable, I just though no harm can come from asking :)

As for the intermediate ISOs - I do not mind them at all - and they are not a "mystery". What is more they would certainly be useful for people shooting JPEG - and when shooting RAW one can achieve the effect "consciously" in RAW conversion and just avoid them.

Thank you again for your reply.

Best regards,
Peter Ruevski

P.S. Everyone else - the mystery remains :)
 
>>Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?<<

Hi, Asher:

Those questions are precisely the ones that would lead to 'further comments' on image quality that Canon Inc. is not willing to discuss. Sorry!

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.
 

Stan Jirman

New member
Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.
 

Asher Kelman

OPF Owner/Editor-in-Chief
Stan Jirman said:
Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.
Stan, I was thinking the same thing. I do check my histiograms and I don't recall seeing combs on my 5D images. I'll look out for them in my current processing.

Peter, how many files would you need? Which ones?

Asher
 

Peter Ruevski

New member
@Harvey Moore

Indeed I am analyzing the output of dcraw, however I am using it in the mode in which it outputs unmodified RAW data (not even Bayer demosaicing is done) to a 16bit PGM file. I posted a similar thread on dpreview (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19009766) and a similar question came up. I have traced through (debugged) the dcraw code path that outputs the RAW data and I believe that it is indeed untouched 12bit raw data as it came from the ADCs stored in 16bit values inside the PGM.

Bottom line - I am 99.9% certain that the combing is not introduced by dcraw.

@Asher Kelman

I would love to analyze some 5D files. Ideally it would be the same scene shot under different ISO settings - including 1/3 steps - and preferably one that you would not mind me posing the pictures of on the page I mentioned in the original post.

Keep in mind however that the combing may be a rare occurrence - I have a friend with a 350D, he ran the search on about 500 of his CR2 files and only two showed combing.

If you are using a PC (Windows) you can PM me and I can send you the required files and instructions so you can run the search on a big collection of files.

As for seeing the combing in the histograms - I am talking about histograms of the RAW data - even processing the RAW file to a 16 bit linear TIFF would already hide the effect - the bayer demosaicing is in effect an interpolation. The only other program that I am aware of (apart from mine) that can show the real RAW histogram is Iris http://www.astrosurf.com/buil/us/iris/iris.htm - and it shows the combing too. See this (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19131963) for an explanation how to use it - it has a GUI so it is easier to use (and immensely more powerful) than my program. The only advantage of mine is that it can be used to do an automated search through many files.
 

Asher Kelman

OPF Owner/Editor-in-Chief
Peter,

Impressive work. I'm embedded in Mac computers! It would be great if XP users could volunteer to test.

Asher
 

Peter Ruevski

New member
Nill Toulme said:
My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?
You mean "Italian Mafia style" hairdo - lots of gel and combed backwards? :)

No, we are talking about those equally spaced vertical lines (in fact they are single values that are zero) in the second histogram above. It is colloquially called combing because it looks like a comb. It happens for example when you “stretch the histogram” of an image (essentially multiplying each pixel value by a number). Imagine a simple monochrome image, which only has pixels with five shades of gray 0, 1, 2, 3 and 4. If you multiply every pixel by 2 now your image will be "twice as bright" (in fact it will have twice the dynamic range). But it will only have pixels with values 0, 2, 4, 6 and 8 – there will be no pixels whatsoever with values 1, 3, 5 and 7. If you build the histogram every odd value will be 0.

But who knows - maybe the Mafia has something to do with this too ;-)
 

Guy Tal

Editor at Large
If you got a pre-emptive "here's our official position" from Canon, you must be on to something here :)

I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.

Guy
 
D

Doug Kerr

Guest
Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!
 

Asher Kelman

OPF Owner/Editor-in-Chief
Hi Doug,

I knew you'd arrive! Here's the challenge. What are the limits if any to dcRAW combed files?

A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.

Asher
 

BobSmith

New member
Asher Kelman said:
I have a feeling that the file might be less robust.
Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.
 

Peter Ruevski

New member
Hi all,

I will try to answer in a single post so that the thread does not get unnecessarily littered with my posts.

First let me say that I would love to look at any sample RAW files that anyone is willing to send me. For those that do not have any web space to host big files you can use http://www.yousendit.com/ - it is a service for sending big files. Just put your own email as the recipient, upload the file and when the upload is done it will give you a unique link for downloading the file. Email me that link and let me know if you just want me to analyze it and let the forum know how it looks or I can also post it as an example.

Guy Tal said:
If you got a pre-emptive "here's our official position" from Canon, you must be on to something here :)
Well I am "on to something" but it may not be serious at all – they may simply be worried that people get the wrong message – e.g. that the image quality is compromised – which I actually do not think it is – at least not visibly.

Guy Tal said:
I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.
The 30D does not provide ISO 50 when the ISO expansion is enabled – only ISO 3200 – and that is just ISO 1600 pushed a stop – so not interesting. If you send me an ISO 50 file I would certainly look at it.

Doug Kerr said:
Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!
Absolutely! I spent quite a bit of time trying to, but so far I do not have any even remotely defensible conjecture to present.

Asher Kelman said:
A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.
BobSmith said:
Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.
On this I tend to agree with Bob – combing is a "fact of life" in post processing – the first time you do for example Levels, Curves, Brightness/Contrast in Photoshop – you rare doing "much worse" things to the data in the image. So the combing in the raw will hardly matter for image quality – in fact it is gone as soon as the interpolation to an RGB image from the Bayer (CFA) image is done during RAW processing.
The above does not make it any less interesting though - the the combing certainly appears because the firmware did something to the RAW data. Unless this "something" is just an obscure firmware bug then they (Canon engineers) must have felt it is necessary - and they have a proven track record of not touching the RAW data for nothing - so it may be compensating for some deficiency/effect in the image pipeline.

And finally (as a consolation and one more reason to like Canon :) all Nikon RAW files are like that - much worse in fact. Look at the histograms halfway down this page:
http://www.astrosurf.com/buil/d70v10d/eval.htm
in the "6. CODAGE NUMERIQUE / NUMERICAL CODING" section. And more information here:
http://www.majid.info/mylos/weblog/2004/05/02-1.html

Best regards,
Peter
 

Richard McNeil

New member
I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.
 

Richard McNeil

New member
Richard McNeil said:
I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.
Should have added using the EOS 5D.
 

Richard McNeil

New member
I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.
 

Daniel_Hyams

New member
If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done ;) Unix provides much better scripting facilities... O.O
 

Peter Ruevski

New member
Daniel_Hyams said:
If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done ;) Unix provides much better scripting facilities... O.O
Hi Daniel and everyone else,

This is almost exacly what I did. And the DOS batch language is indeed quite limited. Here is my batch file:

=====cut here

@echo off

for /R .\ %%f in (*.cr2) do (
dcraw.exe -D -4 -t 0 -o 0 -v %%f )

for /R .\ %%f in (*.pgm) do (
rawhistogram.exe %%f & (if ERRORLEVEL 1 echo Combed>%%f.flg) & del /q %%f
)

echo List of combed RAW files:> ComedRAWList.txt
echo ------------------------->> ComedRAWList.txt
dir /s /b *.flg >> ComedRAWList.txt

=====cut here

You can use it to easily check many raw files for combing at once. Let us presume that your raw files are stored in a directory "C:\rawfiles" and in many sub-directories.

1. Place "findcombedraws.bat", "dcraw.exe" and "rawhistogram.exe" in the C:\rawfiles directory.
2. Start a command prompt – under Windows XP choose "Start | Run", type "cmd" an press OK.
3. At the command prompt type:
cd c:\rawfiles
and press enter.
4. Type:
findcomedraws.bat
and press enter.
5. The batch file will go through the current directory and all its sub-directories, find all CR2 files, convert them to PGM and then check whether their histograms are combed. This will take a while if you have many RAW files.
6. A text file "ComedRAWList.txt" that lists all of the combed RAW files found will be created in C:\rawfiles.
7. In addition for each *.cr2 file there will be one *.CSV file created that contains the RAW histogram data - you can plot this file in Excel. For all combed *.cr2 files there will be a small text file called *.pgm.flg that simply marks that the corresponding RAW file is combed. The *.PGM files will be deleted to save space.

Since the batch file may be useful and make it easier for peole to search their raw file collections I made a package that contains everything needed and placed it here:
http://www.pages.drexel.edu/~par24/rawhistogram/findcombedraws.zip
I also updated the page to offer a download of this file.

Hopefully this will stimulate more people to search (completely automatically) through their raw files.

Best regards,
Peter
 

Peter Ruevski

New member
Richard McNeil said:
I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.
Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:

The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:


Best regards,
Peter
 

Richard McNeil

New member
Peter Ruevsky said:
Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:

The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:


Best regards,
Peter
Peter,

Yes i used all ISO settings including iso 125. Let me run your batch file and see what it uncovers.

Richard
 

Jason C Doss

New member
So, Peter (or anyone else, really), do you think that the combed histograms in the intermediate ISOs on the 30D have a detrimental effect on image quality? Also, what percentage of intermediate ISO 30D files have combed histograms (I am under the impression that it is 100%).
 

Richard McNeil

New member
Peter Ruevsky said:
This is great! How many files did you analyze? It confirms once again that the 5D imaging pipeline is indeed better than the 20D/30D one.
I only did a quick 200 image test - not exactly sure of the ISO distribution. Unfortunately my test files are home and will have to wait till tonight; however, I don't expect to see any combing in those either. Will post results tonight.

Richard
 

Richard McNeil

New member
I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

Richard
 
Top