Combing artifacts in 30D RAW histograms - what casues them? (Chuck Westfall?)

[Mike Robbins]

I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT). It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.

I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.

 

[Peter Ruevski]

I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

Hi Richard, The ISO 3200 lacking combing does not surprise me – it is well established that ISO 3200 a.k.a. "H" is simply ISO 1600 pushed (i.e. underexposed) one stop and a flag set in the file telling the raw conversion software to do +1 stop exposure adjustment. As far as the actual raw data is concerned ISO3200 is exactly the same as ISO1600 - that is why people say it is not useful when shooting raw. When shooting JPEG the firmware does the pushing so it is useful.

I do not know how ISO 50 is implemented in the 5D – I do not remember reading anything about it. Could it be just ISO 100 overexposed by one stop? It is easy to find out – just put the camera in manual mode on a tripod in front of a gray card/white wall and shoot two photos – one at ISO 50 and one at ISO 100 – with the same aperture and shutter settings. Then build the raw histograms and compare them. If they look the same then ISO 50 is just ISO 100 overexposed.

I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT).

So that means you have 35 combed raw files from the 5D?! Would you mind sending me a few examples? And maybe the one from Rebel XT as well? The effect is certainly not common:
- for your 5D - 35 out of a 1000 is 3.5%
- for your 350D - 1 out of 150 is 0.7%
- for my friend’s 350D is was similar to yours
- for my 30D - 216 out of 1304 is 16.6%

It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.

This is quite interesting. I have been trying to find a correlation since day one. The 50mm f/1.4 lens is certainly a convincing correlation - but on the other hand if you shoot with it all the time it is no correlation at all. For my 30D photos I have pretty much ruled out ISO speed (excluding intermediate ones), shutter and aperture individually as contributing factors – I have both "good" and "bad" files for all kinds of settings of all three. Of course they may be interacting in a "strange" way. I have not looked at the lens used as closely - I should.

I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.

This is a good test and it confirms Richard’s results. It shows that all the ISO speeds are properly implemented in the 5D. And since you still have combed raw files (as does my 30D at round ISO values) - as you said it shows that something more complex is at work and the combing does not seem to be due to simple scaling.

Which leads me to:

So, Peter (or anyone else, really), do you think that the combed histograms in the intermediate ISOs on the 30D have a detrimental effect on image quality?

At least to me - no visible one, but I do not consider myself to have a sharp/discerning eye On the other hand they are certainly not raw data. What is more when shooting RAW they are not really needed. So I personally do not use them.

Also, what percentage of intermediate ISO 30D files have combed histograms (I am under the impression that it is 100%).

Your impression is correct - 100%. The combing is different though - e.g. ISO125 is “good old down to zero” combing as the examples I posted earlier. ISO160 however is a "half hearted not quite down to zeroes" kind of combing – see this example:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1755.png
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1755_zoom.png
My program does not detect ISO160 type combing - automatically .

 

[Mike Robbins]

... This is quite interesting. I have been trying to find a correlation since day one. The 50mm f/1.4 lens is certainly a convincing correlation - but on the other hand if you shoot with it all the time it is no correlation at all. ...

Well, I don't have a hard figure as to how many of my shots were taken with each lens, but the majority of them were taken with the 24-105 L. If I had to guess, I'd say that maybe 10-15% of them are with the 50 1.4. Also, it was very rare for me to shoot above ISO 800 on the Rebel XT due to noise, so perhaps that's why there's not so much evidance of the combing.

I have a suspicion that the combing might be to interference from the AF motor / circuit. I have noticed some horizontal banding occasionally when shooting with the 50 1.4 at high ISO in AI Focus mode on my 5D. After I noticed the problem I have not shot with AI Focus very much if at all; the timing of this realization corresponds with the combed files found.

I will see if I can reproduce this behaviour, and run the scan again. Also, I'd be more than happy to send along some of my photos, provided you do not mind looking at shots from a family birthday party.

 

[Richard McNeil]

I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT). It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.

I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.

I just checked my "old" files and the first picture I took with the 5d used a 50mm f1.8 lens shot at iso 100, f3.2 and in sRGB color space. IT HAS COMBING ARTIFACTS!

I don't think the color space has anything to do with it but I used Adobe color space for all other photos that I tested.

 

[Mike Robbins]

I performed a quick and somewhat informal test this morning to see if I could reproduce the combing that was found in my earlier scan. I did two series of shots using my 5D, shooting first with my 25-105 mm f/4 and again with my 50 mm f/1.4 lens. Both series of shots were taken at a variety of different ISO values (800-1600) in batches of approximately ten shots per ISO. I shot all of the photos in Av mode (f/4 for the 24-105 and f/2 for the 50) with evaluative metering, all handheld. For the shots taken in AI Focus mode I was sure to move forward or back slightly leading up to the shot to force the focus motors to track; all other shots were taken in one shot mode. The resulting photos are all garbage, but the results may be of some interest (we'll call this first batch Test A):

Test A:

5D, 25-105 @ 50 mm (IS disabled):
ISO 800: 0/10 combed
ISO 1000: 0/10
ISO 1250: 0/10
ISO 1600: 0/10
ISO 1600, AI Focus: 0/11

5D, 50 1.4:
ISO 1600, AI Focus: 0/10
ISO 1600: 3/10
ISO 1250: 2/10
ISO 1000: 1/10
ISO 800: 3/10

I was encouraged to see that I was able to reproduce the problem, though I was surprised to see that there were no combed photos taken with the 50 1.4 in AI Focus mode. I then performed a second set of test shots (Test B), all at ISO 1600 to look more closely at the effect of the focus mode (Av mode again, this time f/1.8; evaluative metering). For the first three batches, I was moving slightly just before I took them to force the micro ring USM to track for focus. The final batch of shots was taken on a tripod, in single shot mode with the timer and MLU. Results are as follows:

Test B:

5D, 50 1.4:
ISO 1600, Single Shot: 11/15
ISO 1600, AI Focus: 12/15
ISO 1600, AI Servo: 8/15
ISO 1600, Single Shot w. Tripod: 14/15!

I am now quite convinced that the combing is a result of the lens / body combination, though I am no longer so sure that it has anything to do with the focus mode. If I get a chance later on, I will check to see if I can produce this combing with my other lenses.

 

[Ray West]

So, if a raw file is not always a raw file, and iso setting is not as it should be, is this not the same sort of situation as with compact flash capacity not being as on the label. Class action coming, anyone??

Is it similar with Nikon, whoever?


Best wishes,

Ray

 

[Peter Ruevski]

I performed a quick and somewhat informal test...

Great job Mike you are definitely on to something!

... Test A:....
Test B:...
...
I am now quite convinced that the combing is a result of the lens / body combination, though I am no longer so sure that it has anything to do with the focus mode. If I get a chance later on, I will check to see if I can produce this combing with my other lenses.

So it seems. Time to go back through my files and look for correlation between lens and combing. I downloaded EXIFTool - looks like the perfect tool for the job... now on to writing more batch files to extract and analyze EXIF information of combed files...

For now I can say for sure that I have combed files with the EF 50mm f/1.8 and the EF 100mm f/2 USM.
I also own the EF 24-85mm f/3.5-4.5 USM and the (kit lens; horror! ;-) EF-S 18-55mm f/3.5-5.6 and do not see combed pictures from them - however I only have access to the few photos that are on the web site right now. Will have to check my whole collection later...

 

[Peter Ruevski]

So, if a raw file is not always a raw file, and iso setting is not as it should be, is this not the same sort of situation as with compact flash capacity not being as on the label. Class action coming, anyone??

Is it similar with Nikon, whoever?

Oh it is much worse with Nikon - most NEF (Nikon RAW) files under the sun are not raw at all - first noise reduction is applied, then the file compression itself is not lossless so practically every NEF is combed.

But class action? Oh, come on, the camera works fine as far as the end result - the picture - is concerned. Which is not to say there isn't a small screw up somewhere - but I am willing to sympathize with the Canon engineers - being an engineer myself - these cameras are a complex piece of hardware/firmware! But we will try to get to the bottom of this anyway - wont we? ;-)

 

[Ray West]

Hi Peter,

Its not an engineering issue, all engineers are entirely honest, - it is a marketing issue. I do not want to labour the point, I was surprised at the recent cf (and epson ink) actions, but to me this is much the same - something being said that is accepted by most folk, but on examination is a lie. It happens all the time with most things, from a hamburger to a merc, but we somehow seem to accept it.

We will see what happens...

Best wishes,

Ray

 

[Peter Ruevski]

Fun with ExifTool, mystery partially solved

Hi all,

So after Mike’s revelation that it might be the lens I just had to go back and check my combed photos. Being a geek - and lazy - I decided that it is too much work to do it by hand so I downloaded ExifTool by Phil Harvey (http://www.sno.phy.queensu.ca/~phil/exiftool/). What can I say… it is the mother of all EXIF utilities; the BFG of meta-data extraction; the Pan Galactic Gargle Blaster of EXIF tools;

the…
One Tool to rule them all, One Tool to find them,
One Tool to bring them all and in the darkness bind them​

Did I mention I was impressed? This thing will suck the last bit of metadata out of whatever image file you throw at it!

To make a long story short - after installing Perl; figuring out which of the endless ExifTool options will dump the data (from 775 raw files 215 of wich combed) the way I wanted it; some more batch files and some tinkering in Excel I can say this:

The combing is caused either by the lens, or a combination of the lens and a small aperture. ISO speed, shutter and exposure compensation do not matter it seems. My EF50mm f/1.8 and EF100mm f/2 USM "comb" quite often. My other lenses never do.

I have updated the web page with the full statistics and detailed conclusions as well as some of Mike’s combed 5D photos with the EF50mm f/1.4 USM. Look at the bottom.
http://www.pages.drexel.edu/~par24/rawhistogram/rawhistogram.html

Best regards,
Peter

 

[Ray West]

Hi Peter,

Roughly speaking, am I right in thinking that the combing is related to the amount of light falling on the sensor, i.e. more light, tendency to comb? Are the variations due to sensor temperature? Can you get combing with non-canon lenses, say?

Is it worth persuing it further? I think it may take some time to get the reason/cause behind it. I think it may possibly be due to time taken for ad conversions, if the ad works on the capacitor charge-discharge/time principle. i.e. it takes longer to collect the value for a bright image, so we'll do every other one, then, if time left over, go back and fill in the gaps...... Have you tried it with a digic 1 uP based camera?

Wish I had time,

Best wishes,

Ray

 

[Peter Ruevski]

All very good questions. For all of them the correct answer of course is "I do not know" but the whole point is to build a theory so...

Roughly speaking, am I right in thinking that the combing is related to the amount of light falling on the sensor, i.e. more light, tendency to comb?

I doubt it. I have photos (intentionally done tests) almost completely blown to white (sensor saturated, all pixels at 4095) with a small dark area which show perfectly smooth combing-free histograms.

Are the variations due to sensor temperature?

Possible; however I have a counter example of sorts - IMG_1829 and IMG_1830 on the site. Shot within a minute of each other - one combed, one not. The camera did not move. The only difference - aperture and shutter speed. It is unlikely the sensor temperature could have changed significantly.

Can you get combing with non-canon lenses, say?

I do not know. I only have five photos with a Sigma 170-500 - none of them combed. Now the lens is back to Sigma to be "rechipped" so the aperture works with the 30D. Also a few photos with no lens (a pinhole lens cap) - no combing. But this is a tiny sample.

Is it worth persuing it further? I think it may take some time to get the reason/cause behind it.

Not until there is more data to look at. But that depends on people looking at their collections of raw files. However the issue is an obscure "geeky" one I must confess

I think it may possibly be due to time taken for ad conversions, if the ad works on the capacitor charge-discharge/time principle. i.e. it takes longer to collect the value for a bright image, so we'll do every other one, then, if time left over, go back and fill in the gaps......

Hm-m-m-m - only Canon knows... if the rumors that the ADCs are "one per pixel" are correct (BTW it sounds unlikely to me) then they should be small ones (transistor count wise) - so Sigma/Delta is probable. OTOH if there are "just a few" of them - they must be very fast ones => "proper" parallel conversion ADCs. Either way - why does my EF24-85mm f/3.5-4.5 USM never comb?

Have you tried it with a digic 1 uP based camera?

No, I do not have one.

Regards,
Peter

 

[Daniel_Hyams]

On a directory of 97 30D files, here is a quick result....

none of the "regular ISO" files were combed. The only ones that were shots that I took at ISO 250, and some were detected by the code as being combed, and some were "lazy combed" as described on the web page.

Lenses used in the shoot were the Tokina 12-24, Tamron 28-75, and Canon 70-200 f/4L.

 

[Alan T. Price]

It's not surprising to me that there is combing. In fact, it would be more surprising if there was no combing.

The sensor captures light at each pixel only at its native ISO sensitivity and converts the amount of light to an analog voltage. This is then converted to digital in a 12-bit number, allowing for up to 4096 discrete levels.

At higher ISOs the ISO 100 data needs to be scaled up in some way, but there are three ways this can be done:

1. Do a calculation on the 12-bit data and store that new value.

e.g. for a 1-stop increase the values are doubled. This would result in severe combing because only even values can be stored. [any integer multiplied by 2 is even].

If larger increases are applied (e.g. 5 stops) then only every fifth value can be used in the final output.

Clearly there would be so many gaps in the data that this method cannot be the right one.

2. Do a voltage multiplication / amplification in the analog system. This would give no combing because the analog data is not quantised (?) into integers. Simple voltage doubling circuits could be applied for each whole-stop increase in ISO. For that matter the 1/3 and 1/2 stop amplifiers could be used too.

3. do a combination. e.g. Double the analog signal as often as required for whole-stop increases in ISO, but use a digital calculation for the fractions of a stop. With a partial-stop digital multiplication, there would be a gap every few levels as a non-integer result is rounded up or down to fit in the appropriate integer bin.

I'd probably go with #3 for the 30D, to match your observations. Perhaps the bigger cameras have room for more or faster analog circuits to produce less combing, but I haven't seen the observed results and can't make a conclusion.

Quite clearly, this whole experience shows the benefit of having more levels to play with at 12-bit than at 8-bit, as by the time we dumb the data down to 8-bit for printing there is no combing evident.



[Edit:] Here's another possibility: Maybe the whole thing is done digitally but rather than based on a 12-bit initial value, it is based on a 16-bit initial value, as if a 16-bit analogue to digital converter is used. Consider this as being 2^16 quarter-integer levels instead of 2^12 whole-integer values. After scaling, the new 16-bit values are converted to 12-bit values for use in the raw data file. This could be done simply by taking any four adjacent levels at 16-bit depth and puting them into a single level at 12-bit depth, for each pixel.

Third and half stop increases would still lead to combing.

A 16-bit value allows for a 4-stop increase over the conceptual 12-bit value, corresponding to a maximum sensitivity of ISO 1600 if the sensor has a native rating of ISO 100.

Just a thought.

 

[Asher Kelman]

Thanks Alan for revisiting this issue. Your comments appear valid, however, that is not my field. I'm interested in the responses. I think I'm going to learn some more.

Asher

 

[Peter Ruevski]

The page has moved

I had to move the page to a new URL:
http://ruevs2.tripod.com/rawhistogram/rawhistogram.html
The old one will stay for some more time but will soon disappear.


Hi Alan,

Your thoughts are quite reasonable and make sense, but (it seems) not all apply to the Canon DSLRs specifically.

It's not surprising to me that there is combing. In fact, it would be more surprising if there was no combing.

With raw files from all Canon DSLRs that I have looked at (300D, 350D, 10D, 20D, 30D, 5D) there is no combing most of the time. The only exceptions that I am aware of are intermediate ISOs on 30D and the phenomenon we were discussing in this thread, which happens at FULL ISOs.

The sensor captures light at each pixel only at its native ISO sensitivity and converts the amount of light to an analog voltage. This is then converted to digital in a 12-bit number, allowing for up to 4096 discrete levels.

Yes

At higher ISOs the ISO 100 data needs to be scaled up in some way, but there are three ways this can be done:

True

1. Do a calculation on the 12-bit data and store that new value.
e.g. for a 1-stop increase the values are doubled. This would result in severe combing because only even values can be stored. [any integer multiplied by 2 is even].

If larger increases are applied (e.g. 5 stops) then only every fifth value can be used in the final output.

Clearly there would be so many gaps in the data that this method cannot be the right one.

This is certainly an option but as you pointed out a very bad one - high ISO becomes meaningless. And it certainly is not the one actually used.
The one exception is 1/3 stop ISO increments on the 30D where the intermediate stops are achieved in this way (with the dark current taken into account) from the closest properly amplified full stop. I will sit down and write a page describing this in detail some day.

2. Do a voltage multiplication / amplification in the analog system. This would give no combing because the analog data is not quantised (?) into integers. Simple voltage doubling circuits could be applied for each whole-stop increase in ISO. For that matter the 1/3 and 1/2 stop amplifiers could be used too.

And this is the method that is apparently used. Including the 1/3 stop increments in the 5D and higher end cameras.

3. do a combination. e.g. Double the analog signal as often as required for whole-stop increases in ISO, but use a digital calculation for the fractions of a stop. With a partial-stop digital multiplication, there would be a gap every few levels as a non-integer result is rounded up or down to fit in the appropriate integer bin.

I'd probably go with #3 for the 30D, to match your observations. Perhaps the bigger cameras have room for more or faster analog circuits to produce less combing, but I haven't seen the observed results and can't make a conclusion.

You are correct.

Quite clearly, this whole experience shows the benefit of having more levels to play with at 12-bit than at 8-bit, as by the time we dumb the data down to 8-bit for printing there is no combing evident.

True

Here's another possibility: Maybe the whole thing is done digitally but rather than based on a 12-bit initial value, it is based on a 16-bit initial value, as if a 16-bit analogue to digital converter is used. Consider this as being 2^16 quarter-integer levels instead of 2^12 whole-integer values. After scaling, the new 16-bit values are converted to 12-bit values for use in the raw data file. This could be done simply by taking any four adjacent levels at 16-bit depth and puting them into a single level at 12-bit depth, for each pixel.

Third and half stop increases would still lead to combing.

A 16-bit value allows for a 4-stop increase over the conceptual 12-bit value, corresponding to a maximum sensitivity of ISO 1600 if the sensor has a native rating of ISO 100.

Just a thought.

Well, it is certainly a possibility but I consider it unlikely. 16bit ADCs are significantly more expensive (both price wise and to design low noise circuits for) than 12 bit ones and amplifiers are quite simple (cheap) by comparison. If Canon had gone to the trouble of putting 16 bit ADCs in they certainly would have advertised the fact loudly and kept the amplifiers to provide higher ISOs instead of wasting the 16bit data with scaling. In fact I think I have seen a direct statement from Cannon that they use 12bit ADCs, I can not find the reference now though.

To sum it up, the combing is not strange on its own - it is a fact of life as soon as "you" start mathematically manipulating the data. The mystery is the combing seen on different models (confirmed personally by me on 350D, 30D and 5D) in JUST A FEW photos taken at FULL STOP ISO settings when most other photos have perfectly smooth histograms.

 

[Alan T. Price]

The mystery is the combing seen on different models (confirmed personally by me on 350D, 30D and 5D) in JUST A FEW photos taken at FULL STOP ISO settings when most other photos have perfectly smooth histograms.

In your original post you showed the histograms for two full stop ISO 100 images but one had a third stop compensation, making it in effect a non-full-stop ISO 125. Are there other examples with full stop ISO and no EC that have combing ?

 

[Peter Ruevski]

In your original post you showed the histograms for two full stop ISO 100 images but one had a third stop compensation, making it in effect a non-full-stop ISO 125. Are there other examples with full stop ISO and no EC that have combing ?

This is a very good point and exposure compensation (EC) being handled as (1/3) ISO change was my first guess too. But now I am convinced that it is not the cause. Here are my reasons:

1. There are combed photos from my 30D that were shot with EC=0 I do not have any posted on the site though.

2. There are combed photos from my 30D that were shot in manual mode

3. All of the combed photos from my friends 350D are shot in manual mode

4. The 5D combed photos that Mike Robbins gave me are shot with EC=0

5. The combing in intermediate ISO photos is much mode severe (dense) than the "mysterious" combing. Here is an example at ISO 125:

The graph looks solid because of the dense combing. Here is a zoomed in section:

6. And last but most important - if you point the camera to a constantly illuminated scene on a tripod in Av or Tv mode and change the exposure compensation you will see that the aperture or shutter speed respectively will be adjusted accordingly. If the ISO were adjusted instead they would have stayed constant - otherwise an over/under exposure would occur.

Of cource it is seems that sometimes the camera is doing some very slight scaling for some reason, which causes the "mysterious" combing... but why?... and when?...

 

[Alan T. Price]

well Peter I can't think of anything else to help out, except one last desperate idea....

Could there be any EC applied to raw data in a program such as DPP, and could your reading of the raw data be influenced by that EC ? This would depend, I assume, on whether you are reading the data directly or by using a Canon-supplied routine that presents the data to your program.


Also, have you zoomed in on the combing close enough to determine how many values are being sacrificed and whether or not they are consistent in some way. e.g. is it every 4th value, or every other odd value, etc. There are twice as many green pixels as red or blue. Are both sets of green losing the same values ? May not help at all but this has now got to the point where it's just fascinating.

 

[John Sheehy]

Hi, remember me?

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

Yes and no. There are other ways of obtaining non-optimum ISOs.

I started another thread about the 30D intermediate ISOs after you stopped visiting DPReview (or you just missed that thread), but someone gave me 5D blackframe files to look at, and I found a problem with the noise levels, that made me recant my earlier declaration that 5D had amplified, real, intermediate ISOs. Amplified they seem to be, but they are still not as real as the 100/200/.../1600 series. The histograms look good at all ISOs (except that every 15.5th value is unrepresented, in the samples I received), but the actual noise values for the 5D did no smoothly graduate from 100 to 1600; they are, in fact, separately amplified before going into the A2D converter, and therefore have more blackframe noise than they should at the intermediate ISOs.

The pattern is this:

100 2.0
125 2.4
160 2.9

200 2.0
250 2.5
320 3.0

400 2.2
500 2.7
640 3.3

800 3.0
1000 3.7
1250 4.6

1600 4.6

Horizontal banding noise is directly proportional to the numbers within each triplet (but not the whole list); vertical banding noise is not (but vertical banding noise is rarely visible). These are the numbers that determine dynamic range; 160 has nearly as much blackframe noise as ISO 800.

Of course, blackframe noise is the "noise floor" that affects DR and deep shadows mostly, and the noise in the highlights and the midtones (and the darker midtones at higher ISOs) is photon shot noise, which is affected by photon collection in the sensor wells and has nothing to do with ISO settings per se (except as they affect metering, and therefore, exposure).

Conclusion: Don't use these extra 5D ISOs; underexpose at the lower base ISO, and get the extra headroom (or expose all the way to the right and get a better capture).

 

[John Sheehy]

Just a thought.

Lots of interesting ideas there, but an important point missed is that what we call "real" ISOs on the 30D are heavily optimized - ISO 1600 only has 2.26x the readout (blackframe) noise of ISO 100, even though it amplifies 16x as much. ISO 400 only has 1.18x the readout noise as ISO 100, even though it amplifies 4x as much. 1600 has 1.19x as much as 400, with 4x the amplification. The higher ISOs are highly optimized to avoid readout noise proportional to amplification. When the 30D decides to create its ISO 125 by digitizing a shot metered for 125 and amplified for 100, it multiplies the readout noise by exactly the amount of the "amplification", which is totally non-optimal in the trend set by the 100/200/.../1600 group. In fact, the readout noise of 125 on the 30D is about the same as the readout noise at 640, because 640 has its noise *divided* from the amount at ISO 800.

 

[John Sheehy]

I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Did you try zooming into the histogram so that it is only a couple hundred RAW values wide, and widen it on the desktop?

 

[Asher Kelman]

Hi, remember me?



Yes and no. There are other ways of obtaining non-optimum ISOs.

I started another thread about the 30D intermediate ISOs after you stopped visiting DPReview (or you just missed that thread), but someone gave me 5D blackframe files to look at, and I found a problem with the noise levels, that made me recant my earlier declaration that 5D had amplified, real, intermediate ISOs. Amplified they seem to be, but they are still not as real as the 100/200/.../1600 series. The histograms look good at all ISOs (except that every 15.5th value is unrepresented, in the samples I received), but the actual noise values for the 5D did no smoothly graduate from 100 to 1600; they are, in fact, separately amplified before going into the A2D converter, and therefore have more blackframe noise than they should at the intermediate ISOs.

The pattern is this:

100 2.0
125 2.4
160 2.9

200 2.0
250 2.5
320 3.0

400 2.2
500 2.7
640 3.3

800 3.0
1000 3.7
1250 4.6

1600 4.6

Horizontal banding noise is directly proportional to the numbers within each triplet (but not the whole list); vertical banding noise is not (but vertical banding noise is rarely visible). These are the numbers that determine dynamic range; 160 has nearly as much blackframe noise as ISO 800.

Of course, blackframe noise is the "noise floor" that affects DR and deep shadows mostly, and the noise in the highlights and the midtones (and the darker midtones at higher ISOs) is photon shot noise, which is affected by photon collection in the sensor wells and has nothing to do with ISO settings per se (except as they affect metering, and therefore, exposure).

Conclusion: Don't use these extra 5D ISOs; underexpose at the lower base ISO, and get the extra headroom (or expose all the way to the right and get a better capture).

In the noise measurements shown, could you explain how these are arrived at? You may well have mentioned that in a previous post. Is there a PC/Mac utility to measure this?

The points you make are very important. I have always thought I was doing myself good to shoot at 160 rather htan 200 ISO!

I'd like to see some examples of the type of image where either the noise would have an effect on the printed image or where the combing of every 15.5th value in the histogram might degrade the final image too.. (BTW, What do you use to demonstrate this?).

Asher

 

[John Sheehy]

In the noise measurements shown, could you explain how these are arrived at? You may well have mentioned that in a previous post. Is there a PC/Mac utility to measure this?

That was my first post here, but you probably recognize my name from the old RG forums. The PC program IRIS http://www.astrosurf.com/buil can load the RAW blackframe, and then issuing the "stat" command in the console window gives the noise ("sigma").

The points you make are very important. I have always thought I was doing myself good to shoot at 160 rather than 200 ISO!

If the scene is high-key, then you probably are, as shot noise will be the main noise, and there will be less of it metering for 160. If you are concerned with DR, though, 160 is not the place to do it.

What I am saying is that there isn't any monolithic "noise" in an image; there are two basic groups of independent noise; noise that is related to the signal (such as shot noise) and noise that is a blanket over the entire image (readout, or blackframe noise). The former is directly related to real, analog exposure. The latter is dependent upon the ISO setting, also.

I'd like to see some examples of the type of image where either the noise would have an effect on the printed image or where the combing of every 15.5th value in the histogram might degrade the final image too.. (BTW, What do you use to demonstrate this?).

Seeing combing at that level will be very difficult. I'm not saying it's a big problem. In the camera I received the files form, RAW values 107, 123, 138, and 154 were grossly under-represented (the few that were there may be the result of interpolation for mapped-out pixels). Black is 128 in recent Canon cameras. It would take a very slow gradient in the deep shadows at a very low ISO to see the effect. The combing itself is not a big problem, as there is too much noise in all cameras at all ISOs for minor posterization to jump out at you. The posterization is a symptom, though, of some unnecessary math going on; math that should not be done at all IMO; I believe that digitized number should be stated literally in the RAW data, and that all scaling should be done by the converters, by stating figures in RAW metadata, or by external knowledge. That's how the 10D differed it's 3200 from 1600; the RAW levels were proportional to absolute exposure; IOW, turning the ISO from 1600 to 3200 on the 10D in manual mode did not affect the RAW levels; the converters knew to scale the 3200 data by +1 stop more than 1600 data in the conversion, and it worked. Why scale anything, and clip away highlights, or turn two RAW values irreversibly into one?

As far as the increased noise is concerned at ISOs like 160, try shooting the same scene with deep shadows at 160 and 200 (with the same EC), and bring the shadows up and see what they look like. I'm sure you'll see bolder noise and banding in the 160.

 

[John Sheehy]

Imagine a simple monochrome image, which only has pixels with five shades of gray 0, 1, 2, 3 and 4. If you multiply every pixel by 2 now your image will be "twice as bright" (in fact it will have twice the dynamic range).

The noise floor doubles in value, too, so there is no gain in DR.

 

[John Sheehy]

I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

You might want to try blackframes (short exposures with the lens cap on) at ISO 1600 and 3200 on the 5d, load them into IRIS, run the command console and type "stat" for each. If the 3200 "sigma" value is 2x the 1600 value, then the camera is probably using a second, low gain amplifier following the optimized 100/200/.../1600 amplifier system. Perhaps it is applying 1.25x gain for the 125/250/... group, 1.6x for the 160/320/... group, and 2x for 3200.

 

[John Sheehy]

Hi Richard, The ISO 3200 lacking combing does not surprise me – it is well established that ISO 3200 a.k.a. "H" is simply ISO 1600 pushed (i.e. underexposed) one stop and a flag set in the file telling the raw conversion software to do +1 stop exposure adjustment. As far as the actual raw data is concerned ISO3200 is exactly the same as ISO1600 - that is why people say it is not useful when shooting raw. When shooting JPEG the firmware does the pushing so it is useful.

For what camera? The 10D does this, and I assume the 300D hack. The 20D does not do it that way; the 20D actually doubles all the RAW values for 3200, clipping away a stop of highlights that would be saved if it were done the way you suggest, which, I believe, is the way it should be done, especially when you consider the fact that many scenes that call for exposure indices like 3200 have light sources within the scenes, and need the expanded headroom. Canon seems to be more interested in maintaining a consistent relationship between RAW levels and relative exposure levels, to the detriment of the quality or usefulness of the data. It would explain every quirk we're discussing here - the light combing at ISOs 100/200 etc could just be the camera saying "I think the amplification was only 96% of what it should have been, so I will scale the digitized data", rather than just leaving it alone.

 

[John Sheehy]

Your impression is correct - 100%. The combing is different though - e.g. ISO125 is “good old down to zero” combing as the examples I posted earlier. ISO160 however is a "half hearted not quite down to zeroes" kind of combing

The 160 group's artifacts in the histogram are the high spikes, not the low ones. The high spikes are two digitized values becoming one.

 

[Peter Ruevski]

Hi, remember me?

Hi John,

Of course I remember you. Your early posts on the intermediate ISOs at dpreview are what started me "digging in raw data"

Unfortunately lately I have been quite busy and have visited this forum only occasionally and dpreview not at all. So I have missed your thread, but I will go back and look for it when I find some time.

As usual you have made some very good points and I will consider them carefully and write a single reply - hopefully sometime this week, maybe next.

Best regards,
Peter

 

[Adrian Warren]

As a matter of interest, are the shots exhibiting combing in focus? Does the combing appear on entirely OOF shots?

I'm wondering about Airy Discs:
http://en.wikipedia.org/wiki/Airy_disc