…and replace them with lossy DNG proxies? Would I ever see a visual difference?
A) Yes. B) No.
So, a little background:
- Lightroom & the free DNG Converter added the ability to apply lossy compression when creating DNG files.
- When you apply this compression, your raw data get mapped from a higher bit depth (10-14 bits per channel) to 8 bit.
- That sounds horrible (“what about my highlight & shadow data?!”), but the mapping (quantization) is done cleverly, before a perceptual curve is applied. (See nerdy footnote if interested.)
- You retain the same white balance flexibility you always had.
- You save a lot of disk space—between 40% & 70% in my experience. (You can also elect to save at a reduced resolution, in which case you’ll obviously save a lot more.)
What I’ve always wondered—but somehow never got around to testing—is whether I’d be able to see any visual differences between original & proxy images. In short, no.
Here’s how I tested:
- I started with a typical photo taken by my wife—one with really under- and over-lit areas.
- I imported the original file into Lightroom, then exported a copy as DNG with lossy compression, then imported the copy back into LR (so that the original & proxy would sit side-by-side).
- Just to stress-test, I cranked up the Shadows to +100 and cranked down Highlights to -100.
- Then to stress things further, I used a brush to open up the shadows by another full stop.
- I copied & pasted settings from the original to the proxy.
- Failing to notice any visual differences at all, finally I opened the original & proxy versions as layers in Photoshop. I set the blending mode of the top layer to Difference in order to highlight any variation between the two versions.
- Having failed to see any difference even then (i.e. the result of Difference appeared to be pure black—i.e., identical pixels), I applied an Exposure adjustment layer and—just for yuks—cranked it up 14 stops.
I repeated the experiment with other images, including some with subtle gradients (e.g. a moonrise at sunset). The results were the same: unless I was being pretty pathological, I couldn’t detect any visual differences at all.
I did find one case where I could see a difference between the lossy & lossless versions: My colleague Ronald Wotzlaw shot a picture of the moon, and if I opened up the exposure by more than 4 stops, I could see a difference (screenshot). For +4 stops or less, I couldn’t see any difference. Here’s the original NEF & the DNG copies (lossless, lossy) if you’d like to try the experiment yourself.
No doubt a lot of photographers will tune out these findings: “Raw is raw, lossless is lossless, the end.” Fine, though I’m bugged by some photogs’ fetishistic, gear-porn qualities (the kind of guys who insist on getting a giant lens & an offsetting full-frame camera) & old-wives’ mentalities (“You can’t reformat your memory card with your computer: this one time, in 2003, my buddy tried it and it made his house burn down…”).
So, to each his or her own. As for me, I’m really, really encouraged by these findings, and I plan to start batch-converting my DNGs to be “lossy” (a great misnomer, it seems).
——
Nerdy footnote: Zalman Stern spent many years building Camera Raw & now works with me on Google Photos. He’s added a bit more detail about how things work:
“Downsampling” is reducing the number of pixels. Reducing the bit-depth is “quantizing.” The quantization is done in a perceptual space, which results in less visible loss than doing quantization in a linear space. Raw data of the sensor is linear where the data going into a JPEG has a perceptual curve applied. (“Gamma” and sRGB tone curves are examples of the general thing around perceptual curves.)
Dynamic range should be preserved and some small amount of quantization error is introduced. (Spatial compression artifacts, as in normal JPEG, are a different form of quantization error. That happens with proxies too.) Quantization error is interesting in that if it is done without patterning, it takes a very large amount of it to be visible.
The place you’d look for errors with lossy raw technology are things like noise in the shadows and patterning via color casts in highlights after a correction. That is the quantization error gets magnified and somehow ends up happening differently for different colors.
Great info! Can lossless compression be accomplished when importing using Adobe Bridge? Bridge Photo Downloader does not expose the options using the same terminology. They have “Compressed (lossless)” checkbox, and a Convert to Linear image radio button.
I’ve done this myself to save disk space, and also to speed procesing. I find Lossy DNGs to load a bit quicker in Develop than the full X-Trans RAF files from my Fujis.
There’s one big drawback though, which I’ve learned to live with. Only Lightroom and Photoshop can actually open Lossy DNGs. I don’t get QuickLook previews on Mac OS X and they don’t display in, for instance, Picasa.
The latter case is particularly weird: you get thumbnails when importing Lossy DNGs, and when you open them for a split-second you see the embedded JPEG preview, but after that the image is blank. Presumably Picasa tries to parse the DNG’s raw data and then encounters data it can’t handle.
I had hoped that other apps would pick up the latest DNG spec, but it’s been two years and to my knowledge neither Aperture nor Picasa, and likely other raw processors, can handle Lossy DNGs.
Regrettable!
[Given that I’m now the PM responsible for Picasa (among many other things), I guess I should get that sorted out. 🙂 —J.]
After reading this blog, I took several files from several different jobs and tried exporting from raw CR2, DNG, and DNG lossy files. With careful examination, I did see a small degradation in the DNG lossy export. May not be significant depending on the end use and based upon the subject matter. The product shots that showed the most degradation were very detailed while executive portraits were virtually identical.
Thanks for the tip John.
Thanks for the explanation.
I tend to say: make that files DNGs lossy which are just keepers. The picks can definitly stay as “normal/original” DNGs. No need to convert all.
But never convert your files to jpg. Lossy DNGs are much better for a good quality : filesize ratio.
Martin
A follow-up: OS X Yosemite now supports Lossy DNGs at the system level; they can be opened in Quick Look and Aperture 3.6. So the interoperability disadvantage is diminishing.
I was looking into this to advise people who are not daving raw at all, due to space. So the question is not “shall I use lossy instead of normal dng?” But rather “can I use lossy dng instead of keeping only jpeg?”
Better yet, make in-camera jpeg the “advanced” feature on high-end dslr’s so people don’t keep using jpeg out of habit, and disable jpeg as an option if the lens cost more than a thousand dollars. 🙂