{"id":847,"date":"2014-06-18T08:10:08","date_gmt":"2014-06-18T15:10:08","guid":{"rendered":"http:\/\/jnack.com\/blog\/?p=847"},"modified":"2014-06-18T01:51:42","modified_gmt":"2014-06-18T08:51:42","slug":"heresy-should-i-just-wholesale-blow-away-my-raw-files","status":"publish","type":"post","link":"http:\/\/jnack.com\/blog\/2014\/06\/18\/heresy-should-i-just-wholesale-blow-away-my-raw-files\/","title":{"rendered":"Heresy: Should I just wholesale blow away my raw files?"},"content":{"rendered":"<p>\u2026and replace them with lossy DNG proxies? Would I <em>ever<\/em> see a visual difference?<\/p>\n<p>A) Yes. B) No.<\/p>\n<p>So, a little background:<\/p>\n<ul>\n<li>Lightroom &amp; the free DNG Converter added the ability to apply lossy compression when creating DNG files.<\/li>\n<li>When you apply this compression, your raw data get mapped from a higher bit depth (10-14 bits per channel) to 8 bit.<\/li>\n<li>That sounds horrible (&#8220;what about my highlight &amp; shadow data?!\u201d), but the mapping (quantization) is done cleverly, before a perceptual curve is applied. (See nerdy footnote if interested.)<\/li>\n<li>You retain the same white balance flexibility you always had.<\/li>\n<li>You save a lot of disk space\u2014between 40% &amp; 70% in my experience. (You can also elect to save at a reduced resolution, in which case you\u2019ll obviously save a lot more.)<\/li>\n<\/ul>\n<p>What I\u2019ve always wondered\u2014but somehow never got around to testing\u2014is whether I\u2019d be able to see <em>any<\/em> visual differences between original &amp; proxy images. In short, no.<\/p>\n<p>Here\u2019s how I tested:<\/p>\n<ul>\n<li>I started with a typical photo taken by my wife\u2014one with really under- and over-lit areas.<\/li>\n<li>I imported the original file into Lightroom, then exported a copy as DNG with lossy compression, then imported the copy back into LR (so that the original &amp; proxy would sit side-by-side).<\/li>\n<li>Just to stress-test, I cranked up the Shadows to +100 and cranked down Highlights to -100.<\/li>\n<li>Then to stress things further, I used a brush to open up the shadows by another full stop.<\/li>\n<li>I copied &amp; pasted settings from the original to the proxy.<\/li>\n<li>Failing to notice any visual differences at all, finally I opened the original &amp; proxy versions as layers in Photoshop. I set the blending mode of the top layer to Difference in order to highlight any variation between the two versions.<\/li>\n<li>Having failed to see any difference even then (i.e. the result of Difference appeared to be pure black\u2014i.e., identical pixels), I applied an Exposure adjustment layer and\u2014just for yuks\u2014cranked it up 14 stops.<\/li>\n<\/ul>\n<div><img decoding=\"async\" loading=\"lazy\" title=\"ProxyComparison_sm.jpg\" src=\"http:\/\/jnack.com\/blog\/wp-content\/uploads\/2014\/06\/ProxyComparison_sm2.jpg\" alt=\"ProxyComparison sm\" width=\"523\" height=\"600\" border=\"0\" \/><\/div>\n<div>\u00a0<\/div>\n<div>You can <a href=\"https:\/\/www.dropbox.com\/s\/bjmajywynhkud4k\/Pool_comparison.tif\">download the layered TIFF<\/a>\u00a0if you really want to compare things for yourself.<\/p>\n<p>I repeated the experiment with other images, including some with subtle gradients (e.g. a moonrise at sunset). The results were the same: unless I was being pretty pathological, I couldn\u2019t detect any visual differences at all.<\/p>\n<p>I did find one case where I could see a difference between the lossy &amp; lossless versions: My colleague Ronald Wotzlaw shot a picture of the moon, and if I opened up the exposure by more than 4 stops, I could see a difference (<a href=\"https:\/\/www.dropbox.com\/s\/91h6mthvmfomnja\/MoonComparison.jpg\">screenshot<\/a>). For +4 stops or less, I couldn\u2019t see any difference. Here\u2019s the <a href=\"https:\/\/www.dropbox.com\/s\/wqnu3iza7xqpawv\/Moon.nef\">original NEF<\/a> &amp; the DNG copies (<a href=\"https:\/\/www.dropbox.com\/s\/idwbbt4jx1y32rn\/Moon.dng\">lossless<\/a>, <a href=\"https:\/\/www.dropbox.com\/s\/7mvecgtiwlgsp9o\/Moon_lossy.dng\">lossy<\/a>) if you\u2019d like to try the experiment yourself.<\/p>\n<p>No doubt a lot of photographers will tune out these findings: \u201cRaw is raw, lossless is lossless, the end.\u201d Fine, though I\u2019m bugged by some photogs&#8217; fetishistic, gear-porn qualities (the kind of guys who insist on getting a giant lens &amp;\u00a0an offsetting full-frame camera) &amp; old-wives\u2019 mentalities (\u201cYou can\u2019t reformat your memory card with your computer: this one time, in 2003, my buddy tried it and it made his house burn down\u2026\u201d).<\/p>\n<p>So, to each his or her own. As for me, I\u2019m really, really encouraged by these findings, and I plan to start batch-converting my DNGs to be \u201clossy\u201d (a great misnomer, it seems).<\/p>\n<p>&#8212;&#8212;<\/p>\n<p>Nerdy footnote: Zalman Stern spent many years building Camera Raw &amp; now works with me on Google Photos. He\u2019s added a bit more detail about how things work:<\/p>\n<p><em>&#8220;Downsampling&#8221; is reducing the number of pixels. Reducing the bit-depth is &#8220;quantizing.&#8221; The quantization is done in a perceptual space, which results in less visible loss than doing quantization in a linear space. Raw data of the sensor is linear where the data going into a JPEG has a perceptual curve applied. (&#8220;Gamma&#8221; and sRGB tone curves are examples of the general thing around perceptual curves.)<\/em><\/p>\n<p><em>Dynamic range should be preserved and some small amount of quantization error is introduced. (Spatial compression artifacts, as in normal JPEG, are a different form of quantization error. That happens with proxies too.) Quantization error is interesting in that if it is done without patterning, it takes a very large amount of it to be visible.<\/em><\/p>\n<p><em>The place you&#8217;d look for errors with lossy raw technology are things like noise in the shadows and\u00a0patterning via color casts in highlights after a correction. That is the quantization error gets magnified and somehow ends up happening differently for different colors.<\/em><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>\u2026and replace them with lossy DNG proxies? Would I ever see a visual difference? A) Yes. B) No. So, a little background: Lightroom &amp; the free DNG Converter added the ability to apply lossy compression when creating DNG files. When you apply this compression, your raw data get mapped from a higher bit depth (10-14 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/847"}],"collection":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/comments?post=847"}],"version-history":[{"count":3,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/847\/revisions"}],"predecessor-version":[{"id":851,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/posts\/847\/revisions\/851"}],"wp:attachment":[{"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/media?parent=847"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/categories?post=847"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/jnack.com\/blog\/wp-json\/wp\/v2\/tags?post=847"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}