"The DNG format was supposed to be the future, an open standard for RAW files that every manufacturer could use," writes Digital Photo Pro’s Dave Willis. "Here’s a look at how the revolution has panned out." Dave talks with my boss Kevin Connor about the problem that gave rise to DNG:
"Our philosophy on this from the beginning, sort of my personal belief," continues Connor, "is that eventually the proprietary system is just going to break. When we came out with the first camera RAW plug-in, we were supporting around 25 cameras. We’re now supporting more than 175 cameras—in other words, more than 175 different file formats. And when you’re talking about images, people don’t want to keep those images for just five or 10 years. Professional photographers want to know those images will be fine for 50 years—100 years—from now. If you think about the rate of new-camera introductions, how many new file formats will there be? A hundred thousand? It just seems that it’s going to reach a point when it becomes unmanageable."
It’s true that we haven’t yet seen big camera vendors like Canon and Nikon adopt DNG, though maybe we’ll see more progress now that DNG has been submitted to the ISO as a vendor-independent standard. In any case, the format is providing real-world benefits today:
- Converting to DNG saves disk space and eliminates the need to use separate sidecar files for raw settings. (I knocked 1.5GB off the 7GB of photos from our wedding photographer.)
- Because of these benefits, customer feedback indicates that 40% of Lightroom users are converting to DNG on import. (It’s a one-click set-and-forget option that’s also available in Adobe Bridge CS3.)
- DNG lets Adobe support new cameras in older versions of Camera Raw without having to constantly revise and test those versions. Photographers and use the free DNG Converter (Win | Mac) to process their proprietary raw images to DNG. The upshot is that we can spend our time building good new functionality instead of updating old software.
[Via]
[Update: I neglected to mention that yes, Adobe will be providing a DNG-viewing codec for Windows Vista, making it possible to view DNG files right within the operating system. Expect this free download to be posted soon. –J.]
Is there a .dng codec for Windows Vista Photo Gallery, hopefully available in x64 flavor too?
[Yes, we’ll be providing one very soon. –J.]
Nikon has a RAW codec for it that photo gallery auto prompts for. I really would like to see this support, or I’ll never support it.
Interesting…
From Adobe’s point of view, I wonder why keeping LR/ACR up to date is considered a task soon to be unmanagable, while keeping the DNG convertor up to date is OK’ although it is a task of the same magnitude…
[Well, it’s certainly a bunch of work either way, but revving the converter means having to rev fewer pieces of software. We don’t, for example, have to go back and keep updating the CS1 or CS2 code bases to support new proprietary raw formats, then testing all their editing functions with each format. Rather we rev the converter & automatically get support in other DNG-compatible tools. –J.]
Is this a new action? Or publication of an old one?
I learned in March 2007 that Adobe had given permission for the ISO group (TC42 WG18) revising ISO 12234-2 (TIFF/EP, on which DNG was based) “to incorporate modifications and developments made for DNG in the new standard”.
Does this latest news refer to that previous permission, or is it extra? What is the progress on TIFF/EP?
[I’ll ask Kevin and others for more details here. My understanding is that these discussions have been in the works for quite some time, but I hadn’t previously seen them discussed publicly. –J.]
It seems to me that offering a Windows codec for DNG would significantly increase that 40% number of people who routinely convert to DNG.
With more consumers using the format, it should be easier to convince the manufacturers to adopt the format.
Does that mean that the DNG SDK will be licensed under a clear and well known “open source” license like for example, to be consistent, under the 3-clause BSD license that is already used by the XMP SDK?
I already asked the question a while back to the Adobe Forums and got a less than receptive answer.
[I don’t know, but let me see what I can find out. –J.]
> Converting to DNG […] eliminates the need to use separate sidecar files for raw settings.
This, incidentally, strikes me as more negative than positive.
In a nutshell: Differential backups.
For example, my Nikon D300 generates ~12mb .NEF Raw files. I back those up once (to Amazon S3), and then they never change, so I never have to upload them again. Instead, I only have to re-upload the ~5 kb .xmp files should I change anything. A much more desirable state of affairs, and consequently I’d rather stick with the two file solution.
(Not for nothing, but I’d love if you guys found a way to split up the Lightroom catalog file for the same reason)
Still, standardization is a good thing and I’m glad this step is being taken.
John,
Is the plan (or should I say desired plan) that DNG would replace RAW inside the camera?
[Well, DNG is raw, and I’ve always considered the all-caps “RAW” usage to be misleading (suggesting that “RAW” is a particular file format, an acronym, or a proper name, instead of just a generic descriptor for a whole class of file formats.) In any case, a number of camera vendors already offer the ability to select DNG as one’s raw format of choice right within the camera, and others could do so as well. –J.]
I can certainly see the issue with all the different RAW formats. Is it possible that current RAW camera’s could receive a firmware update that would convert it to DNG?
[I’m no expert on what the circuitry inside various cams supports, but it certainly sounds like the kind of thing a firmware update could address. –J.]
Or does DNG require different camera hardware configuration. If different hardware is required then I think the hardware vendors like Canon and Nikon will need some sort of motivation to make an investment to change hardware. Does this make sense or am I completely lost?
[All good questions. Whether or not Canon and Nikon come to see enough benefit in an open standard that they start offering support inside their devices, DNG can & does offer real-world benefits today. –J.]
re rev’ing DNG:
and if the camera manufacturers get off their asses and have the camera raw formats be DNG, the will be no keeping up to consider.
Eric,
The trick to get XMP with DNG files, is to set them as read-only. I guess that John will correct me if this workaround brings some issues.
To the fellow talking about differential backups, it should be of interest that good online backup software should already do differential backups. I know my Iron Mountain Connected does.
This works well for large mail folders, which are constantly changing, and for DNG data files with small metadata changes as well.
I like the concept of DNG, and have considered converting my D50’s NEFs over. It seems more future proof. The possibility of standardizing on DNG would ultimately be good for the camera manufacturers, as they could replace all the different RAW outputs with the single standard DNG.
Until you can get Canon and Nikon to embrace DNG, I think you’ll have an uphill battle (at least as far as easing the burden on your development team). DNG is surely an advantage for the smaller players who have already adopted it, since it’s easier to use an existing standard than to develop one from scratch. But what do Canon and Nikon have to gain from replacing their raw formats with DNG?
You might be able to convince Canon to come on board for the sake of standardization, especially if enough users see the benefits of DNG and make their feelings known (something I haven’t seen). But Nikon is another matter. Unlike Canon, which gives away their raw converters, Nikon has a financial stake in maintaining their proprietary format and the profits from the sale of their advanced raw converter.
I’ll also agree with Eric about the sidecar files. I prefer to preserve my raw files exactly as the camera writes them, and save any modifications in separate files as ACR does. One of the things I dislike about Canon’s DPP is that it modifies raw files.
I’m still skeptical about DNG as a solution to the babel of raw files. It’s certainly an optimal solution for Adobe, which stands to benefit from both reduced development costs and ownership of a standard format. I’m not sure it’s quite as clearly beneficial for camera manufacturers and photographers.
[Photographers benefit by getting immediate support for their cameras, because there’s no need to wait for a software vendor to get the new model/files, build custom support for them, and release an update. Camera companies reap the same benefit. Companies like Nikon and Canon have less motivation on this latter point because software vendors will be compelled to support the proprietary formats no matter what. That’s why smaller vendors have adopted DNG first. –J.]
You’ve got a chicken-and-egg problem, since any benefits require a critical mass to embrace and evangelize the format. Microsoft has the same problem with its “HD Photo” format, although they’re in a better position to force it down our throats whether or not anyone but Microsoft really benefits from it.
[I don’t really see them as being in a position to force anything on that front. Nor do they, I’m guessing, which is why they’ve pushed that format towards being a non-MSFT-controlled standard. –J.]
Well Hurray for Vista (especially x64) support! I now have a reason to convert my RAW files — knowing the support is coming for viewing in the OS.
I’ve been waiting for this. Now if only Adobe would publish an x64 bit iFilter…
As an aside, I see a place for both DNG and HD Photo, the former as a RAW archival format while the latter as a better quality point & shoot format. Perhaps other benefits will follow.
Imagine:
Dateline 23.09.2008 Cologne, Germany – photokina 2008. Canon, the worldwide market leader in digital cameras, announces a new strategic development direction for all of its future imaging products. Canon will transition from its legacy RAW image formats to an emerging new open standard called the Digital Negative created by Adobe Sytems. The new digital format will enable easier software development for all Canon imaging products, save costs and launch the photographic industry on a path of increased innovation and productivity.
“Canon’s 70-plus-year success in the photographic market enables us to see that the proprietary image formats of the past are holding us back. It is time for the industry to make a bold change. We are proud to be the camera of choice for consumers and professionals and intend to remain so by demonstrating our leadership” said Canon executive XXX YYYYYYYYY.
—————
In numerous after-show coverage articles, Nikon was quoted as saying “were studying the matter”.
From the original post:
[Update: I neglected to mention that yes, Adobe will be providing a DNG-viewing codec for Windows Vista, making it possible to view DNG files right within the operating system. Expect this free download to be posted soon. –J.]
Should this be interpreted to mean that there will not be a corresponding OSX codec?
[Apple already supports DNG via Quick Look, so there’s no need for Adobe to reinvent the wheel. –J.]
Could this be interpreted as part of a larger move to emphasize Windows over Macintosh?
[Of course it could, because no matter what Adobe does, some Mac users want to view everything as an oppressive conspiracy & thus will do so. –J.]
Re: Camera (vendor) specific processing
Canon, Hasselblad, presumably Nikon & others have conversion capabilities based on deep knowledge about their private tags. DxO also has unique features w/i their software.
I’m not sure how best to do this, but it would be nice if there were a way to provide multiple “filters” on top the the lightroom workflow. These filters should provide early development steps before the lightroom steps.
One possible way to do this is to create standard “plug-ins” the appear before the lightroom development settings. Rather than impact the raw file directly, the plug-in would create vendor specific XML to include into the lightroom development file. Subsequent rendering of the raw file (to the screen, tif or dng) would take the lightroom development file and pass it onto the next filter.
This would provide the feature for
3fr -> hasselblad specific conversions -> lightroom -> dng/a.
Another example
cr2 -> canon specific processing -> DxO specific processing -> lightroom -> dng/a
What would the vendor get? High quality (camera specific) rendering/archival functioning w/o having to create a whole workflow application itself.
The key filter would be the vendor specific XML preprocessing filter.The UI extension for lightroom, &c. May or may not be a value add from the vendor’s perspective. The vendor may choose to use the standard development “transaction” log (e.g. curves), it may choose to provide no additional UI.The vendor specific XML “transaction” extensions and UI plug-ins would provide a bridge to speed the transition between an XML standard & the current state.
For lightroom, aperture, &c. that would mean a standardization of the xml transaction logs. This standardization would take some time. Through the standardization process, the “transaction” files would change from
x-canon-x -> x-dxo-y -> lighroom or aperture xml
transaction files to:
x-canon-x -> x-dxo-y -> x-adobe-z -> iso-transactions
The vendors (e.g. canon, adobe, apple, dxo) can all add value and reuse standard transactions (e.g. curve). Workflow vendors may (or may not) provide plug-ins to extend adobe, apple, &c. workflow applications. The manufacturers and other vendors (e.g. DxO) may or may not choose to create those plug-ins.
The end result is that manufactures & software vendors can add value to the development process and create less ui/program behavior. Codecs can not supply this value chain. Vendors could only provide default visualization/development; with out the possibility of adding additional value.
As a user, what do I get? I get to use my favorite workflow program, keep all of the manufacture & independent software provided tools. My archival format will be provide by dng/a. If the common transactions become [themselves] standard, perhaps those could be added to dng/a at a later time.
At end of it all, dng/a provides no processing algorithms, codes, codecs &c. What it would provide is standard & vendor specific development “hints” for the conversion/development of the raw file. If canon, adobe or others find a better way to process “curve” at a future time, they could re-develop the image at that time. But these all all development hints.
Now, the near future and 100 years from now, we still would have our images in an archival format. Today we would have simple workflows and the best vendors can supply to develop images.
Confused? This is nearly stream of thought. I hope that I have provided enough clarity that I’ll have provided food for thought. If someone wants to help flesh out the idea, I’d be happy to work with them.
David
MONOPOLY, adobe get lost, GOOGLE should support gimp project instead of helping “adobe” with wine, thats a bad help.
WE WANT A “OPEN FORMAT” NOT DNG FROM ADOBE, YOU WANT MONEY MONEY AND MONEY, DONT TRY TO SAY YOU WANNA HELP THE COMMUNITY, ..CAMERAS, YEARS ETC… BLAH BLAH
From your response:
[Apple already supports DNG via Quick Look, so there’s no need for Adobe to reinvent the wheel. –J.]
Could this be interpreted as part of a larger move to emphasize Windows over Macintosh?
[Of course it could, because no matter what Adobe does, some Mac users want to view everything as an oppressive conspiracy & thus will do so. –J.]
———————
John, please don’t misinterpret my questions. The scenario is this: I am a Windows user since 1991. In the last few months, I decided to switch to the Mac, and am encountering the expected culture shock. I feel like I moved to a different country, as there are so many little things I just don’t see clearly.
My cousin, a programmer and graphics person was contemplating switching as well, and has been experiencing cold feet. He sent me a link to this post, with a note about how if Adobe is targeting Windows more these days, he doesn’t think it makes sense to switch. My comment was a matter of due diligence to find out if this assumption was true (at least in this case). I suspected that the idea of bias on this was incorrect, and, as you have shown, it was.
[Thanks for the clarification, Jay. For the record, I see no evidence of Adobe targeting Windows more than Mac these days. There *are* cases (e.g. the enterprise market, where Acrobat sells a huge number of seats) where the Mac just doesn’t really play, and consequently Adobe offers some Windows-only tools. In other markets, however, Adobe is bringing more tools to the Mac. For example, Adobe acquired Serious Magic and has been working to deliver the first Mac versions of their previously Windows-only products. –J.]
This was on top of the announcement regarding CS4 being 64bit on Windows only. My understanding on that, however is that the CS4 64bit situation is due to Apple playing Lucy to Adobe’s Charlie Brown – with the 64bit Carbon libraries being the old football. Do you have any additional info on this? Will 64bit wait until CS5?
[Please see my post on the subject. It was never our desire or intention to have the Mac 64 version lag behind the Windows 64 version, but we need to deal with changes in OS roadmaps as they come. –J.]
Anyway, thanks for the clarity on the DNG support.
Where does the bad blood you mention between some Mac uses and Adobe stem from?
[It comes from the fact that for a certain percentage of the Mac-using community, being pissed off isn’t a side effect of X Y or Z event happening; rather, it’s a goal unto itself. That is, it’s more gratifying to opt into being part of an oppressed minority, one that’s just a little too good, a little too tasteful for this dirty old world to understand. The Mac is a way for 40-year-old white guys to feel like victims. (I say all this with confidence as someone who’s been part of the Mac community for a *looooong* time and who was *very* partisan for quite a while during the Bad Ol’ Days of the mid-late 90’s (when Apple lost $750 million in a quarter and there really was reason to feel threatened).)
Btw, I don’t need to hear a recitation of every slight, real or imagined, visited by Adobe on the Mac community–about how FrameMaker was dropped, or old versions of Acrobat didn’t include all the features that the Windows version had, or Premiere going MIA for a while, etc. If I ever heard the flipside of that coin (about how Photoshop offered free updates to PPC & G5 plus a groundbreaking public beta for Intel; how Premiere Pro and a bunch of new Adobe apps have been brought to the Mac; how Lightroom was Intel-native before Aperture & is now the first major 64-bit Cocoa app, etc.), it might be a different story. We never get reminded about those things, though–just the negatives, because those fit better into a certain worldview of Mac user as eternal victim, nobly suffering for the sake of taste and quality. –J.]
what will happen regarding dng format should adobe go under?
[Nothing. That’s kind of the whole point: the format in which people store their intellectual property for decades should be fully & freely documented. That way that the data can always be accessed regardless of the status of whatever group invented the format. Adobe’s goal has been to build up DNG’s momentum (supplying tools, etc.) and then hopefully to turn it over to an independent body. –J.]
I learn of about 3 or 4 more (not-Adobe) products per month that support DNG in some way. My private count is now over 220 such products, of all types.
[Nice! Developers aren’t adding support out of some altruistic or ideological love of standards; they’re doing it because there’s clearly something in it for them & for their customers. –J.]
Source code is available for some products, such as dcraw. So we have both “amateur” and professional products to convert various formats to DNG, and others to render DNG images.
Wih millions of DNGs being created every day for both amateur & professional photographers, both the motivation and the ability for individuals and companies other than Adobe to continue with DNG exists.
John,
thank you for keeping us informed – and your clarifying comments. Can you please provide the ISO number the DNG format is filed under at the ISO-Site? My querry for “DNG” did not show any hit.
Thanks in advance
Dieter
What everyone fails to realize, even still, is that by creating a standardized format (like a DNG) you are trying to force Kodak to share film formulas with Fuji!
I didn’t spend 8,000 dollars on my D3x to have some amateur pick up a canon rebel and be able to “sync” in that same look, that same feel…
the camera companies MAKE the film now!! and that is exactly why I would NEVER allow an .xmp to completely ignore my original raw files instruction set…
if you buy a 30,000 dollar medium format digital back, for you hasselbad and you then open up that 30,000 dollar RAW file with an Adobe product (or ANY third-party raw convertor) you will be ignoring a 30,000 dollar instruction set! which of course was put their for a reason (its there so you can access your original RGB Luminance data, which is the point of shooting a raw file!)
when you process your raw whilst using an .xmp (or a DNG; which is basically what an .nef file already is… [RGB luminance date + instruction set + preview= .nef, a DNG is .nef(or any raw file) + horrible adobe raw conversion that is based of color at this point + .xmp+ preview]) you are going down a destructive path!!!
oh yes, believe it or not, using Adobes Raw conversion is, in fact, destructive! It becomes destructive as soon as you save that edited RAW file to a Tiff or a Jpg!
ok I’m rambling… but the point I’m trying to eventually make is… the reason one should shoot and use a “raw” file is to have access to the original RGB Luminance data, adobe does not provide that access… hence, not cool…
After reading your excellent Web page, I went to the ISO’s site and could find no mention whatsoever of ISO working on a DNG standard. Did Adobe change the name to something other than DNG before submission to the ISO. Or, is it first going through on of the US technical committees associated with the ISO?
John S, Dieter,
An ISO version of DNG is under development with Adobe’s support in the ISO TC42/WG18 committee. It’s not appropriate for Adobe to comment any more specifically on the process.
Regards,
Tom Hogarty
Lightroom, ACR, DNG, Bridge Product Manager
Adobe Systems