Home > Ted’s Commentaries > Technology > Image File Formats

Image File Formats
Updated October 2016

DNG: Archival Solution or (Compact) Flash in the Pan?

The Problem

It’s common to talk about raw files as if they were a single standard like JPEG. But nearly every camera capable of storing the minimally-processed data from its sensor on a memory card actually uses its manufacturer’s unique proprietary format, which may even be specific to that model. That might not seem much of a problem, as those cameras almost always have a CD with raw conversion software packed in the box. There are also third-party raw converters that are often better than what the camera manufacturers provide.

But the babel of formats could cause real trouble sooner than you might think. Like other kinds of computer technology, digital cameras are evanescent. In a mere five years, Canon’s D30— a revolutionary 3-megapixel DSLR introduced in 2001— was successively replaced with the D60, the 10D, the 20D, and the 30D. (As of October 2016, the current representative of that lineage is the 80D.) That’s just one manufacturer, and it doesn’t even include Canon’s full-frame DSLRs, or the “entry-level” Digital Rebel/300D and its generations of successors.

In 2005, Canon removed support for the “legacy” D30 in their Digital Photo Professional raw converter. Complaints from the many people who (unknown to Canon’s management?) still used the D30 forced Canon to restore that support. But who’s to say they won’t drop it again— perhaps along with the D60, 10D, or even the 80D— in a future version?

Even if you’re wealthy enough to regularly replace your cameras with the newest models, the ghosts of cameras past will still haunt your collection of raw files. You could save the raw conversion software that came with those cameras, and reinstall it when necessary. But what if it won’t run on your computer or operating system?

The 64-bit versions of Microsoft’s Vista, Windows 7, Windows 8, and Windows 10 operating systems can no longer run 16-bit legacy software written for DOS or Windows 3.1 (though it might run with an emulator, such as DosBox). And some 32-bit software written for Windows 95, 98, or XP is incompatible with newer versions of Windows because of security enhancements introduced in Vista.

Apple’s Macintosh computers have even more problems with legacy software, as their history includes three distinct, mutually-incompatible types of processors. Current Macintoshes use the same Intel x86-64 processors as Windows PCs. They can no longer run software written for the first Macs, which used the Motorola 68000 processor, or for the second generation that used the Motorola/IBM PowerPC processor. The OS X operating system originally included an emulator (“Rosetta”) that let Intel-based Macs run PowerPC software. But Apple deleted it 2011 with the Lion release.

For that matter, new versions of OS X often break software designed for earlier versions of that operating system, even though the processor and underlying Unix-based architecture are the same. And there are persistent rumors that a future generation of Macintoshes will replace Intel processors with yet another incompatible architecture: the ARM processor used in the iPhone and iPad.

That all means it’s not merely conceivable but quite likely that, say 20 years from now, you’ll have to go to a museum to find a computer capable of running your old 32-bit Windows or Intel Mac raw converters that can read your legacy files.

Adobe’s Answer

Adobe’s solution to this problem is DNG (pronounced ding, but occasionally dingy or dinghy), their universal “Digital NeGative” format for raw files. Like the ubiquitous PDF and TIFF formats (DNG is based on TIFF), Adobe owns and controls the rights to DNG. But they’ve publicly documented the specification, and they grant a free license to anyone who creates software that reads or writes DNG files. They’ve submitted it to International Organization for Standardization (ISO) for inclusion in the international standard for the version of the TIFF format used for photography (TIFF/EP). If DNG becomes widely accepted— preferably as the standard raw format used by all camera manufacturers— the problem of unreadable proprietary formats will be solved. At least that’s what Adobe claims.

Adobe has had limited success persuading camera manufacturers to adopt DNG. A few high-end (Hasselblad, Leica) and minor (Casio, Ricoh, Samsung) manufacturers use DNG as their cameras’ raw format. Coming late to the digital table, they stand to benefit the most from using Adobe’s specification rather than developing their own raw formats. Pentax cameras offer a choice of proprietary or DNG formats. But Canon and Nikon, which produce the majority of cameras capable of writing raw files, along with Sony and Panasonic, have thus far thumbed their corporate noses at DNG and Adobe. And ISO seems to be in no hurry to issue a DNG standard, having made no progress since Adobe submitted it in 2007.

Until they can convince the major camera manufacturers to adopt DNG, Adobe is promoting it as a “future-proof” archival format, in which photographers can save raw files converted from proprietary formats. They’ve had a notable success with the American Society of Media Photographers’ Digital Photography Best Practices and Workflow project, funded by the Library of Congress. The project recommends DNG as “a secure openly documented and forward compatible format for image archiving.”

Adobe provides a free DNG converter that can translate many proprietary formats into DNG. They update it to support new cameras whenever they update Adobe Camera Raw (ACR), the raw conversion “engine” for both Photoshop and Lightroom. Adobe’s DNG converter can optionally embed (and extract) the original raw files into DNG files, to retain the option of using manufacturers’ raw converters that can’t read DNG.

For Photoshop users, Adobe’s DNG converter may offer benefits more compelling than potential archival permanence. The ACR plug-in for each Photoshop version is intentionally incompatible with previous Photoshop versions. And Adobe stops updating old versions of ACR once a new version of Photoshop is released. So if you’re using an old version of Photoshop (CS6 and earlier) and you buy a new camera, you’ll need to either start renting Photoshop CC or buy a copy of Lightroom if you want to process its raw files with ACR.

When Adobe adopted their “Creative Cloud” rental model for Photoshop, they said they would indefinitely provide updates to the CS6 version of ACR to support new cameras. In July 2015, Adobe decided it was time to gently encourage stubborn CS6 users to start paying rent. With the release of ACR version 9.1.1, they announced that it would be the “final version available for use with CS6.”

If you are one of the many Photoshop users who aren’t interested in renting software, Adobe generously offers their DNG converter as a free alternative. Every Photoshop version since CS can read DNG; and the converter installs profiles for every camera the current version of ACR supports. Once you convert your new camera’s raw files to DNG, you can continue to use your “legacy” version of Photoshop.

Adobe’s DNG converter seems to offer a mutually-beneficial arrangement: The user gets a way to avoid the continuing expense of renting Photoshop CC, along with raw files converted to what might be a future-proof archival format. Adobe gets, if not an enthusiastic DNG evangelist, one more stakeholder in Adobe’s vision of DNG as the future of raw files.

Moving the Bandwagon

Will the major camera manufacturers ever embrace Adobe’s vision? It’s too soon to tell. Canon and Sony currently give away their official raw converters to anyone who buys their cameras; Panasonic gives away a customized version of Ichikawa Soft Laboratory’s SilkyPix. So they would seem to have little to lose from switching to DNG, other than control of their proprietary formats. But Nikon sells their advanced Capture NX 2 raw converter as an extra-cost accessory. That “ancillary revenue” is a strong disincentive to give up their proprietary format. (Nikon cameras come with the snapshooter-oriented Nikon View NX software, which provides rudimentary raw conversion.)

Nikon received many complaints when they released new digital SLRs in 2005 that encrypted the white balance information in their raw files. In response, they issued a defiant press release defending the encryption as essential for the “preservation of [Nikon’s] unique technology.” That led a “working group of photographers,” who considered Nikon’s behavior arrogant, to form OpenRaw.

OpenRaw’s goal was to make complete documentation available for past, present, and future proprietary raw file formats. They carefully distinguished their initiative from Adobe’s goal of a single universal standard, which they considered impractical. For a while, Nikon’s avaricious encryption of a minor part of their raw files seemed to have paradoxically increased the demand among photographers for open access to those files. But OpenRaw shut down in 2006, apparently due to a lack of clear direction, without getting documentation for even one camera’s raw format.

I Get Converted

I had been watching from the sidelines as Adobe promoted DNG. Then in 2014, I replaced my 2005-vintage Canon Digital Rebel XT/350D with an SL1/100D. I use Photoshop CS5, for which the last compatible version of ACR was released in May 2012. As the SL1 came out in March 2013, Photoshop CS5 can’t read its raw files. I identified six options to address that problem:

  1. Use Canon’s Digital Photo Professional (DPP) raw converter (free).
  2. Buy the Photoshop CS6 upgrade ($200).
  3. Rent Photoshop CC ($10/month, for now).
  4. Buy Lightroom ($150), or some other third-party raw converter.
  5. Convert the new camera’s raw files to DNG, and keep using Photoshop CS5 (free).
  6. Give up on raw files and use JPEG (free).

The first option was out because, like many Canon users, I intensely dislike DPP. The last option (using JPEG) is not appropriate.

When Adobe released Photoshop CS6, I could see no compelling reason to buy the upgrade. When I got the new camera, the ability to read its raw files suddenly became a useful feature of CS6; but that by itself wasn’t enough to justify the cost of the upgrade. As I also find the idea of renting software distinctly unappealing, Photoshop CC is not a desirable option either.

(Although I dislike the concept of renting software, I fully appreciate why Adobe’s executives needed to impose that model. Creating truly compelling improvements to a mature, feature-rich product like Photoshop becomes more difficult with each version. They could no longer rely on adding enough value to new versions that would make users want to buy upgrades. Taking advantage of their “industry-standard” market position and forcing users to continually pay for the use of Adobe products was, therefore, the only way they believed they could meet their obligations to shareholders.)

Lightroom is still available with a “perpetual license” (though it may not be for long, now that Apple has abandoned Aperture, Lightroom’s main competition). The problem is that Lightroom is primarily an image management database for professional photographers, which can also make “parametric” (non-destructive) adjustments to images using ACR. It would thus require a significant change to the way I work. I might be interested in ascending that learning curve someday, but not now.

By default, DNG became the best option.

Seeing the Light

When I tried the Adobe converter, I was pleasantly surprised to find that the conversion step is much less of a hassle than I expected. It even provides some benefits. First, the converter can traverse all the folders on a memory card. You can then choose to either place the converted DNG files in the same set of folders in a specified location on a hard drive, or in a single “flattened” directory. If you prefer the latter arrangement (as I do), using the converter is much easier than manually moving the files from each folder on the memory card. Second, the DNG files are, on average, 18% smaller than the Canon CR2 files. That size reduction can be welcome when archiving the files to DVDs or Blu-Ray disks with the finished images.

The converted files seem to preserve the complete metadata from the camera, or at least the metadata that ACR can use. I noticed that the SL1 records not just the lens focal length used for a picture, but the make and model of the lens. ACR can thus automatically apply the appropriate lens distortion correction. The old 350D didn’t record that information, so I had to look at the focal length and use two pull-downs to choose the profile manually for each image.

The main practical difference between using DNG and proprietary formats is that ACR modifies DNG files to save the XML image adjustment data. With proprietary formats, it creates a separate “sidecar” file for that data. I really prefer leaving raw files unchanged from what the camera created, but that’s apparently not an option with DNG. (Canon’s DPP also modifies raw files to store adjustment settings.) Fortunately, my method of archiving raw files provides a way around that problem.

I archive every raw file I shoot in folders organized by year, place, and date. (They’re backed up to Blu-Ray disks.) I also archive the roughly 10% of those raw files that yield finished images, along with their “sidecar” files, with the finished images on a hard disk, and also on DVD or Blu-Ray.

Now that I’m using DNG, I copy the original CR2 files to the complete raw file archive. I archive the DNG files that yield finished images with those images. (I discard the DNG versions of the other raw files, as they can be converted again from the complete raw file archive if that’s ever necessary.) I also save as DNG the raw files from my Canon Powershot S100 that yield finished images, even though my version of ACR can read that camera’s files directly. I save them in ACR just before I pass the adjusted image to Photoshop. The DNG versions are smaller, and they make it easy to identify the “keepers” in a folder full of CR2 raw files.

Final Thoughts

Although I now regularly use DNG, I have not become an enthusiastic DNG evangelist. I am not convinced that DNG is the One Ring to Rule All the Raw Files. That’s the main reason I haven’t taken the step of converting my entire archive of raw files to DNG. (The other important reason is that the Adobe converter can’t preserve the contents of “sidecar” files, so I’d lose those adjustments.) But DNG does seem to offer useful advantages, with almost no disadvantages. In particular, I appreciate the way DNG allows me to continue using software I already “own”— or more accurately, that I have a perpetual license to use— with a new camera.

I would consider a mass conversion of my raw files to DNG if I had a collection of raw files from a camera whose manufacturer has gone bankrupt or has left the market (e.g., Konica-Minolta), or whose manufacturer has an insignificant market share (e.g., Sigma). It might be more sensible to save the best output of your favorite raw converter as an uncompressed 16-bit TIFF file and/or a 16-bit flat PSD (native Photoshop) file (see A Bestiary of File Formats for more information). Both formats have enormous user bases and enough history to ensure that they will remain readable for any foreseeable future. The American Society of Media Photographers apparently shares that belief, since their “best practice” recommends those formats for archiving finished images.

Sidebar: My Canon S90 illustrates one shortcoming of DNG as an archival format. Like many compact cameras, the S90 (along with its successors, the S95, S100, S110, and S120) has a lens that is not fully corrected for distortion and chromatic aberration. It relies on software to make those corrections, which in the past would have been made with additional lens elements. This approach significantly reduces the size, weight, and cost of the camera while still providing excellent image quality after correction.

The camera’s firmware automatically applies the corrections when it creates JPEG files, but not raw files. (The S90 and its successors are among the very few compact cameras that can create raw files.) Canon’s DPP raw converter applies the corrections when it processes raw files in the camera’s native format. Conversion to DNG would preclude the use of Canon’s converter, which can’t read DNG. Choosing the option in Adobe’s DNG converter that embeds the complete original raw file in the DNG gets around that limitation, although it more than doubles the size of the file. ACR 6 (which works with Photoshop CS5) can make the lens corrections for native raw files as well as those converted to DNG, since it uses separate (reverse-engineered) lens profiles included with the plug-in or available for download.

However, if I wanted to use the Adobe DNG converter to process my S90 raw files in Photoshop CS3 (for example), its version of ACR would not make the lens corrections. Automatic profile-based lens correction first appeared in ACR 6.1, which works only with CS5. Any other third-party raw converter that can read DNG files would have the same problem, unless it too had its own lens profiling and correction capability. Similarly, if future versions of DPP and ACR dropped support for the long-obsolete S90, DNG conversion might preserve a readable image file (assuming, of course, that the DNG format remains supported). But the distortions and aberrations would have to be corrected manually for each image.

DNG might be a first step toward future-proof raw files. But as this example demonstrates, it’s not a complete solution. For now, the best approach might be to save the save the best output of your favorite raw converter as a 16-bit uncompressed TIFF or flat PSD file.


Commentary Contents

tedsimages.com Virtual Light Table Home Page