Home > Ted’s Commentaries > Technology > Image File Formats

Image File Formats
Updated May 2023

DNG: Archival Solution or (Compact) Flash in the Pan?

The Problem

It’s common to talk about raw files as if they were a single standard like JPEG. But nearly every camera capable of storing the minimally-processed data from its sensor on a memory card actually uses its manufacturer’s unique proprietary format, which may even be specific to that model. That might not seem much of a problem, as those cameras almost always include the manufacturer’s raw conversion software on a CD or as a download. There are also third-party raw converters that are often better than what the camera manufacturers provide.

But the babel of formats could cause real trouble sooner than you might think. Like other kinds of computer technology, digital cameras are evanescent. Planned obsolescence is an essential part of their manufacturers’ business model. In a mere five years, Canon’s D30— a revolutionary 3-megapixel DSLR introduced in 2001— was successively replaced with the D60, the 10D, the 20D, and the 30D. (As of May 2023, the current representative of that lineage is the 90D.) That’s just one manufacturer, and it doesn’t even include Canon’s full-frame DSLRs, the “entry-level” Digital Rebel/300D and its generations of successors, or the “mirrorless” cameras apparently slated to supplant the DSLR.

In 2005, Canon removed support for the “legacy” D30 in their Digital Photo Professional raw converter. Complaints from the many people who, apparently unknown to Canon’s management, still used the D30 forced Canon to restore that support. But who’s to say they won’t drop it again— perhaps along with the D60, 10D, or even the 90D— in a future version?

Even if you’re wealthy enough to regularly replace your cameras with the newest models, the ghosts of cameras past will still haunt your collection of raw files. You could save the raw conversion software that came with those cameras, and reinstall it when necessary. But what if it won’t run on your computer or operating system?

The 64-bit versions of Microsoft Windows have not been able to run 16-bit legacy software written for DOS or Windows 3.1 since Vista was released in 2007 (though it might run with an emulator, such as DosBox). And most 32-bit software written for Windows 95, 98, or XP is incompatible with newer versions of Windows because of security enhancements first introduced in Vista.

Apple’s Macintosh computers have even more problems with legacy software. Their history includes four distinct, mutually-incompatible types of processors. The latest Macintoshes use the M2 CPU, Apple’s version of the ARM processor also used in the iPhone and iPad. Previous Macs used the same Intel x86-64 processors as Windows PCs. They could no longer run software written for the first Macs, which used the Motorola 68000 processor, or for the second generation that used the Motorola/IBM PowerPC processor. The OS X operating system originally included “Rosetta,” an emulator that let Intel-based Macs run PowerPC software; but Apple deleted it 2011 with the Lion release. The M1 and M2 version of OS X includes “Rosetta 2,” an emulator reported to run what is now “legacy” Intel Mac software faster than actual Intel hardware. But it seems inevitable that Apple will pull the plug on “Rosetta 2” as soon as they decide users have had enough time to buy new software.

For that matter, new versions of OS X often break software designed for earlier versions, even with the same hardware and underlying Unix-based architecture. (Apple and its aficionados consider this breakage a highly beneficial feature. It ensures that if a user chooses software from someone other than Apple, that developer will have the commitment and resources to actively support the product, and to provide timely updates whenever Apple updates its operating system.) So if a camera manufacturer deletes a “legacy” camera from its raw converter, an older version that can read the files may not run on a current Mac.

That all means it’s not merely conceivable but quite likely that, say 20 years from now, you’ll have to go to a museum to find a computer capable of running the old 32-bit Windows or Intel Mac software that can read your “legacy” raw files.

Adobe’s Answer

Adobe’s solution to this problem is DNG (pronounced ding, but occasionally dingy or dinghy), their universal “Digital NeGative” format for raw files. Like the ubiquitous PDF and TIFF formats (DNG is based on TIFF), Adobe owns the patents and controls the rights to DNG. But they’ve publicly documented the specification, and they grant a free license to anyone who creates software that reads or writes DNG files. They’ve submitted it to International Organization for Standardization (ISO) for inclusion in the international standard for the version of the TIFF format used for photography (TIFF/EP). If DNG becomes widely accepted— preferably as the standard raw format used by all camera manufacturers— the problem of unreadable proprietary formats will be solved. At least that’s what Adobe claims.

Adobe has had limited success persuading camera manufacturers to adopt DNG. A few high-end (Hasselblad, Leica) and minor (Casio, Ricoh, Samsung) manufacturers use DNG as their cameras’ raw format, as do the manufacturers of high-end smartphones that can produce raw files. Coming late to the table, they stand to benefit the most from using Adobe’s specification rather than developing their own raw formats. Pentax cameras offer a choice of proprietary or DNG formats. But Canon, Nikon, Sony, and Panasonic, which produce the majority of cameras capable of writing raw files, have thus far thumbed their corporate noses at DNG and Adobe. And ISO seems to be in no hurry to issue a DNG standard. Adobe submitted it in 2007, but as of May 2023 its page on the ISO Web site shows it as “under development” in an early “preparatory” life cycle stage.

Until they can convince the major camera manufacturers to adopt DNG, Adobe is promoting it as a “future-proof” archival format, in which photographers can save raw files converted from proprietary formats. They’ve had a notable success with the American Society of Media Photographers’ Digital Photography Best Practices and Workflow project, funded by the Library of Congress. The project recommends DNG as “a secure openly documented and forward compatible format for image archiving.”

Adobe’s Free DNG Converter

Adobe provides a free DNG converter for Macintosh and Windows that can translate proprietary raw files into DNG. They update it to support new cameras whenever they update Adobe Camera Raw (ACR), the raw conversion “engine” for both Photoshop and Lightroom. Adobe’s DNG converter can optionally embed (and extract) the original raw files into DNG files, to retain the option of using manufacturers’ raw converters that can’t read DNG.

For Photoshop users, Adobe’s DNG converter may offer benefits more compelling than potential archival permanence. The ACR plug-in for each Photoshop version is intentionally incompatible with previous Photoshop versions. And Adobe stops updating old versions of ACR once a new version of Photoshop is released. So if you’re using an old version of Photoshop (CS6 and earlier) and you buy a new camera, you’ll need to start renting Photoshop CC or Lightroom if you want to process its raw files with ACR.

The Adobe DNG converter provides a free alternative. Every Photoshop version since CS (released in 2003) can read DNG; and the converter installs profiles for every camera the current version of ACR supports. Once you convert your new camera’s raw files to DNG, you can continue to use your “legacy” version of Photoshop.

Adobe’s DNG converter seems to offer a mutually-beneficial arrangement for users of old Photoshop versions who don’t want to rent Photoshop CC: The user gets continuing support for new cameras, along with what might be a “future-proof” archival format for raw files. Adobe gets one more stakeholder in their vision of DNG as the future of raw files.

Moving the Bandwagon

Will the major camera manufacturers ever embrace Adobe’s vision? It’s still too soon to tell. Canon and Sony have always given away their official raw converters to anyone who buys their cameras. Panasonic gives away a customized version of Ichikawa Soft Laboratory’s SilkyPix. So they would seem to have little to lose from switching to DNG, other than control of their proprietary formats. For years, Nikon included Nikon View NX, a rudimentary snapshooter-oriented raw converter, with their DSLR cameras. They sold their advanced Capture NX 2 raw converter as an extra-cost accessory. That “ancillary revenue” may have been a strong disincentive to give up their proprietary format. They now provide Capture NX-D, the successor to Capture NX 2, for free; but their cameras still use Nikon’s proprietary raw format.

Nikon received many complaints when they released new digital SLRs in 2005 that encrypted the white balance information in their raw files. In response, they issued a defiant press release defending the encryption as essential for the “preservation of [Nikon’s] unique technology.” That led a “working group of photographers,” who considered Nikon’s behavior arrogant, to form OpenRaw.

OpenRaw’s goal was to make complete documentation available for past, present, and future proprietary raw file formats. They carefully distinguished their initiative from Adobe’s goal of a single universal standard, which they considered impractical. For a while, Nikon’s avaricious encryption of a minor part of their raw files seemed to have paradoxically increased the demand among photographers for open access to those files. But OpenRaw shut down in 2006, apparently due to a lack of clear direction, without getting documentation for even one camera’s raw format. Meanwhile, Nikon quietly provided Adobe with software that could access the specific encrypted data needed for ACR.

I Get Converted

I had been watching from the sidelines as Adobe promoted DNG. Then in 2014, I replaced my 2005-vintage Canon Digital Rebel XT/350D with an SL1/100D. I use Photoshop CS5, for which the last compatible version of ACR was released in May 2012. As the SL1 came out in March 2013, Photoshop CS5 can’t read its raw files. I identified six options to address that problem:

  1. Use Canon’s Digital Photo Professional (DPP) raw converter (free).
  2. Buy the Photoshop CS6 upgrade ($200).
  3. Rent Photoshop CC ($10/month, for now).
  4. Buy Lightroom ($150), or some other third-party raw converter.
  5. Convert the new camera’s raw files to DNG, and keep using Photoshop CS5 (free).
  6. Give up on raw files and use JPEG (free).

The first option was out because, like many Canon users, I intensely dislike DPP. The last option (using JPEG) is not appropriate.

When Adobe released Photoshop CS6, I could see no compelling reason to buy the upgrade. When I got the new camera, the ability to read its raw files suddenly became a useful feature of CS6; but that by itself wasn’t enough to justify the cost of the upgrade. As I also find the idea of renting software distinctly unappealing, Photoshop CC is not a desirable option either.

In 2014, Lightroom was still available with a “perpetual license.” After Apple abandoned Aperture, Lightroom’s main competition, in October 2015, it was inevitable that Adobe would force Lightroom users into their rental scheme. They did that in October 2017. The problem I have with Lightroom is that it’s primarily an image management database that can also make “parametric” (non-destructive) adjustments to images using ACR. It would thus require a significant change to the way I work.

(Although I dislike the concept of renting software, I fully appreciate why Adobe’s executives needed to impose that model on users. Creating truly compelling improvements to a mature, feature-rich product like Photoshop becomes more difficult with each version. They could no longer rely on adding enough value to new versions that would make users want to buy upgrades. Taking advantage of their “industry-standard” market position— i.e., monopoly power— and forcing users to continually pay for the use of Adobe products was, therefore, the only way they believed they could meet their paramount obligation to shareholders. People who rely on Adobe products for their livelihood generally appreciate the rental model, as they benefit from the Adobe’s continual incremental improvements. The benefit for everyone else is that it gives Adobe’s competitors an incentive to improve their products. There are now several potentially viable alternatives for users who don’t want to rent Adobe’s software.)

By default, DNG conversion became the best option.

Seeing the Light

When I tried the Adobe DNG converter, I was pleasantly surprised to find that the conversion step is much less of a hassle than I expected. It even provides some benefits. First, the converter can traverse all the folders on a memory card. That’s useful when a camera is set to create a new folder each day it’s used. You can then choose to either place the converted DNG files in the same set of folders in a specified location on a hard drive, or in a single “flattened” directory. If you prefer the latter arrangement (as I do), using the converter is much easier than manually moving the files from each folder on the memory card. Second, the DNG files are, on average, 18% smaller than the Canon CR2 files. Even a relatively modest size reduction can be welcome when archiving the files to DVDs or Blu-Ray disks with the finished images.

The converted files seem to preserve the complete metadata from the camera, or at least the metadata ACR can use. I noticed that the SL1 records not just the lens focal length used for a picture, but the make and model of the lens. ACR can thus automatically apply the appropriate lens distortion correction. The old 350D didn’t record that information, so I had to look at the focal length and use two pull-downs to choose the profile manually for each image.

The main practical difference between using DNG and proprietary formats is that ACR modifies DNG files to save the XML image adjustment data. With proprietary formats, it creates a separate “sidecar” file for that data. I really prefer leaving raw files unchanged from what the camera created, but that’s apparently not an option with DNG. (Canon’s DPP also modifies raw files to store adjustment settings.) Fortunately, my method of archiving raw files provides a way around that problem.

I archive every raw file I shoot in folders organized by year, place, and date. (They’re backed up to Blu-Ray disks and to a cloud-based backup server.) I also archive the roughly 10% of those raw files that yield finished images, along with their “sidecar” files, with the finished images on a hard disk, and also on DVD or Blu-Ray.

Now that I’m using DNG, I copy the original CR2 files to the complete raw file archive. I archive the DNG files that yield finished images with those images. (I discard the DNG versions of the other raw files, as they can be converted again from the complete raw file archive if that’s ever necessary.) I also save as DNG the raw files from my Canon Powershot S100 that yield finished images, even though my version of ACR can read that camera’s files directly. I save them in ACR just before I pass the adjusted image to Photoshop. The DNG versions are smaller, and they make it easy to identify the “keepers” in a folder full of CR2 raw files.

Final Thoughts

Although I now regularly use DNG, I have not become an enthusiastic DNG evangelist. I am not convinced that DNG is the One Ring to Rule All the Raw Files. That’s the main reason I haven’t taken the step of converting my entire archive of raw files to DNG. (The other important reason is that the Adobe converter can’t preserve the contents of “sidecar” files, so I’d lose those adjustments.) But DNG does seem to offer useful advantages, with almost no disadvantages. In particular, I appreciate the way DNG allows me to continue using software I already “own”— or more accurately, that I have a perpetual license to use— with a new camera.

I would consider a mass conversion of my raw files to DNG if I had a collection of raw files from a camera whose manufacturer has gone bankrupt or has left the market (e.g., Konica-Minolta), or whose manufacturer has an insignificant market share (e.g., Sigma). It might be more sensible to save the best output of your favorite raw converter as an uncompressed 16-bit TIFF file and/or a 16-bit flat PSD (native Photoshop) file (see A Bestiary of File Formats for more information). Both formats have enormous user bases and enough history to ensure that they will remain readable for any foreseeable future. The American Society of Media Photographers apparently shares that belief, since their “best practice” recommends those formats for archiving finished images.

Sidebar: My Canon S90 and S100 illustrate one shortcoming of DNG as an archival format. These cameras (along with the S95, S110, and S120) have lenses that are not fully corrected for distortion and chromatic aberration. They rely on software to make those corrections, which in the past would have been made with additional lens elements. This approach significantly reduces the size, weight, and cost of the camera while still providing excellent image quality after correction.

The cameras’ firmware automatically applies the corrections when they create JPEG files, but not raw files. Canon’s Digital Photo Professional raw converter applies the corrections when it processes raw files in the cameras’ native format. Conversion to DNG would preclude the use of Canon’s converter, which can’t read DNG. Adobe’s DNG converter provides a way around that problem, with an option to embed the complete original raw file in the DNG file. The Adobe DNG converter can then extract the embedded native raw file if it’s ever needed. But embedding the original file more than doubles the size of the DNG file.

ACR 6 (which works with Photoshop CS5) and later versions of Photoshop and Lightroom can make the lens corrections for native raw files as well as those converted to DNG. They use Adobe’s reverse-engineered lens profiles, included with the plug-in or installed with the Adobe DNG converter.

However, suppose I wanted to use the Adobe DNG converter to process my S90 raw files in Photoshop CS2. The Windows version of CS2 will run on Linux with the wine compatibility layer. But its version of ACR would not make the lens corrections. Automatic profile-based lens correction first appeared in ACR 6.1, which works only with CS5. Any other third-party raw converter that can read DNG files would have the same problem, unless it had its own lens profiling and correction capability. Similarly, if future versions of DPP and ACR dropped support for the long-obsolete S90, DNG conversion might preserve a readable image file (assuming, of course, that the DNG format remains supported). But the distortions and aberrations would have to be corrected manually for each image.

DNG might be a first step toward future-proof raw files. But as this example demonstrates, it’s not a complete solution. For now, the best approach might be to save the save the best output of your favorite raw converter as a 16-bit uncompressed TIFF or a flat PSD file.


Commentary Contents

tedsimages.com Virtual Light Table Home Page