DATA STORAGE: A KEY TO POST INTEGRITY
Michael Joyce, Senior Director of Storage Marketing at AMCC
Issue: February 2006

DATA STORAGE: A KEY TO POST INTEGRITY

Digital data is a fact of life in the movie business. Most of the recent blockbusters have been shot in one form or another on digital media, and the results are most evident in the labs and studios that work to enhance and modify what the camera recorded.

Post production facilities have been eager to harness the power of the digital changeover with studios pushing the technology envelope. Special effects, remastering and other technically-complex operations take a whole lot of computing power. When Jurassic Park first played, Silicon Graphics, a leading company in the desktop workstation market, garnered a ton of publicity and endorsements by advertising themselves as the key to all special effects. Mostly, they were responsible for the digital magic that we all now take for granted.

The modern post production facility manager is more likely - with his budget firmly in mind - to line up and interconnect a bank of PCs or Macs and feed off their combined computing power. This lowers the cost of participation, and allows more power to be added in an economical manner simply by buying more computers. The whole investment is laced with complex software and expensive peripherals that allow almost any effect, any detail to be scrutinized, altered and rendered.

The thrust of the creative inputs from post production systems lies in the creation of new sequences costing thousands of dollars per frame working directly with and on the digitally shot master files. The output, also in digital format, is enshrined on a DVD and sold by the millions.

But what of old movies? What happens to the millions of feet of celluloid shot when digits only referred to fingers? Just like old 8mm and Super 8 home movies used to be converted to Beta or VHS and later reconverted to DVDs, old movies are being digitized and produced in digital formats.

In most cases the old reels are taken off the shelf where they have been languishing, sometimes for more than 75 years, and run through a digitizing process, essentially projecting each frame onto an image capturing graphic sensor, i.e. a digital camcorder. What the sensor captures is the raw footage of the original movie, warts and all which is downloaded, copied and distributed.

Because of their popularity, box office hits suffer the most degradation. Every time the master film is dusted off and recopied for another production run, the basic footage suffers from a number of injuries. Scrapes, pinpoint holes, jitter, dirt, and static burns all appear deposited either individually or on large segments of the film. To make it whole once again requires that HD images be made of each frame to capture all the details of the old celluloid and eliminate the accumulated dirt, grime and sprocket jitters. In rare cases, where there are no acceptable masters, the film restorer has to digitize from the negatives. For black and white movies, this is a single pass operation with a simple negative to positive alteration. Color movies require four passes through the digitizer and synchronized reassembly in post production.

The way technology advances, what used to be acceptable degradation for the consumer is no longer marketable. High definition TV, projection units and home entertainment centers have reached a level of viewing perfection that the old "run it as it is" criteria makes the defects unacceptable, detracting from the viewing experience.

John Lowry is arguably the industry's preeminent film restoration guru. Working closely with George Lucas, he recently restored and delivered the early Star Wars trilogy first produced in 1977. He has worked on films as early as a 1933 version of The Ghoul and as recent as the Indiana Jones series.

Using a bank of hundreds of Apple G5 and G4 Power Macs linked together to form a virtual supercomputer, Lowry reviewed and edited every frame of the three classics from the negative. Digitizing a full length feature film requires sophisticated editing tools, fast computing power and massive amounts of digital storage.

The technical wizardry has already been fully explained in numerous articles, what is less well understood is the critical importance of the role played by the storage devices. In Lowry's case, he needed enough raw storage to house over two hours of film, additional data to clean up the film and processing storage to do the actual restoration in realtime and with significant security so that nothing would get lost during the operations.

First, the complete movie had to be available to the restoration team to ensure uniformity throughout the production, and each 4K frame had to be digitized in high resolution to capture each detail. There are about 172,000 frames in a full length feature which means that 6.9 GB of storage is required.

To store this data, and a favorite amongst traditional photographers, are the vast tape banks that transition between the world of analog and digital. Now with inexpensive SATA drives, RAID 5 controllers and PC and Mac technology used to create large Petabyte servers, the SATA RAID revolution is well underway. A SATA/RAID layer costs only a fraction of SCSI or FC and requires exponentially less physical space than magnetic or tape storage strategies - $1.05 per GB SATA versus $5.50 FC and $5.15 SCSI.

John Lowry took his cue from the corporate world and installed a 378 TB storage facility by connecting a large number of SATA drives through an AMCC RAID controller.

One of the keys to this shift to SATA is the technological advances achieved by the ATA drive industry. The largest capacity drives are the SATA 7200 RPM disks. Today they are 500 GB, next year they are projected to be 720 GB and by 2007 they should be at 1 TB. Along with this dramatic increase in capacity will come much higher performance. If twice the amount of data is moving under the read/write head, then the drive reads and writes will be at twice the speed. The newer SATA II drives bring a number of key performance improvements including 3Gb/second interface speed, Native Command Queuing (NCQ), and Port Multiplier capability.

SATA II is designed for scalability. Port Multiplier enables 15 drives to be connected to a single SATA controller. This is similar to USB connectivity, but with the performance benefits of an aggregated switch. The host knows that it is communicating to multiple drives, but the drives are unaware that they are being multiplexed and function as if they were directly attached to the host adapter. Bottom line ? a one channel SATA II RAID adapter can support up to 2.8 TB of storage! And scalability is not accomplished at the expense of performance. Sustained I/O rates from the drives are kept to within the 3Gb/second host port connection.

What of the old standby, tape? When all is said and done, tape is used for the final depository of the finished product and the preservation of the all the artifacts that went into the creation of the master copy. It is used as an insurance policy, and a guarantee that in later years, the final cut will be available in a pristine condition. This is the hard lesson learned from the degradation of those millions of nitrocellulose cells and frames that have turned into powder in storage facilities around the world.