How Toy Story 2 Almost Got Deleted... Except That One Person Made A Home Backup

from the did-it-break-the-rules? dept

Here's a random story, found via Kottke, highlighting how Pixar came very close to losing a very large portion of Toy Story 2, because someone did an rm * (non geek: "remove all" command). And that's when they realized that their backups hadn't been working for a month. Then, the technical director of the film noted that, because she wanted to see her family and kids, she had been making copies of the entire film and transferring it to her home computer. After a careful trip from the Pixar offices to her home and back, they discovered that, indeed, most of the film was saved:

Now, mostly, this is just an amusing little anecdote, but two things struck me:
  1. How in the world do they not have more "official" backups of something as major as Toy Story 2. In the clip they admit that it was potentially 20 to 30 man-years of work that may have been lost. It makes no sense to me that this would include a single backup system.
  2. I wonder if the copy, made by technical director Galyn Susman, was outside of corporate policy. You would have to imagine that at a place like Pixar, there were significant concerns about things "getting out," and so the policy likely wouldn't have looked all that kindly on copies being used on home computers.
The Mythbusters folks wonder if this story was a little over-dramatized, and others have wondered how the technical director would have "multiple terabytes of source material" on her home computer back in 1999. That resulted in an explanation from someone who was there that what was deleted was actually the database containing the master copies of the characters, sets, animation, etc. rather than the movie itself. Of course, once again, that makes you wonder how it is that no one else had a simple backup. You'd think such a thing would be backed up in dozens of places around the globe for safe keeping...


Reader Comments (rss)

(Flattened / Threaded)

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 5:07am

    It's called Godzilla-proofing. Know it. Live it.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Lazureus, May 18th, 2012 @ 5:16am

    And this is why the Canadian/American governments should stop trying to make the act of backing up videos illegal.

    You could say it's a different thing all together, but it really isn't.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    silverscarcat (profile), May 18th, 2012 @ 5:22am

    There's a trope for that!

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 5:23am

    Shame on this horrible woman for making an illegal backup of this movie. By doing so, she cost so many workers over a year of salary for restoring it, all for her own selfish greed. Pathetic. When will these freetards think of the workers and the customers?

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 5:29am

    This story gave me a woody

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 5:29am

    so, no back ups allowed, even when your copy of the movie is deleted means all the work for the whole thing is lost. good strategy, guys! i suppose that the one 'illegal' copy, regardless of who had it and that it saved some serious arse kicking from happening and a hell of a lot of money is irrelevant? as it applies to any download by anyone for any reason, all i can say is 'but but but piracy'!

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Hans B PUFAL (profile), May 18th, 2012 @ 5:53am

    Reminds me of ....

    Some decades ago I was called to a customer site, a bank, to diagnose a computer problem. On my arrival early in the morning I noted a certain panic in the air. On querying my hosts I was told that there had been an "issue" the previous night and that they were trying, unsuccessfully, to recover data from backup tapes. The process was failing and panic ensued.

    Though this was not the problem I had been called on to investigate, I asked some probing questions, made a short phone call, and provided the answer, much to the customer's relief.

    What I found was that for months if not years the customer had been performing backups of indexed sequential files, that is data files with associated index files, without once verifying that the backed-up data could be recovered. On the first occasion of a problem requiring such a recovery they discovered that they just did not work.

    The answer? Simply recreate the index files from the data. For efficiency reasons (this was a LONG time ago) the index files referenced the data files by physical disk adresses. When the backup tapes were restored the data was of course no longer at the original place on the disk and the index files were useless. A simple procedure to recreate the index files solved the problem.

    Clearly whoever had designed that system had never tested a recovery, nor read the documentation which clearly stated the issue and its simple solution.

    So here is a case of making backups, but then finding them flawed when needed.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 18th, 2012 @ 6:00am

      Re: Reminds me of ....

      That's why, in the IT world, you ALWAYS do a "dry run" when you want to deploy something, and you monitor the heck out of critical systems.

       

      reply to this | link to this | view in chronology ]

  •  
    identicon
    john, May 18th, 2012 @ 6:30am

    Yeah good point, you think that they would be more secure, whats to stop a employee from "saving" a film on their home computer then selling it to someone who would release it online or something? That must have been a wake up call.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Rich Kulawiec, May 18th, 2012 @ 6:30am

    Two notes on backups

    1. Everyone who has worked in computing for any period of time has their own backup horror story. I'll spare you mine, but note that as a general observation, large organizations/corporations tend to opt for incredibly expensive, incredibly complex, incredibly overblown backup "solutions" sold to them by vendors rather than using the stock, well-tested, reliable tools that they already have. (e.g., "why should we use dump, which is open-source/reliable/portable/tested/proven/efficient/etc., when we could drop $40K on closed-source/proprietary/non-portable/slow/bulky software from a vendor?"

    Okay, okay, one comment: in over 30 years of working in the field, the second-worst product I have ever had the misfortune to deal with is Legato (now EMC) NetWorker.

    2. Hollywood has a massive backup and archiving problem. How do we know? Because they keep telling us about it. There are a series of self-promoting commercials that they run in theaters before movies, in which they talk about all of the old films that are slowly decaying in their canisters in vast warehouses, and how terrible this is, and how badly they need charitable contributions from the public to save these treasures of cinema before they erode into dust, etc.

    Let's skip the irony of Hollywood begging for money while they're paying professional liar Chris Dodd millions and get to the technical point: the easiest and cheapest way to preserve all of these would be to back them up to the Internet. Yes, there's a one-time expense of cleaning up the analog versions and then digitizing them at high resolution, but once that's done, all the copies are free. There's no need for a data center or elaborate IT infrastructure: put 'em on BitTorrent and let the world do the work. Or give copies to the Internet Archive. Whatever -- the point is that once we get past the analog issues, the only reason that this is a problem is that they made it a problem by refusing to surrender control.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      saulgoode (profile), May 18th, 2012 @ 6:38am

      Re: Two notes on backups

      "Real Men don't make backups. They upload it via ftp and let the world mirror it." - Linus Torvalds

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        You, May 18th, 2012 @ 7:08am

        Re: Re: Two notes on backups

        I should make stickers for hard drives/USB drives/disks:
        Proud to be a backup

        Let people print 'em for free and let them go wild on their media collection of choice (and I'll never lose the sticker image either). Go ahead, add some character to that aging collection of DVDs, they'll all mold over eventually!

         

        reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 7:02am

    What I suspect is that she was copying the rendered footage. If the footage was rendered at a resolution and rate fitting to DVD spec, that'd put the raw footage at around 3GB to 4GB for a full 90min, which just might fit on the 10GB HDD that were available back then on a laptop computer (remember how small OSes were back then). Even losing just the rendered raw footage (or even processed footage), would be a massive setback. It takes a long time across a lot of very powerful computers to render film quality footage. If it was processed footage then it's even more valuable as that takes a lot of man hours of post fx to make raw footage presentable to a consumer audience.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Jeremy, May 18th, 2012 @ 7:31am

      Re:

      Nah, she says she made the copies because she needed to "work" from home. You don't work on rendered footage, it's basically done. Also, the other guy doesn't say that the movie was disappearing from the drive, he says characters or parts of characters were vanishing.

      I'm guessing she's a texture artist who was making backups of the character/scene models and she was copying them home so she could see final result with her artwork included.

      Though, when I think about more, does Toy story even have complicated textures? I have a strong memory of mostly solid colors.

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        polypusher, May 18th, 2012 @ 9:40am

        Re: Re:

        Toy Story 2 had very complicated shaders for it's time. They don't just use solid-color textures. Even shaders that appear solid in texture to you have very subtle variations that insinuate reality. This is part of what made and makes Pixar so special.

         

        reply to this | link to this | view in chronology ]

      •  
        identicon
        I-Blz, May 18th, 2012 @ 9:46am

        Re: Re:

        Not so much complicated as detailed. In Toy Story 2, Woody's arm has athread pattern, and the texture works with this. The stuffing in his arm is also very detailed. Just a few examples

         

        reply to this | link to this | view in chronology ]

  •  
    icon
    W Klink (profile), May 18th, 2012 @ 7:07am

    Backups

    Everyone knows to make backups, but not everyone thinks to validate and test restoring them.

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Overcast (profile), May 18th, 2012 @ 8:18am

    Human complacency and dependance on computers is scarey sometimes...

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    aldestrawk (profile), May 18th, 2012 @ 8:34am

    a retelling by Oren Jacob

    Oren Jacob, the Pixar director featured in the animation, has made a comment on the Quora post that explains things in much more detail. The narration and animation was telling a story, as in storytelling. Despite the 99% true caption at the end, a lot of details were left out which misrepresented what had happened. Still, it was a fun tale for anyone who had dealt with backup problems. Oren Jacob's retelling in the comment makes it much more realistic and believable.
    The terabytes level of data came from whoever posted the video on Quora. The video itself never mentions the actual amount of data lost or the total amount the raw files represent. Oren says, vaguely, that it was much less than a terabyte. There were backups! The last one was from two days previous to the delete event. The backup was flawed in that it produced files that when tested, by rendering,
    exhibited errors. They ended up patching a two-month old backup together with the home computer version (two weeks old). This was labor intensive as some 30k files had to be individually checked.
    The moral of the story. Firstly, always test a restore at some point when implementing a backup system. Secondly, don't panic! Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer. Thirdly, don't panic! Despite, somehow, deleting a large set of files these can be recovered apart from a backup system. Deleting files, under Linux as well as just about any OS, only involves deleting the directory entries. There is software which can recover those files as long as further use of the computer system doesn't end up overwriting what is now free space.

     

    reply to this | link to this | view in chronology ]

    •  
      identicon
      Mason Wheeler, May 18th, 2012 @ 10:01am

      Re: a retelling by Oren Jacob

      Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer.


      What's worse? Corrupting some files or deleting all files?

       

      reply to this | link to this | view in chronology ]

      •  
        icon
        aldestrawk (profile), May 18th, 2012 @ 10:38am

        Re: Re: a retelling by Oren Jacob

        In this case they were not dealing with unknown malware that was steadily erasing the system as they watched. There was, apparently, a delete event at a single point in time that had repercussions that made things disappear while people worked on the movie. I'll bet things disappeared when whatever editing was being done required a file to be refreshed. A refresh operation would make the related object disappear when the underlying file was no longer available. Apart from the set of files that had already been deleted, more files could have been corrupted when the computer was unplugged. Having said that, this occurred in 1999 when they were probably using the Ext2 filesystem under Linux. These days most everyone uses a filesystem that includes journaling which protects against corruption that may occur when a computer loses power. Ext3 is a journaling filesystem and was introduced in 2001.

        In 1998 I had to rebuild my entire home computer system. A power glitch introduced corruption in a Windows 95 system file and use of a Norton recovery tool rendered the entire disk into a handful of unusable files. It took me ten hours to rebuild the OS and re-install all the added hardware, software, and copy personal files from backup floppies. The next day I went out and bought a UPS. Nowadays, sometimes the UPS for one of my computers will fail during one of the three dozen power outages a year I get here. I no longer have problems with that because of journaling.

         

        reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 10:27am

    who the hell punched in RM* to begin with?

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    Danny (profile), May 18th, 2012 @ 10:49am

    I've gotta story like this too

    Ive posted in athe past on Techdirt that I used to work for Ticketmaster. The is an interesting TM story that I don't think ever made it into the public, so I will do it now.

    Back in the 1980s each TM city was on an independent computer system (PDP unibus systems with RM05 or CDC9766 disk drives. The drives were fixed removable boxes about the size of a washing machine, the removable disk platters about the size of the proverbial breadbox. Each platter held 256mb formatted.

    Each city had itts own operations policies, but generally, the systems ran with mirrored drives, the database was backed up every night, archival copies were made monthly. In Chicago, where I worked, we did not have offsite backup in the 1980s. The Bay Area had the most interesting system for offsite backup.

    The Bay Area BASS operation, bought by TM in the mid 1980s, had a deal with a taxi driver. They would make their nightly backup copies in house, and make an extra copy on a spare disk platter. Tis cabbie would come by the office about 2am each morning, and they'd put the spare disk platter in his trunk, swapping it for the previous day's copy that had been his truck for 24 hours. So, for the cost of about two platters ($700 at the time) and whatever cash they'd pay the cabbie, they had a mobile offsite copy of their database circulating the Bay Area at all times.

    When the World Series earthquake hit in October 1988, the TM office in downtown Oakland was badly damaged. The only copy of the database that survived was the copy in the taxi cab.

    That incident led TM corporate to establish much more sophisticated and redundant data redundancy policies.

     

    reply to this | link to this | view in chronology ]

    •  
      icon
      aldestrawk (profile), May 18th, 2012 @ 11:30am

      Re: I've gotta story like this too

      I like that story. Not that it matters anymore, but taxi cab storage was probably a bad idea. The disks were undoubtedly the "Winchester" type and when powered down the head would be parked on a "landing strip". Still, subjecting these drives to jolts from a taxi riding over bumps in the road could damage the head or cause it to be misaligned. You would have known though it that actually turned out to be a problem. Also, I wouldn't trust a taxi driver with the company database. Although, that is probably due to an unreasonable bias towards cab drivers. I won't mention the numerous arguments with them (not in the U.S.) over fares and the one physical fight with a driver who nearly ran me down while I was walking.

       

      reply to this | link to this | view in chronology ]

      •  
        identicon
        Huw Davies, May 19th, 2012 @ 1:20am

        Re: Re: I've gotta story like this too

        RM05s are removable pack drives. The heads stay in the washing machine size unit - all you remove are the platters.

         

        reply to this | link to this | view in chronology ]

    •  
      identicon
      Anonymous Coward, May 18th, 2012 @ 12:29pm

      Re: I've gotta story like this too

      Very interesting story, Danny...

      & I Don't mean to nit-pick but the quake was Oct. '89. After reading the sloppy coding entries, I've been paying attention 2 details way more than I'd like to. I've been seeing 0's & 1's in my sleep. Damn nerds

       

      reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 12:19pm

    So, making copies of an entire copyrighted film isn't Piracy?

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 4:18pm

    This is why limiting copying of a work isn't "protecting" it.

    It may protect to company's monetary interest in it, but once the product is out there it hinders the growth of the work and in some cases cause it to be lost forever if it isn't "stolen" by those damn dirty pirates

     

    reply to this | link to this | view in chronology ]

  •  
    icon
    That One Guy (profile), May 18th, 2012 @ 5:00pm

    What I want to know is this...

    She copied bits of a movie to her home system... how hard did they have to pull in the leashes to keep Disney's lawyers from suing her to infinity and beyond after she admitted she'd done so(never mind the fact that he doing so saved them apparently years of work...)?

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 11:05pm

    Mike, 'deleting' a file does not delete it from HDD unless it's overwritten.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Anonymous Coward, May 18th, 2012 @ 11:06pm

    BTW once again "Linux" is mentioned.

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    Legetųj, Nov 13th, 2012 @ 9:06pm

    Fake

    I really don belive this, cant be :)

     

    reply to this | link to this | view in chronology ]

  •  
    identicon
    HisNameWasDoug, Mar 28th, 2014 @ 6:19pm

    Not nearly as exciting but i saved my boss's job one year when he accidentally wiped our database. I had been making copies to save my work in case I quit. I hated this boss more than anything but I couldn't find it in me to allow him to be fired, so I lied and said I had been making regular backups.

     

    reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here
Get Techdirt’s Daily Email
Save me a cookie
  • Note: A CRLF will be replaced by a break tag (<br>), all other allowable HTML will remain intact
  • Allowed HTML Tags: <b> <i> <a> <em> <br> <strong> <blockquote> <hr> <tt>
Follow Techdirt
A word from our sponsors...
Essential Reading
Techdirt Reading List
Techdirt Insider Chat
A word from our sponsors...
Recent Stories
A word from our sponsors...

Close

Email This