How Toy Story 2 Almost Got Deleted… Except That One Person Made A Home Backup

from the did-it-break-the-rules? dept

Here’s a random story, found via Kottke, highlighting how Pixar came very close to losing a very large portion of Toy Story 2, because someone did an rm * (non geek: “remove all” command). And that’s when they realized that their backups hadn’t been working for a month. Then, the technical director of the film noted that, because she wanted to see her family and kids, she had been making copies of the entire film and transferring it to her home computer. After a careful trip from the Pixar offices to her home and back, they discovered that, indeed, most of the film was saved:


Now, mostly, this is just an amusing little anecdote, but two things struck me:

  1. How in the world do they not have more “official” backups of something as major as Toy Story 2. In the clip they admit that it was potentially 20 to 30 man-years of work that may have been lost. It makes no sense to me that this would include a single backup system.
  2. I wonder if the copy, made by technical director Galyn Susman, was outside of corporate policy. You would have to imagine that at a place like Pixar, there were significant concerns about things “getting out,” and so the policy likely wouldn’t have looked all that kindly on copies being used on home computers.

The Mythbusters folks wonder if this story was a little over-dramatized, and others have wondered how the technical director would have “multiple terabytes of source material” on her home computer back in 1999. That resulted in an explanation from someone who was there that what was deleted was actually the database containing the master copies of the characters, sets, animation, etc. rather than the movie itself. Of course, once again, that makes you wonder how it is that no one else had a simple backup. You’d think such a thing would be backed up in dozens of places around the globe for safe keeping…

Filed Under: , ,
Companies: pixar

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “How Toy Story 2 Almost Got Deleted… Except That One Person Made A Home Backup”

Subscribe: RSS Leave a comment
49 Comments
Anonymous Coward says:

so, no back ups allowed, even when your copy of the movie is deleted means all the work for the whole thing is lost. good strategy, guys! i suppose that the one ‘illegal’ copy, regardless of who had it and that it saved some serious arse kicking from happening and a hell of a lot of money is irrelevant? as it applies to any download by anyone for any reason, all i can say is ‘but but but piracy’!

Hans B PUFAL (profile) says:

Reminds me of ....

Some decades ago I was called to a customer site, a bank, to diagnose a computer problem. On my arrival early in the morning I noted a certain panic in the air. On querying my hosts I was told that there had been an “issue” the previous night and that they were trying, unsuccessfully, to recover data from backup tapes. The process was failing and panic ensued.

Though this was not the problem I had been called on to investigate, I asked some probing questions, made a short phone call, and provided the answer, much to the customer’s relief.

What I found was that for months if not years the customer had been performing backups of indexed sequential files, that is data files with associated index files, without once verifying that the backed-up data could be recovered. On the first occasion of a problem requiring such a recovery they discovered that they just did not work.

The answer? Simply recreate the index files from the data. For efficiency reasons (this was a LONG time ago) the index files referenced the data files by physical disk adresses. When the backup tapes were restored the data was of course no longer at the original place on the disk and the index files were useless. A simple procedure to recreate the index files solved the problem.

Clearly whoever had designed that system had never tested a recovery, nor read the documentation which clearly stated the issue and its simple solution.

So here is a case of making backups, but then finding them flawed when needed.

Rich Kulawiec (profile) says:

Two notes on backups

1. Everyone who has worked in computing for any period of time has their own backup horror story. I’ll spare you mine, but note that as a general observation, large organizations/corporations tend to opt for incredibly expensive, incredibly complex, incredibly overblown backup “solutions” sold to them by vendors rather than using the stock, well-tested, reliable tools that they already have. (e.g., “why should we use dump, which is open-source/reliable/portable/tested/proven/efficient/etc., when we could drop $40K on closed-source/proprietary/non-portable/slow/bulky software from a vendor?”

Okay, okay, one comment: in over 30 years of working in the field, the second-worst product I have ever had the misfortune to deal with is Legato (now EMC) NetWorker.

2. Hollywood has a massive backup and archiving problem. How do we know? Because they keep telling us about it. There are a series of self-promoting commercials that they run in theaters before movies, in which they talk about all of the old films that are slowly decaying in their canisters in vast warehouses, and how terrible this is, and how badly they need charitable contributions from the public to save these treasures of cinema before they erode into dust, etc.

Let’s skip the irony of Hollywood begging for money while they’re paying professional liar Chris Dodd millions and get to the technical point: the easiest and cheapest way to preserve all of these would be to back them up to the Internet. Yes, there’s a one-time expense of cleaning up the analog versions and then digitizing them at high resolution, but once that’s done, all the copies are free. There’s no need for a data center or elaborate IT infrastructure: put ’em on BitTorrent and let the world do the work. Or give copies to the Internet Archive. Whatever — the point is that once we get past the analog issues, the only reason that this is a problem is that they made it a problem by refusing to surrender control.

You says:

Re: Re: Two notes on backups

I should make stickers for hard drives/USB drives/disks:
Proud to be a backup

Let people print ’em for free and let them go wild on their media collection of choice (and I’ll never lose the sticker image either). Go ahead, add some character to that aging collection of DVDs, they’ll all mold over eventually!

Anonymous Coward says:

What I suspect is that she was copying the rendered footage. If the footage was rendered at a resolution and rate fitting to DVD spec, that’d put the raw footage at around 3GB to 4GB for a full 90min, which just might fit on the 10GB HDD that were available back then on a laptop computer (remember how small OSes were back then). Even losing just the rendered raw footage (or even processed footage), would be a massive setback. It takes a long time across a lot of very powerful computers to render film quality footage. If it was processed footage then it’s even more valuable as that takes a lot of man hours of post fx to make raw footage presentable to a consumer audience.

Jeremy says:

Re: Re:

Nah, she says she made the copies because she needed to “work” from home. You don’t work on rendered footage, it’s basically done. Also, the other guy doesn’t say that the movie was disappearing from the drive, he says characters or parts of characters were vanishing.

I’m guessing she’s a texture artist who was making backups of the character/scene models and she was copying them home so she could see final result with her artwork included.

Though, when I think about more, does Toy story even have complicated textures? I have a strong memory of mostly solid colors.

aldestrawk says:

a retelling by Oren Jacob

Oren Jacob, the Pixar director featured in the animation, has made a comment on the Quora post that explains things in much more detail. The narration and animation was telling a story, as in storytelling. Despite the 99% true caption at the end, a lot of details were left out which misrepresented what had happened. Still, it was a fun tale for anyone who had dealt with backup problems. Oren Jacob’s retelling in the comment makes it much more realistic and believable.
The terabytes level of data came from whoever posted the video on Quora. The video itself never mentions the actual amount of data lost or the total amount the raw files represent. Oren says, vaguely, that it was much less than a terabyte. There were backups! The last one was from two days previous to the delete event. The backup was flawed in that it produced files that when tested, by rendering,
exhibited errors. They ended up patching a two-month old backup together with the home computer version (two weeks old). This was labor intensive as some 30k files had to be individually checked.
The moral of the story. Firstly, always test a restore at some point when implementing a backup system. Secondly, don’t panic! Panic can lead to further problems. They could well have introduced corruption in files by abruptly unplugging the computer. Thirdly, don’t panic! Despite, somehow, deleting a large set of files these can be recovered apart from a backup system. Deleting files, under Linux as well as just about any OS, only involves deleting the directory entries. There is software which can recover those files as long as further use of the computer system doesn’t end up overwriting what is now free space.

aldestrawk says:

Re: Re: a retelling by Oren Jacob

In this case they were not dealing with unknown malware that was steadily erasing the system as they watched. There was, apparently, a delete event at a single point in time that had repercussions that made things disappear while people worked on the movie. I’ll bet things disappeared when whatever editing was being done required a file to be refreshed. A refresh operation would make the related object disappear when the underlying file was no longer available. Apart from the set of files that had already been deleted, more files could have been corrupted when the computer was unplugged. Having said that, this occurred in 1999 when they were probably using the Ext2 filesystem under Linux. These days most everyone uses a filesystem that includes journaling which protects against corruption that may occur when a computer loses power. Ext3 is a journaling filesystem and was introduced in 2001.

In 1998 I had to rebuild my entire home computer system. A power glitch introduced corruption in a Windows 95 system file and use of a Norton recovery tool rendered the entire disk into a handful of unusable files. It took me ten hours to rebuild the OS and re-install all the added hardware, software, and copy personal files from backup floppies. The next day I went out and bought a UPS. Nowadays, sometimes the UPS for one of my computers will fail during one of the three dozen power outages a year I get here. I no longer have problems with that because of journaling.

Danny (profile) says:

I've gotta story like this too

Ive posted in athe past on Techdirt that I used to work for Ticketmaster. The is an interesting TM story that I don’t think ever made it into the public, so I will do it now.

Back in the 1980s each TM city was on an independent computer system (PDP unibus systems with RM05 or CDC9766 disk drives. The drives were fixed removable boxes about the size of a washing machine, the removable disk platters about the size of the proverbial breadbox. Each platter held 256mb formatted.

Each city had itts own operations policies, but generally, the systems ran with mirrored drives, the database was backed up every night, archival copies were made monthly. In Chicago, where I worked, we did not have offsite backup in the 1980s. The Bay Area had the most interesting system for offsite backup.

The Bay Area BASS operation, bought by TM in the mid 1980s, had a deal with a taxi driver. They would make their nightly backup copies in house, and make an extra copy on a spare disk platter. Tis cabbie would come by the office about 2am each morning, and they’d put the spare disk platter in his trunk, swapping it for the previous day’s copy that had been his truck for 24 hours. So, for the cost of about two platters ($700 at the time) and whatever cash they’d pay the cabbie, they had a mobile offsite copy of their database circulating the Bay Area at all times.

When the World Series earthquake hit in October 1988, the TM office in downtown Oakland was badly damaged. The only copy of the database that survived was the copy in the taxi cab.

That incident led TM corporate to establish much more sophisticated and redundant data redundancy policies.

aldestrawk says:

Re: I've gotta story like this too

I like that story. Not that it matters anymore, but taxi cab storage was probably a bad idea. The disks were undoubtedly the “Winchester” type and when powered down the head would be parked on a “landing strip”. Still, subjecting these drives to jolts from a taxi riding over bumps in the road could damage the head or cause it to be misaligned. You would have known though it that actually turned out to be a problem. Also, I wouldn’t trust a taxi driver with the company database. Although, that is probably due to an unreasonable bias towards cab drivers. I won’t mention the numerous arguments with them (not in the U.S.) over fares and the one physical fight with a driver who nearly ran me down while I was walking.

DELETE TOY STORY 3 MOVIE (user link) says:

TOY STORY 1 YES AND TOY STORY 2 YES NOW MOVIES

DELETE TOY STORY 3 MOVIE
YES
NOW
NO TOY STORY 3 MOVIE DELETE NOW
YES
AND ANDY BOY SMALL YES
ANDY NO MAN BIG DELETE
NOW YES
GO NOW
TOY STORY 1 YES AND TOY STORY 2 YES
NOW
ANDY BOY SMALL YES TOYS
TOYS ANDY TOY STORY 1 YES
TOYS ANDY TOY STORY 2 YES
NOW YES
GO NOW
DELETE TOY STORY 3 ANDY MAN
ANDY NO MAN BIG DELETE
ANDY BOY SMALL YES TOYS
GO NOW
DELETE TOY STORY 3 MOVIE
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
NOW
NETWORK DELETE TOY STORY 3 MOVIE
TV CHANNEL DELETE TOY STORY 3 MOVIE
ANDY BOY SMALL YES TOYS
ANDY LOVE MUMMY WOMAN
ANDY BOY YES
TOY STORY 1 MOVIE YES
TOY STORY 2 MOVIE YES
GO NOW
YES
OK

Add Your Comment

Your email address will not be published.

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...