When a poor backup almost deprives us from Toy Story 2
When a Poor Backup Almost Deprives us of Toy Story 2
Despite that at first they tried to make us believe it was a myth, later Pixar itself recognized they had severe problems with Toy Story 2 production.
Up to this point, everyone has watched already Toy Story 2, or at least knows about its existence. A movie that was nominated for the Oscars, but that it came to the verge of failing just before its premiere.
It was in 1998 when Pixar was working on the movie "A Bug's Life" while at the same time they worked on "Toy Story 2". The team of workers of the company for those productions was huge, almost 400 people and they worked simoultaneously.
Everything happened very fast, as explained by Oren Jacob to different media, the associated technical director of Toy Story 2 at the time. “We were gathered at Larry Cutler's office together with Larry Aupperle looking at the directory where the elements for Woody's character were stored, such as his hat, the boots, other characters, etc. And then all of a sudden they started disappearing without any trace".
Everything worked as usual until a member of the team executed by mistake a special command; the rm* command, which was used to delete files from the main root
At that time, all those involved in a project at Pixar (and in other companies) had access to the whole project itself, including the key files. To that end, Pixar had a single central server such as Unix where all files were saved. Everything worked as usual until a member of the team executed by mistake a special command; the rm* command which was used to delete files from the main root.
Right after realizing what was happening they rushed to unplug the whole system, but the damages were unavoidable: 90% of the movie had already disappeared.
Finding the backup
Right away the team started trying to recover the content of the backup, but they found the second problem. When starting the project they calculated a space for the backup, 4GB. A space that ended up being insufficient, since when they needed to resort to it, the movie took 10GB of space, so the content that could be recovered was not valid.
Those were times of absolute panic at Pixar's offices, since the resulting work of a whole year had just disappeared and they had to do the impossible to make it on time to the premiere date.
Then by chance, in a meeting to discuss any possible ways to come to a solution, Galyn Susman, the supervisor technical director of Toy Story 2 at that time mentioned the possibility of having a copy of the movie at her home. After becoming a mother, Susman had the chance of working from home, so Pixar provided her with a working station from Sillion Graphics at home. The computer was connected to the local network, she worked and received the incremental updates through her Internet connection through RDSI. Although the last update she received was from a couple weeks ago, it was far more than what they had at that moment.
Report: Reasons to acquire an external backup solution
Control, security and compliance
Galyn Susman, the supervisor technical director of Toy Story 2 at that time mentioned the possibility of having a copy of the movie at her home
With the hope of having the solution to the problem, Jacob and Susman got into the car heading towards Galyn's. Upon their arrival, the carefully took and packaged the computer, wrapping it up with blankets and using the seat belts to fasten it to the back of the car. All the efforts the whole Pixar team had made were travelling by car to the study! Once they got there, a team of eight people moved the computer to the machine room. Now they had to trust that it would switch on and work once connected to the power supply, and that was in fact the case. They connected the computer to the network and they immediately copied the whole disk. They had saved the movie!
However, the recovery process did not end there. They managed to verify through the computer around 70.000 files but there were still 30.000 left to verify and they had to do so manually. “Around 12 people took turns working from Friday until Monday morning, non-stop, taking turns for having lunch and using sleeping bags” says Jacob to The Next Week. After that, they put back together again almost the whole project.
Who's responsible for it?
A keypoint to be highlighted was the ability of the whole team to work towards the same direction to save the movie. “Although it's true that there was a meeting where it was said who was the person at fault for the deletion, the discussion quickly moved forward towards how to solve the problem” explained Jacob. “We had to recover the movie and we had no time to waste looking for those who should be blamed”.
When there's a disaster such as this one, the business staff, from technical directors to staff had to trust each other to solve the issue. After that the team made sure to modify their backup strategy to make sure nothing like that would ever happen again.
The importance of having a good backup
Making a backup is not only an action that's always a good choice just in case, but it is almost required for businesses. At least for being certain that they're able to keep the company working whatever happens.
You never know when you can be affected by a ransomware cyberattack or when a human error could spoil the work you've been doing for a long time. The difference between suffering a small issue or losing everything can be saved by having your own backup.
Even if when all of this took place Pixar claimed that it was merely a rumor, they later recognized it and even made a short film about the story.
