It’s that time of year again, when I look to the digital assets created in the last year, and consider the data archiving needed. I know from experience the utter terror and anguish caused by a data loss or a hard drive crash – I had a bad on in the late nineties, and since then I have been relatively vigilant about backup up my data regularly. But computers being what they are, I always get lulled into a false sense of security, and I find it’s been too long since my last backup. It just takes a minor system glitch to remind me though, back up your data NOW!
So this weekend is archive weekend.
And already a minor catastrophe has struck. I bought two, 2 TB USB drives for backups, and one has failed on plugging in – a manufacturing fault, but it makes me nervous when a drive fails as it reminds me that I need at least 2 copies of everything. And due to this failure, right now I have only ONE copy of one of my major projects. The replacement drive will arrive midweek, but until then, I am twitching a little!
I heard via Twitter recently of the woes of fellow Brit film maker, Allin Kempthorne, oh his cult horror parody, ‘The Vampires of Bloody Island’, how he suffered two drive failures and very, very nearly lost his entire film. Luckily he managed to reconstruct it but it was a close call.
For me, data falls into two categories.
Firstly it’s just standard computer files and data – emails, word docs, scripts, graphics etc. That stuff I try and backup every few weeks. I should really do it every day, I know! All my daily data is stored on a small USB drive that I carry around with me, and when I back it up, I do that to a USB drive on my home PC. I use a free little PC utility called SyncBack to automate the process – you can download it here. It works very well for my needs and is free.
The second category is the video media for any editing work I have done. And this stuff is big. Really big.
The Gone Fishing workshop, which I have recently done some extra work on, is now running at 1.5TB in size, and it took pretty much a day to make a copy of the whole project from one external USB drive to another.
It really is very worrying how insecure data on a drive actually is, compared to data on a tape for instance.
Below are a bunch of random thoughts on this issue (which I wrote down as I watched the drives move from 7% backed up to 8% backed up…)
1. Data on a drive is subject to a drive failure that can wipe out everything, not just a snagged tape where you loose a moment or two.
2. Data on a drive is much harder to keep track of than tapes. It just is. I know it shouldn’t be, but it is. You can’t write on a drive and toss in a draw in the same way that you can with a tape.
3. We all assume our data is secure. That is until we lose it. I can’t tell you how often I have helped a film maker and asked… ‘where is your backup…?’ only to get the blank expression.
4. Apple and FCP users are more vulnerable than they think. As there is very little threat from viruses, and because Apple users tend to think of their operating system as being superior to Windows and 100% secure, they forget that drive failure is operating system agnostic. Drive failure just does not care if it’s Apple or PC.
5. FCP users are also vulnerable IF they do not take care to track media within in a project. The very ‘open’ way that FCP works invites users to ‘loose’ their data on a long project. Avid forces you to use its own media management process and thus, it’s harder to lose media (though you still need to use your brain when digitising and make sure you know where everything is going).
6. I have a separate laptop for video editing only. I am amazed at how most film makers use their editing laptop for all manner of other stuff and then wonder why, six months into a project, they run into problems. On any big project, set up the system and DON’T upgrade the operating system or editing software until the project is complete.
7. Video media is HUGE. No matter the film project, by the end of it, I have needed nearly twice as much data storage than I first imagined. Don’t underestimate this, and don’t underestimate the time needed to copy, backup and transfer these files.
8. I have switched to an external desktop USB box that accepts E-IDE bare drives, making it cheaper to buy the drives and easy to switch.
9. I regularly replace my USB memory stick, roughly once a year, as I know these devices have a limited amount or read / write before they fail. So I replace before I ever get close to failure. Usually, the new one costs about £10 and is twice as big as the old one.
10. I also archive my email and Outlook daily – these files are, in my experience, the ones most likely to be damaged or corrupted.
11. I really worry about friends with Mac’s who really believe their computers are fail safe and secure. They just aren’t. Backup your data now (never mind how secure the OS can be, what about a spilled cup of coffee or a laptop thief?)
12. Online backups of data is a great idea, but for me, impractical as my data storage needs are so great as to make it unviable. I have tried several times and abandones each time as its SOOOOO slow.
13. I know I should build a home and office Network Storage Device system, but being an indie film maker, that would mean I am IT support, and I have enough on my plate already! So it’s swapping USB drives for now.
14. When doing post on ‘Gone Fishing’, the raw scans (when archived) ended up filling a half terabyte drive when we completed the grade at Midnight Transfer. That was a 13 minute film and it took their tech guys a whole night to transfer the media (note – format your drives with NTFS and NOT FAT32 as, if memory serves, FAT32 only supports up to 64,000 files, which is not enough if you are saving your movie as individual frames – if I am wrong on this, please correct me – I do recall this being a problem with Gone Fishing and caused us to need a second drive in order to copy all the frames).
So ask yourself, is your data secure? If not, add that 2TB USB drive to Santas list… They cost about £100 now, and 2 TB is a lot of empty space!
Onwards and upwards!
Chris Jones, Film Maker and Author
Have you ever considered purchasing a Drobo? They’re great for local backups, and are incredibly simple to set up and use. Great for indie film-makers who want RAID-like protection, but don’t have a degree in computer science.
Also, have you ever considered Carbonite for online backups? Obviously you can’t back up all your media there – but you can at least backup all your emails, documents and Avid/FCP project files.
Back a few years ago when we were shooting everything on HDV, we actually backed up all our FCP projects to DV tape as well. Very good archiving solution!
Now days, we just have a couple of SATA drives for each individual project, plus a copy on a Lacie Rugged, plus a backup on the Drobo, as well as a daily copy of the project file stored up in “the cloud” (iDisk in our case).
LTO is really the way to go – but it’s a tad expensive and time consuming if you’re a real low-budget film-maker.