by Mike Shea on 14 July 2006
Note: My backup plans have changed since I wrote this. You can read about my new procedures in the article Your Personal Digital Archive.
For a while now I've become more and more obsessed with my personal data protection plan. I seem to swim in piles of USB drives and every three months I buy a new hard drive to add to my ever-widening circle of backup external drives. My latest drive, a Western Digital My Book 500GB Premium will go into rotation next week. Yet backup hardware and software is no good without a plan and procedure. The following outlines my current procedure:
What Causes Data Loss?
From Americal Data Recovery:
40% Hardware Failure 29% Human Error 13% Software Corruption 9% Theft 6% Computer Viruses 3% Hardware Destruction
What can help prevent these factors?
40% Hardware Failure - Redundant Drives - Off-site backup
29% Human Error - Automated backups - non-deleting backups - version-controlled backups - timestamped backups
13% Software Corruption - redundant backups - off-site backups - automated device checking
9% Theft - Off-site backups
6% Computer Viruses - Always-on Virus protection - automated virus protection - disconnected backups
3% Hardware Destruction - redundant drives - off-site backup
Backup Philosophy:
The backup system should be as simple as possible.
The backup system should be automated.
Restores should be tested.
Restores should require no single piece of software.
Raid-1 mirrors protect immediately but not against accidental deletes, overwrites, or software bugs. It will pass bad data to the mirror.
Raid arrays require their own hardware and formatting. The hardware that builds the raids becomes a single point of failure.
Off-site copies are critical to protect against natural disaster, theft, viruses, and other issues that may directly affect local attached storage devices. Off-site copies should be automated.
What do I want to save?
Stuff I created (~50 megs uncompressed with 5 megs a year increase) - my web sites (critical) - my photos (important) - my stories (critical)
Stuff I collected: (~22gb with 2gb a year) - my audiobooks (critical) (16gb) - my music (important) (6gb) - my programs (not important) (200 megs) - my top 50 dvds (medium importance) (300+gb)
Stuff I want to save: - Everquest (~5gb 1gb per year (5gb) - Warcraft (~5gb 1gb per year) (6gb)
Backup routine: 1 internal 400gb drive 1 external 500gb drive 1 remote 80gb drive with everything but DVDs swapped in and out every 30 days.
Nightly HD Backup Solution:
Backup every morning at around 3am using xcopy to copy all of my data to a large local external hard disk, "My Documents" (no DVDs) to a smaller portable hard disc, and the Lifebackup two USB thumb drives.
Strengths: - It will not remove accidentally deleted files. - The backup drive can be accessed directly requiring no extra software. - Every file is uncompressed and will degrade gracefully unless a hardware problem destroys the drive.
Weaknesses: - It will pass along accidentally overwritten files. - It will not protect files until up to 24 hours later. - Data that changes more often will not be backed up. - Files that are created and accidentally overwritten within a 24 hour block will not be protected.
Available devices:
Online Drives: 74GB Raptor HD (Drive D:) for Applications 400GB Internal drive (Drive C:) for My Documents, My Movies, and a backup of Everquest and World of Warcraft 500GB Western Digital My Book (Drive ?) for the nightly backup of My Documents, My Movies, Everquest, and Warcraft
Off-line backups: 80GB Lacie portable drive for My Documents, Everquest, and Warcraft (no movies) 80GB External CompUSA Drive for off-site copies of My Documents, Everquest, and Warcraft
Lifebackup Flash drives: Cruiser 512mb Flash Drive for Lifebackup 2gb Flash Drive for Lifebackup 512 SD USB Flash card for Lifebackup
Procedures:
The Lifebackup
Every evening, Raitax archives my online Lifebackup. This includes HTML web pages, xml files, selected images, stories, and other critical items. These are put into a tarball at http://mikeshea.net/lifebackup.tar.gz.
"tar -cvzf /usr/home/mshea/public_html/mikeshea/lifebackup.tar.gz pictures/*
liquidtheater/.html loral/.html loral/everquest.css mikesheanet/*.html
mikeshea.net/*.xml mikesheanet/notes.txt mikesheanet/vrenna_icon.jpg
mikesheanet/vrenna_cover_med.png mikesheanet/style.css .html stories/.txt bobshea/*.html
bobshea/*.jpg"
This file comes out to 15mb and expands to about 50 megs of HTML files with an index.
Local Backup
A batch script on my primary home machine runs at 3am every morning. First, it fetches the lifebackup file above and extracts it within "My Documents". Then it backs up "My Documents", "World of Warcraft", and "Everquest" to my Lacie 80gb portable drive and to my Western Digital My Book 500gb. This includes everything in the lifebackup, all of my other files in "My Documents", and copies of my two main games with all of their most current patches.
Here is the batch script, "backup_daily.bat":
del lifebackup.tar.gz wget http://mikeshea.net/lifebackup.tar.gz 7za.exe x lifebackup.tar.gz -y 7za.exe x lifebackup.tar -olifebackup -y copy lifebackup.tar.gz lifebackup del lifebackup.tar
dir /S > file_manifest.txt
xcopy /E /D /Y lifebackup G:\lifebackup xcopy /E /D /Y lifebackup K:\lifebackup xcopy /E /D /Y lifebackup M:\lifebackup
xcopy /D /E /Y C:\"My Documents" F:\"My Documents" xcopy /D /E /Y D:\"Program Files"\Everquest F:\Everquest\ xcopy /D /E /Y D:\"World of Warcraft" F:\"World of Warcraft"
Lifebackup to USB
The same script above will copy the "lifebackup" directory in "My Documents", extracted from "lifebackup.tar.gz" downloaded from my webserver with wget, to two USB thumb drives. These drives will also contain a manual copy of my "inbox" mbox format mail file and the "Robert_Shea_Interviews.mp3" file that contains recordings of interviews with my father. This file would be very hard to reconstruct.
Every week I will manually copy the lifebackup to my 512 meg USB SD card that I carry in my wallet.
As these devices get bigger, I will add more to them such as my "My Pictures" directory which has about 500 megs of photos.
I want these USB devices to be tiny archives that record my whole life.
Movie Backup
My primary machine has a "Movies" directory containing my top 50 favorite movies. Every night the script above will incrementally xcopy the "Movies" directory on the 400gb internal drive to the "movies" directory on the 500gb external Western Digital My Book.
Off-site Backup
Every month I will move a copy of my data on an external hard disc to an off-site location. I would like to find a better way to automate this but the sheer amount of data I have (about 400gb) makes it costly to automate it. The minimum I care about, the Lifebackup, is already backed up off-site on Pair.com's servers. The next level up is about 25gb of data which is too much to automate.