August 10th, 2008
You know how when someone accidentally deletes their files or their hard drive crashes or some other apocalyptic event occurs, the first thing people ask is "where is your backup"? Of course, we've all seen it (*ahem* been there :/ ). It's a bit unintuitive, because backups have no equivalent in the real world. If you drive your car into a lake, there's no way to get it back. But making backups is the single best way to prevent losing your stuff. So do it!
Don't backup "my files"
But don't just backup "my documents, my pictures, my whatever". If you computer crashes and you have a backup of "my files", then sure, it's not a total loss. It's better than nothing. But it's not what you actually need. You need the whole thing
This "my files" nonsense is born out of the fact that the delightful company that produced your operating system doesn't want you to be able to make a backup of it. Because if you did, you could make trivial copies of the operating system, and they don't like that idea. Have you ever asked yourself why in 30 years, through all manner of viruses, blue screens of death and hardware crashes Microsoft has never given you or sold you a full system backup program? It's not because they never thought of it. (Or because no one asked for it).
Making full backups is easy
If you've ever installed Gentoo manually (ie. not with one of the automated installers)... yes, the demographic for this one is not immense. But then that's why we're here, to spread the happy message! :happy: Anyway, if you have, then you know immensely easy (this is astonishing especially if you have a Windows background) it is to make a full system backup. In the course of a Gentoo install (and yes, I'm about to reveal the big secret here...*drumroll*), you boot from the livecd, you mount your root partition, you download a tar of a minimal Gentoo filesystem that has your basic /bin/ls and so on, and then you just.... untar it. That's it. No magic, no voodoo, no secret foobared locations on the filesystem that can't be written to, just extract the archive and you're done!
To put it bluntly, this is all you have to do:
tar zcvf backup.tar.gz /
And to restore the backup:
tar zxpvf backup.tar.gz -C /mnt/root_partition
Put that in perspective to the Windows world where a whole industry has sprung up to solve problems that Microsoft deliberately introduced with their *ahem* novel engineering. Idiotic programs like Norton Ghost that you have to get your hands on just to do the same simple thing that you can do with tar on a decent operating system.
Making it more convenient
Granted, you could just use the above tar command, but you may want something a little more convenient. For starters, you may want to skip some files on your file system. The method I use is inspired by a script posted on the gentoo forums a long time ago. I used that script for years without really understanding it, but a while back I decided to rewrite it to suit me better.
Besides just tarring the files it also writes a log file that you can grep to see if some particular file of interest is in the backup, it timestamps the backup with the current date/time and it keeps track of how many backups you want to keep.
Backups are made in a special backup_dir location. This directory is supposed to hold lists of files (recipes, if you like) you want to backup. For example, a simple recipe could be called full.lst:
/
--exclude=/backup/*.tgz*
--exclude=/proc/*
--exclude=/sys/*
--exclude=/tmp/*
--one-file-system
The syntax for the file is that of tar, and it's a list of things to backup. / means the full file system will be included. But certain directories are excluded, /backup because we don't want to include our old backup files in new backups, /proc and /sys because they are virtual file systems and don't contain "real" files, and we don't care about /tmp. Finally, we say --one-file-system, which prevents mounted disks, cds and things like that to be included.
And here is the script that makes this possible. Run it, it will produce a backup file that is compressed. Try to get it below 4.3gb and write it on a dvd+rw, now you have a backup system. :party:
#!/bin/bash
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
backup_dir=/backup
num_backups=1
verbose="$@"
lists=$backup_dir/*.lst
ext=tgz
date_params="%Y-%m-%d-%H%M"
nice_val="nice -n20"
# colors
wh="\e[1;37m"
pl="\e[m"
ye="\e[1;33m"
cy="\e[1;36m"
re="\e[1;31m"
if [[ "$verbose" && "$verbose" != "-v" ]]; then
echo "Usage: $0 [-v]"
exit 1
fi
if [ ! -d $backup_dir ]; then
echo -e "${re}Backup dir $backup_dir does not exist.${pl}"; exit 1
fi
for list in $(ls $lists); do
name=$(basename $list .lst)
file_root=$backup_dir/$name.$(date +$date_params)
stdout="1> /dev/null"
stderr="2> $file_root.$ext.err"
if [ "$verbose" ]; then
stdout=""
fi
cmd="cat $list | $nice_val xargs tar zlcfv \
$file_root.$ext $stderr | tee $file_root.$ext.log $stdout"
trap 'echo -e "${re}Received exit signal${pl}"; exit 1' INT TERM
echo " * Running \`$name\` job..."
if [ "$verbose" ]; then echo -e ${ye}$cmd${pl}; fi
echo -en $cy; bash -c "$cmd" ; echo -en $pl
status_code=$?
if [ $status_code -gt 0 ]; then
# Dump error log
echo -en $re ; cat $file_root.$ext.err
echo -en $pl ; echo "Tar exit code: $status_code"
else
# Kill error file
rm $file_root.$ext.err
fi
# Evict old backups we don't want to keep
num=$num_backups
for evict in $(ls -t $backup_dir/$name.*.$ext); do
if [ $num -le 0 ]; then
rm -f "$evict"
else
num=$(($num-1))
fi
done
# Report number of files in backup
echo -n "$(wc -l < $file_root.$ext.log) files"
echo ", $(ls -l $file_root.$ext | awk '{ print $5 }') bytes"
done
Worse is better
I've been thinking about how to handle backups most effectively, and it occurs to me that backups are a case of "worse is better". The thing is you could make a really nice and easy application to make backups, but .tar.gz is still the optimal format to store them in. Wherever you are, you have tar and gzip available, and restoring backups usually happens under somewhat constricted conditions, sometimes without network access. So you want to avoid introducing dependencies, it's safer to make do with the tools that are there already.
So it may not be the most elegant system, but it's damn reliable.
Limitations (NEW)
Basically what I'm saying is that if you have no backup system then using tar is a pretty decent system. At the very least it has worked well for me the last 5 years. That isn't to say you shouldn't use a different method if you have different needs.
What about scaling? Well, I think this works quite well up to backups of say 4gb or so. My root partition is using 12gb of space at the moment. The purpose of this method is to back up your working system with all the configuration, applications and so on. Not to back up your mp3 collection, I would exclude that (not least because it's pointless to compress mp3 files and other formats that already are well compressed).
What about the bootloader? (NEW)
Some people have asked how this backup method concerns the bootloader. The answer is that it does backup the files that belong to the bootloader (in /boot). It does not backup the actual boot sector of your hard drive (which isn't represented as a file). So if, for example, you want to restore the backup on another computer (which I've done lots of times), you'll still need to use grub/lilo to update the boot sector.
UPDATE: Apologies to the indignant Windows users. I pretty much wrote this in rant mode, without doing any research and what I wrote about Windows is just from my own experience. I would have been more thorough if I had known this would make the frontpage of digg and draw so many hits.
Ahhh that magical script! I used it so many times Martin!
I want to get back to Gentoo :(
I ran it as on the website and got (although it's still running and doing something):
./backup.sh: line 50: full: command not found
Could you run it with with the -v switch? Then it will echo the command it's about to execute before actually doing it, should shed some light on this.
I prefer to use dd and gzip together over ssh. There are a million backup solutions, and all are good. Unix is great that way. If you have quite a few computers to backup, consider backuppc. It pools common files (of which there will be many).
RE: "[Microsoft] doesn’t want you to be able to make a backup of it."
I've been using Windows Backup for 4 years now, on XP Home and Pro, to make full system backups. Differential, incremental and full backups, incl. Registry, while running (shadow copy, w/ adequate disk space). Can be automated to run as a Scheduled Task.
Comes bundled with XP; pro comes pre-installed, Home requires user to install from CD or .cab.
I've been using DAR for all of my backups. It's got a lot of flexible command line switches for doing just about anything you want. Some notable features I don't think TAR offers are encryption and differential backups.
Want to use tar on Windoze? Not a problem.
Install the free Windows Services for Unix (SFU).
TAR is part of the included options.
Great stuff, thanks :) Found this on digg, using it now.
Hmm, but a solution like R1Soft can do it much faster and keep actual snapshots of the entire disk. With rsync will choke with alot of files, and files will be out of sync by the time rsync gets around to actually copying them.
I know you say Gentoo... but will this work on Ubuntu? or even *all* linuxes? What about The GRUB loader?
c.
I have been looking around recently for something to "clone" my system and I ran across this.
http://www.ubuntugeek.com/creating-custom-ubuntu-live-cd-with-remastersys.html
There are only a couple minor issues after a successful clone that I have found and they are covered in the developers forum.
Love you solution too.
Thanks!
I'm having the same bug/issue with the script:
desktop ~ # ./backup.sh -v
./backup.sh: line 50: full: command not found
* Running \ job...
cat /backup/full.lst | nice -n20 xargs tar zlcfv \ /backup/full.2008-08-10-2137.tgz 2> /backup/full.2008-08-10-2137.tgz.err | tee /backup/full.2008-08-10-2137.tgz.log
it's still compressing, so I'm not super worried :)
If there's a solution, please feel free to email me at computingsolutionsbiz (AT) gmail com
Seems like a good method, but does it break down for large amounts of data? How easy would it be to adapt this to handle disk spanning?
I have close to 1TB of storage on my system and the logistics of a backup on multiple pieces of media are as big an obstacle as the software is. A full backup on DVD would come out to (1000/4.3 =) ~233 disks (yikes!).
WARNING: tar will destroy all of your filesystem metadata. Maybe you're using a system where that doesn't matter? tar is fairly safe to use for backups of /home on systems where you only need the basic wheel bits preserved, though. Using tar to do a backup of Fedora, RHEL, Novell, Solaris and other "secure" systems will not result in a (properly) working installation after restoration. (There are probably some scripts you can run later to unhose the system somewhat but there's no need for that if you do it right the first time.)
tar can't include metadata, like extended attributes and access control lists, in your archive. Lots of stuff uses EAs and ACLs. Especially SELinux, AppArmor and things like that. Even GNOME stores some metadata in EAs if EAs are available on your filesystem. MacOSX makes extensive use of EAs and ACLs, also. All toast with tar.
You should be using pax instead of tar for a full system backup. pax superceded both tar and cpio in the POSIX 2001 spec.
Look unher the "Backup and Restore" section here:
http://www.suse.de/~agruen/acl/linux-acls/online/
Just because you sir are ignorant, doesnt mean you should take your readers as such.
Microsoft trying to stop users from making copies of Windows? BAH! That is why the activation process was invented. Just try to clone 2 PCs and see what happens. Windows finds new hardware, and barks at you.
But oh wait, you already knew about Ghost... whatever.
The Redmond folks really don't have a problem with Full OS backups, trust me.
And for your information, sir; Windows is as easy to backup as linux, with the correct tool.
True, NTBACKUP sucks. I won't argue with that.
But you can certainly do a bare-metal restore of Windows. For free. And without ever going offline to make an image of your system either! (Try http://www.runtime.org/dixml.htm for example)
And that's a fact, unlike your fanboy FUD.
Also, FYI, I happen to use Linux at home and on my servers. Yup, you heard me, I mainly use all flavors of the pinguin.
But that doesnt mean I'm stupid and try to bash other OSes without knowing what I'm talking about. Unlike YOU.
Now get your facts straight, and get the head out of your ass.
Also, rsync is a WAY better alternative to tar. And duplicity is an even better one if you want remote backups (who doesnt?!?) and security (encryption, etc).
Google is your friend. (Or is it?)
you may want to add --update to the tar command - that way it isn't overwriting every file, just the ones that have been updated since it was last made.
Also, keeping full disk backups tends to get a little expensive (especially daily), if you really want to be able to go back like that, a commercial solution like r1soft is your best bet.
Microsoft Vista does can do a complete backup.
Don't know about Gentoo but on Ubuntu, you probably also want to exclude:
--exclude=/home/*/.local/share/Trash
in my case, it's 13G off my total 20G ... but I won't delete anything as long as I have 480G remaining ;)
i heard different thing about backing up /dev. what do you think?
Will this work for fedora9? or do i need to find a customized script...
is it really as simple as tarring root and untarring later on a clean / mount?
I have a dual boot hackintosh and I use Symantic Ghost 8.2 (an 80's dos util) to image my windows partition. I keep my data on a separate partition and use apple's disk util to image my mac partition. (3 partitions = 2 systems and a data partition). for the third partition, i keep a mirror on a separate hard disk.
Dudes,
why can nobody program a simple solution like TimeMachine (Mac OSX) for Linux. It has to be easy to handle for my mom and my pop...
Tom
One thing that the author (and a lot of other people) forget is the boot loader. Although his command backup all your files, after a restore, all you've got is the files. But your system will not boot until you restore the boot loader. This requires a boot loader (Grub or LILO) command to accomplish.
ugh, unix has had incremental backup since v5 unix (dump(8) and restore(8)), tar is not a backup solution.
easy peasy full system backup ~ numerodix blog...
You know how when someone accidentally deletes their files or their hard drive crashes or some other apocalyptic event occurs, the first thing people ask is “where is your backup”? Of course, we’ve all seen it (*ahem* been there :/ ). It’s a bi...
[...] you or sold you a full system backup program? Never mind MS crap, here is how to do it on Linuxread more | digg [...]
Time Machine on MacOS X?
Time Vault on Linux:
https://wiki.ubuntu.com/TimeVault
Its integrated into GNOME fairly similarly to the way Time Machine is on Mac. If you note the dates, Time Vault was released before Time Machine.
Good tutorial, but you are missing something. What about incremental backups? Your version of an rsync tutorial would work great with this post.
Thanks!
@AntX - wow! you have a hard time understanding a legitimate point. the worlds largest monopoly has to hide lots of hidden keys inside each operating system so that they can maintain their secrecy. Their secrecy does not do anything for the users, so the problems that Ghost resolve are features that do not serve the user. Is that a feature?
The ability to reliably backup and restore a system is kind of an obvious and important engineering task, so I think your "fanboy" accusation = FAIL.
@Augur:
Thanks a lot for this hint! I'll have a look into this right away...
If TimeVault really IS as good as TimeMachine (in MacOS), then Ubuntu has a HUGE marketing deficit ! -> TimeMachine was one of the MAIN recent USPs (unique selling propositions) of OSX. If Linux has the same capabilities, PEOPLE NEED TO GET THE WORD OUT !!!
(Having a dead-simple and trustworthy back-up solution has become SO important to EVERYBODY, nowadays! Such a feature must be integrated into Ubuntu as a main standard feature!)
Tom
I really dislike working with Windows systems, but the submitter is just not correct. There's NTBackup, which, any way you look at it, has more built in features than tar.
You must also include the --numeric-owner switch. I found this out the hard way after suffering from subtle and progressive corruption of my system. If you don't include this switch, tar doesn't restore the filesystem's group and owner attributes exactly as before, but tries to match the group and owner names in the tarfiles to those of the live-cd and will *change* the *numeric* group and owner attributes. Bastard.
I've started using lvm snapshots of my home partition for backups. I have all the manual changes to configuration files saved in an svn repo, so in the event I experience a crash on my desktop I just have to reinstall the OS, import the snapshot, and checkout the appropriate changes from the repo.
Of course if you need your system up and running quickly, the above method is going to restore your system faster.
So.... you're telling people to dump Windows, all the applications they have purchased, the years of experience they have with it and move to Gentoo so that they can perform a *full* backup/restore, should they ever need to? What are you smoking?
Urhhh, I'm a *nix user myself and so totally with you on the choice of operating system - but Gentoo is not mainstream friendly and so this article doesn't really move the debate forward.
80% of the world's desktops will be running Seattle's finest for sometime, and there's plenty of good reasons for that, including that for many people it works just fine.
@James:
I wasn't making a point out of ghost. I was making a point out of WGA. Which (should?!?) prevent users from making illegal copies of the OS.
So his shameless accusations on Microsoft are indeed fanboy-ism.
I basicaly said it's stupid to bash on Microsoft, stating that they "do not want full OS backups". What a load of crap! They don't friggin care. If they did, they would have removed NTBackup all along.
Also, you can use the "Shadow Volume" service to make snapshots of your data, and recover it later. Vista also has a full backup feature built-in.
ANYWAYS, the point is that Numerodix knows nothing about Windows, and he knows even less about backups.
Making a TAR.GZ of the OS? Holy Cow!
If I wanted to remove all permissions on my OS, and then preventing it to boot when I recover it, that would be the best solution around! Thanks for that.
I would *highly* recommend *against* compressing the backup file if you truly do care about data retention. if you have a bad byte in a tar file, it's something you can work around, but a bad byte in a tar.gz file and you suddenly have a complete waste of space.
Full system backup using tar...
Nice tutorial about using tar for complete system backup -- not Ubuntu-specific...
[...] Read more at http://www.matusiak.eu/numerodix/blog [...]
[...] Source: http://www.matusiak.eu/numerodix/blog/index.php/2008/08/10/easy-peasy-full-system-backup/ [...]
its - Apparently making a back up of my files is behavior encouraged by the manufacturer of my operating system? Is this guy for real? Maybe some people do it because they don t have gigabyte and gigabytes of storages available to back up their whole system. I use Ubuntu, for the record
Worked really well, gave me a good insite into Linux backups. Thanks for your hard worl
[...] Link: Numerodix Blog [...]
Actually you can make a backup of your MBR (including your boot loader and also your partition table)
You just need to copy the first 512 bytes (full mbr) or first 446 bytes (boot loader only) from your device file. You can do this *very* easily with dd.
[...] 26, 2008 at 9:59 pm (App, Linux, Ubuntu) easy peasy full system backup ~ numerodix blog easy peasy full system backupAugust 10th, [...]
Thanks and can someone tell me about backtrack backup system.
I was developing similar scripts since years. This one looks very good!
All backup solutions: pc or network based are freely available in every Linux distro repositories. It is pointless to compare Microsoft vs. Linux on this matter because both offer free backup solutions. What matters is that: LInux is for free........... M$ is for pay.
I was a Microsoft OS user for the past 12 years and I WILL NEVER BE AGAIN!!!!!!!!
Very good and very easy solution, for backup on a running system;
Yes, boot sector could be added, but for me and for now, it's no vital.
Good work and good tutorial ***********
Ciao, from Italy!
Ney...
Norton Ghost is decent because it backs up boot loader as well, and it does more than that.
Using Linux does not mean you are privileged to be biased.
I am not able to use the script. I get this error:
ls: cannot access /home/user/backup/*.lst: No such file or directory
All I have to do is
./full_system_backup.sh
right?
So that means $backup_dir is set to /home/user/backup. And in there the script is looking for files called something.lst. A file like that defines a recipe for a backup. And that's what the error message means, it can't find any .lst files.
i think you really need very good back up systems so that your system gives very reliable system functioning, this is very needed
This script could be very helpful. Great post!
damn... thought this was about backing up Easy Peasy systems :(
Seems to be a problem: out-of-the-box Debian Squeeze. No configuration change after install other than attempting to setup this system.
‘./backupsys: line 50: full: command not found’
Hi
How do I run this script on AIX (ksh)?
Thanks
You can't run it on ksh since it's written in bash (obviously). Furthermore the version of bash you might have on AIX might be too ancient to work (last time I used AIX at any rate).