[plug] Backups

[Craig Ringer]

James Devenish wrote:

>>Also, would I use dd or some other method for [backups]?
> 
> This
> question about dd can be considered in the general case (i.e. not linked
> to DVDs), in which you'd have to ask yourself whether `dd` will produce
> a self-consistent backup. If the disk partition is in use for writing
> (e.g. databases, log files), then this is unlikely. But depending on the
> filesystem and nature of your files, it *might* still be acceptable, for
> *your* purposes, to use dd (even if a bit of `fsck`ing is necessary).
> But if you need to be careful, you'll have to take down your daemons,
> sync, take a filesystem snapshot, then do a backup using the snapshot.
> This sort of thing is mentioned in the list archives. This is in a sense
> a "grey area" and people do whatever is gives the best tradeoff between
> practicality and recoverability.

In general, I would tend to advise you NOT to use dd unless you have 
good reasons to do so. For regular backups, I do not think it is a good 
idea at all. Imagine if you discovered that all your recent backups had 
the same subtle filesystem corruption as your main filesystem when you 
when to recover a file...

dd is also very inefficient for backups, because it'll happily copy all 
the space the filesystem has marked as 'free'. That space is likely to 
contain old, deleted data which might well not be very compressible - or 
it might contain data you'd prefer was NOT archived anywhere.

To top things off, to safely make a copy of a filesystem with dd you 
must either unmount the filesystem or remount it read only. This means 
you can't be doing much else with it at the time, and it's very 
difficult to automate reliably. The alternative, using dd to clone the 
filesystem while it's mounted read/write and in use, is a recipe for 
massively corrupt backups. If you are using something like LVM, then you 
can reasonably safely clone a filesystem that's mounted read/write by 
using an LVM snapshot. You will still need to perform filesystem 
recovery if you restore it later, but it should not need much and at 
worst a few files may be corrupt if they were being written to as the 
snapshot was taken. Without LVM, you've got to be crazy.

You may have guessed by now that my attitude to using dd for backups is 
"just don't."

> Something like `dump` is generally
> better, because you can restore files on a arbitrary basis onto
> arbitrary partitions (whereas `dd` limits you to restoring an entire
> filesystem onto an identical partition). There are various suggestions
> and examples in the list archives.

The time honoured approach is using an archiver like tar, pax, or cpio. 
   This is much safer but can be somewhat slower than image based 
backups. (It can also be a lot faster if you're backing up a mostly 
empty volume or only doing incremental backups). In general, I think it 
should be considered the default approach for backups, to be used unless 
you have a good reason to choose something else.

Another increasingly common option is to use backups on large 
random-access media like hard disks, storing a normal filesystem on it. 
The backup is then updated periodically, rather than rewritten entirely. 
IMHO this is good for rather large chunks of data that change 
infrequently and don't need to be backed up too regularly.

If you're writing to DVD, I think that an archiver is the safest choice. 
Creating a UDF or ISO9660 image with the files on it directly may not 
preserve all file metadata.  Using 'dd' to create an image to archive 
isn't overly safe - one scratch, and you could potentially lose the 
entire filesystem image. Unfortunately, IIRC compressed archives suffer 
from the same problem to some extent, so ideally I'd archive my data 
uncompressed or compress it before, rather than after, archiving. That 
way an unreadable block will result in, at worst, one unusable file 
instead of potentially large chunks of the archive being lost.

A quick aside: Does anybody here know of a tool that can create 
compressed-then-archived archives, similar to what the Zip tool does 
under DOS? I'd be looking for something that fully supported all the 
UNIX file permissions, extended attributes, etc but knew how to run each 
file through gzip before archiving it. I'm not too comfortable with 
tar's -z option, because AFAIK gzip's compression deals very poorly with 
corruption or unreadable blocks, and a bad block could render the rest 
of the archive unreadable.

--
Craig Ringer