[plug] 'best fit' data archiving tools

[Richard Meyer]

On Mon, 2005-09-19 at 19:16 +0800, Craig Ringer wrote:
> On Mon, 2005-09-19 at 14:54 +0800, Ryan wrote:
> > I'm not sure how to name what I want  ...
> > 
> > I want to break a large directory of data (600GB+) into smaller chunks
> > of a specified size limit (~16GB in this case).  In this particular case
> > the finest granularity will be a 2nd level directory, everything below
> > that must stay together with it's 2nd level parent, so essentially a
> > 'file' from a size perspective is the contents of each 2nd level
> > directory.
> 
> I spent some time looking for one a while ago, with nearly identical
> needs to you. I wasn't successful. In my case, rather than buying one, I
> solved it by getting some giant hard disks, sharing out the directory
> across them so that I'd have two complete copies of the data, and then
> storing them off-site.
> 
> This may not suit your needs.
> 
> Writing a tool to do this is significantly complicated by compression.
> If you're compressing, you don't know what each directory will compress
> down to until you've finished archiving it. If the compression is
> archive wide rather than per-directory it's even worse, since each
> directory will compress differently depending on what else is compressed
> with it.

You have just described a perfect use for DAR at
http://dar.linux.free.fr/

Quote form site:
>         Slices
>                 Dar stands for Disk ARchive. From the beginning, it
>                 was designed to be able to split an archive over
>                 several pieces of removable media -- no matter how
>                 many or what size. Thus, dar is able to save over
>                 floppy disks, CD-Rs, DVD-Rs, CD-RWs, DVD-RWs, Zip
>                 disks, Jazz disks, etc. Dar is not concerned by
>                 un/mounting a removable medium; instead it operates
>                 independently of the hardware. Given the size, it will
>                 split the archive in several files (called slices),
>                 pausing before creating each new slice. This allows
>                 the user to un/mount a medium, burn the slice to a
>                 CD-R, send it by email (if your mail system does not
>                 allow huge file in emails, dar can help you here
>                 also). By default, (no size specified), dar will make
>                 only one slice. If a slice size is specified and dar
>                 creates multiple slices, the size of the first slice
>                 can be specified separately. This is useful if, for
>                 example, you want to fill up a partially filled disk
>                 before starting use of an empty one. At restoration
>                 time, dar will look for the slices it needs, asking
>                 for a slice only if it is missing and required.

-- 
Richard Meyer <meyerri at westnet.com.au>
Cutting the space budget really restores my faith in humanity.  It
eliminates dreams, goals, and ideals and lets us get straight to the
business of hate, debauchery, and self-annihilation.
                -- Johnny Hart