[plug] 'best fit' data archiving tools

[Craig Ringer]

On Mon, 2005-09-19 at 14:54 +0800, Ryan wrote:
> I'm not sure how to name what I want  ...
> 
> I want to break a large directory of data (600GB+) into smaller chunks
> of a specified size limit (~16GB in this case).  In this particular case
> the finest granularity will be a 2nd level directory, everything below
> that must stay together with it's 2nd level parent, so essentially a
> 'file' from a size perspective is the contents of each 2nd level
> directory.

I spent some time looking for one a while ago, with nearly identical
needs to you. I wasn't successful. In my case, rather than buying one, I
solved it by getting some giant hard disks, sharing out the directory
across them so that I'd have two complete copies of the data, and then
storing them off-site.

This may not suit your needs.

Writing a tool to do this is significantly complicated by compression.
If you're compressing, you don't know what each directory will compress
down to until you've finished archiving it. If the compression is
archive wide rather than per-directory it's even worse, since each
directory will compress differently depending on what else is compressed
with it.

--
Craig Ringer