[plug] Managing files across diverse media?

[Tim]

I'd be interested to know what your solution is too. My situation doesn't
sound as bad as yours, but in the past it has gotten pretty bad and many
hours have been spent de-duping backups of backups.

Tim


On 1 September 2014 22:10, John McCabe-Dansted <gmatht at gmail.com> wrote:

> I have many different disks (primarily sitting on online servers or
> offline removable harddisks) storing various files. I want to know
> that they are all backed up in some form. Given that I have terabytes
> of files somehow (backups of backups apparently) I don't want to just
> backup everything onto new media yet again. I'd like to be able to
> quickly maintain a list of sha256 or md5 sums that could be used to:
>
> 1) To list all files on X that are not duplicated/backed up on other media
> 2) Deduplicate files on X quickly (using existing md5 hashes).
> 3) To list all files that are not duplicated onto offline or WORM storage
> 4) To list all files that are not duplicated onto offsite storage
> 5) Match JPGs by EXIF date.
>
> It seems to me that I wouldn't be the only person in this boat.
> However, there doesn't seem to be a tool even just to quickly update a
> list of file hashes. For example, md5deep wants to regenerate hashes
> for unmodified files on every run.
>
> I am looking at writing a tool to record and manage file IDs across
> media [1], but doing this right could take quite a while.
>
> How do other people handle this?
>
>
> [1] https://github.com/gmatht/joshell/tree/master/mass_file_management
>
> --
> John C. McCabe-Dansted
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>



-- 
Timothy White - Somewhere in Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20140907/ed0e0349/attachment.html>