[plug] RPi USB drive backups

Thu Mar 20 15:21:04 AWST 2025

Thinking outside the box: Unison is often somewhat to much much faster 
than rsync (depends on content - uses different heuristics with stored 
information)

I use now use DRFB for mirroring - very fast on usb attached storage 
(odroid HC2's - 32bit arm systems - better than most older pi's)

I also use it for lxc images with duplicated hardware including storage 
- stop on one host, make destination primary (microseconds to switch 
unless not synced) then instantly start image on new host.

Think of it as a permanent tunnel continuously transferring low level 
file system changes between two hosts - the primary end of the tunnel is 
the one you attach the host to as info is unidirectional 
(primary->secondary).  I usually have only only one host running - 
starting the other when I need a sync (which may need some manual 
intervention unfortunately if there are a lot of changes).

Another suggestion is use a backups system like borgbackup or perhaps 
dirvish (makes a hardlinked versioned copy) may be better for your use 
case - both also much faster than rsync.  dirvish makes a versioned 
mirror when run so restore is just a copy away.

BillK

On 20/3/25 10:33, Chris Hoy Poy wrote:
> Hi Chris,
>
> Remounting the drive read only is a standard error handling process 
> for a disk experiencing certain errors
>
> The smartctl test is performed by the drive itself, good news : it's 
> probably not the drive, however I've had drives that were clearly 
> failing happily do these tests (probably an issue on their connector etc)
>
> USB cables are pretty notorious for being flakey etc
> Have had them work with some drives and not others, total pita
>
> Rsync over a large volume , particularly with lots of small files and 
> directories, is a pretty brutal stress test of several subsystems.
>
> /Chris
>
> On Thu, 20 Mar 2025, 10:26 am Chris McCormick, <chris at mccormick.cx> wrote:
>
>     Hi,
>
>     Thanks for your reply.
>
>     smartctl seems to think the hardware is fine. I did smartctl -a
>     and it
>     reported nothing unusual. I also ran a self test and this was the
>     result:
>
>     $ sudo smartctl -l selftest /dev/sdb
>     smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+] (local build)
>     Copyright (C) 2002-20, Bruce Allen, Christian Franke,
>     www.smartmontools.org <http://wwwsmartmontools.org>
>
>     === START OF READ SMART DATA SECTION ===
>     SMART Self-test log structure revision number 1
>     Num  Test_Description    Status                  Remaining
>     LifeTime(hours)  LBA_of_first_error
>     # 1  Short offline       Completed without error       00%   6625
>          -
>
>     If the drive or the USB hub was overheating would that cause IO
>     errors?
>     I should also mention that the OS seems to remount the drive a
>     read only
>     once this happens.
>
>     Chris.
>
>
>     On 20/03/2025 08:09, Chris Hoy Poy wrote:
>     > Hi Chris
>     >
>     > If it's timing out with IO errors, you probably have a hardware
>     problem
>     > somewhere in the mix.
>     >
>     > Cable , connector, hub or USB hard drive (probably the
>     destination drive
>     > by the sounds of it).
>     >
>     > Shouldn't matter which software you use to copy (dd, rsync etc)
>     - though
>     > very different technical processes, if it's dropping data
>     somewhere then
>     > they will bomb out.
>     >
>     > Cheers
>     > /Chris
>     >
>     >
>     > On Thu, 20 Mar 2025, 8:03 am Chris McCormick, <chris at mccormick.cx
>     > <mailto:chris at mccormick.cx>> wrote:
>     >
>     >     Hey all,
>     >
>     >     Unfortunately I couldn't make it to the recent RPi meetup,
>     otherwise
>     >     that would have been a good place to ask about this
>     >
>     >     Anyway, I have an RPi with a powered USB hub and two USB
>     hard drives
>     >     plugged in doing backups. The second drive is a mirror of
>     the first
>     >     which I have traditionally done with rsync. I recently
>     replaced one of
>     >     the drives and I now need to mirror the full drive to the
>     new drive.
>     >     I've been using rsync to do this but it takes ages just to
>     do the
>     >     initial diff and craps out after a while with IO errors and
>     unmounts
>     >     the
>     >     destination drive. The total size of data is 2.3Tb.
>     >
>     >     What is a good way to do a fast copy of all of the data on
>     one USB
>     >     drive
>     >     to the other? I'd prefer not to have to stuff around with
>     partitions if
>     >     possible as they have slightly different partitions on each
>     drive and
>     >     different labels etc.
>     >
>     >     LLMs were no help at all with this so I'm relying on the
>     PLUG brains
>     >     trust! 😅 Thanks for any suggestions.
>     >
>     >     Chris.
>     >
>     >     --
>     > https://mccormick.cx/ <https://mccormick.cx/>
>     >     _______________________________________________
>     >     PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>     > http://lists.plug.org.au/mailman/listinfo/plug
>     >     <http://lists.plug.org.au/mailman/listinfo/plug>
>     >     Committee e-mail: committee at plug.org.au
>     <mailto:committee at plug.org.au>
>     >     PLUG Membership: http://www.plugorg.au/membership
>     <http://www.plug.org.au/membership>
>     >     <http://www.plug.org.au/membership>
>     >
>
>     -- 
>     https://mccormick.cx/
>
>
> _______________________________________________
> PLUG discussion list:plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail:committee at plug.org.au
> PLUG Membership:http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20250320/f558cee9/attachment.html>