[plug] RPi USB drive backups

Gregory Orange home at oranges.id.au
Thu Mar 20 15:49:45 AWST 2025


I've been using syncthing for replication across devices, and then an
old rdiff-backup script to a USB HDD (with a new one once in a while)
for years. I sometimes wonder if that system should be replaced for some
reason, but it's been incredibly reliable so I mostly leave the topic alone!


On 20/3/25 15:21, William Kenworthy wrote:
> Thinking outside the box: Unison is often somewhat to much much faster
> than rsync (depends on content - uses different heuristics with stored
> information)
> 
> I use now use DRFB for mirroring - very fast on usb attached storage
> (odroid HC2's - 32bit arm systems - better than most older pi's)
> 
> I also use it for lxc images with duplicated hardware including storage
> - stop on one host, make destination primary (microseconds to switch
> unless not synced) then instantly start image on new host.
> 
> Think of it as a permanent tunnel continuously transferring low level
> file system changes between two hosts - the primary end of the tunnel is
> the one you attach the host to as info is unidirectional
> (primary->secondary).  I usually have only only one host running -
> starting the other when I need a sync (which may need some manual
> intervention unfortunately if there are a lot of changes).
> 
> Another suggestion is use a backups system like borgbackup or perhaps
> dirvish (makes a hardlinked versioned copy) may be better for your use
> case - both also much faster than rsync.  dirvish makes a versioned
> mirror when run so restore is just a copy away.
> 
> BillK
> 
> 
> On 20/3/25 10:33, Chris Hoy Poy wrote:
>> Hi Chris,
>>
>> Remounting the drive read only is a standard error handling process
>> for a disk experiencing certain errors 
>>
>> The smartctl test is performed by the drive itself, good news : it's
>> probably not the drive, however I've had drives that were clearly
>> failing happily do these tests (probably an issue on their connector etc)
>>
>> USB cables are pretty notorious for being flakey etc
>> Have had them work with some drives and not others, total pita 
>>
>> Rsync over a large volume , particularly with lots of small files and
>> directories, is a pretty brutal stress test of several subsystems. 
>>
>> /Chris 
>>
>> On Thu, 20 Mar 2025, 10:26 am Chris McCormick, <chris at mccormick.cx> wrote:
>>
>>     Hi,
>>
>>     Thanks for your reply.
>>
>>     smartctl seems to think the hardware is fine. I did smartctl -a
>>     and it
>>     reported nothing unusual. I also ran a self test and this was the
>>     result:
>>
>>     $ sudo smartctl -l selftest /dev/sdb
>>     smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+] (local build)
>>     Copyright (C) 2002-20, Bruce Allen, Christian Franke,
>>     www.smartmontools.org <http://wwwsmartmontools.org>
>>
>>     === START OF READ SMART DATA SECTION ===
>>     SMART Self-test log structure revision number 1
>>     Num  Test_Description    Status                  Remaining
>>     LifeTime(hours)  LBA_of_first_error
>>     # 1  Short offline       Completed without error       00%      6625
>>          -
>>
>>     If the drive or the USB hub was overheating would that cause IO
>>     errors?
>>     I should also mention that the OS seems to remount the drive a
>>     read only
>>     once this happens.
>>
>>     Chris.
>>
>>
>>     On 20/03/2025 08:09, Chris Hoy Poy wrote:
>>     > Hi Chris
>>     >
>>     > If it's timing out with IO errors, you probably have a hardware
>>     problem
>>     > somewhere in the mix.
>>     >
>>     > Cable , connector, hub or USB hard drive (probably the
>>     destination drive
>>     > by the sounds of it).
>>     >
>>     > Shouldn't matter which software you use to copy (dd, rsync etc)
>>     - though
>>     > very different technical processes, if it's dropping data
>>     somewhere then
>>     > they will bomb out.
>>     >
>>     > Cheers
>>     > /Chris
>>     >
>>     >
>>     > On Thu, 20 Mar 2025, 8:03 am Chris McCormick, <chris at mccormick.cx
>>     > <mailto:chris at mccormick.cx>> wrote:
>>     >
>>     >     Hey all,
>>     >
>>     >     Unfortunately I couldn't make it to the recent RPi meetup,
>>     otherwise
>>     >     that would have been a good place to ask about this
>>     >
>>     >     Anyway, I have an RPi with a powered USB hub and two USB
>>     hard drives
>>     >     plugged in doing backups. The second drive is a mirror of
>>     the first
>>     >     which I have traditionally done with rsync. I recently
>>     replaced one of
>>     >     the drives and I now need to mirror the full drive to the
>>     new drive.
>>     >     I've been using rsync to do this but it takes ages just to
>>     do the
>>     >     initial diff and craps out after a while with IO errors and
>>     unmounts
>>     >     the
>>     >     destination drive. The total size of data is 2.3Tb.
>>     >
>>     >     What is a good way to do a fast copy of all of the data on
>>     one USB
>>     >     drive
>>     >     to the other? I'd prefer not to have to stuff around with
>>     partitions if
>>     >     possible as they have slightly different partitions on each
>>     drive and
>>     >     different labels etc.
>>     >
>>     >     LLMs were no help at all with this so I'm relying on the
>>     PLUG brains
>>     >     trust! 😅 Thanks for any suggestions.
>>     >
>>     >     Chris.
>>     >
>>     >     --
>>     >     https://mccormick.cx/ <https://mccormick.cx/>
>>     >     _______________________________________________
>>     >     PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>>     >     http://lists.plug.org.au/mailman/listinfo/plug
>>     >     <http://lists.plug.org.au/mailman/listinfo/plug>
>>     >     Committee e-mail: committee at plug.org.au
>>     <mailto:committee at plug.org.au>
>>     >     PLUG Membership: http://www.plugorg.au/membership
>>     <http://www.plug.org.au/membership>
>>     >     <http://www.plug.org.au/membership>
>>     >
>>
>>     -- 
>>     https://mccormick.cx/
>>
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership

-- 
Gregory Orange




More information about the plug mailing list