[plug] RPi USB drive backups
Gregory Orange
home at oranges.id.au
Thu Mar 20 15:49:45 AWST 2025
I've been using syncthing for replication across devices, and then an
old rdiff-backup script to a USB HDD (with a new one once in a while)
for years. I sometimes wonder if that system should be replaced for some
reason, but it's been incredibly reliable so I mostly leave the topic alone!
On 20/3/25 15:21, William Kenworthy wrote:
> Thinking outside the box: Unison is often somewhat to much much faster
> than rsync (depends on content - uses different heuristics with stored
> information)
>
> I use now use DRFB for mirroring - very fast on usb attached storage
> (odroid HC2's - 32bit arm systems - better than most older pi's)
>
> I also use it for lxc images with duplicated hardware including storage
> - stop on one host, make destination primary (microseconds to switch
> unless not synced) then instantly start image on new host.
>
> Think of it as a permanent tunnel continuously transferring low level
> file system changes between two hosts - the primary end of the tunnel is
> the one you attach the host to as info is unidirectional
> (primary->secondary). I usually have only only one host running -
> starting the other when I need a sync (which may need some manual
> intervention unfortunately if there are a lot of changes).
>
> Another suggestion is use a backups system like borgbackup or perhaps
> dirvish (makes a hardlinked versioned copy) may be better for your use
> case - both also much faster than rsync. dirvish makes a versioned
> mirror when run so restore is just a copy away.
>
> BillK
>
>
> On 20/3/25 10:33, Chris Hoy Poy wrote:
>> Hi Chris,
>>
>> Remounting the drive read only is a standard error handling process
>> for a disk experiencing certain errors
>>
>> The smartctl test is performed by the drive itself, good news : it's
>> probably not the drive, however I've had drives that were clearly
>> failing happily do these tests (probably an issue on their connector etc)
>>
>> USB cables are pretty notorious for being flakey etc
>> Have had them work with some drives and not others, total pita
>>
>> Rsync over a large volume , particularly with lots of small files and
>> directories, is a pretty brutal stress test of several subsystems.
>>
>> /Chris
>>
>> On Thu, 20 Mar 2025, 10:26 am Chris McCormick, <chris at mccormick.cx> wrote:
>>
>> Hi,
>>
>> Thanks for your reply.
>>
>> smartctl seems to think the hardware is fine. I did smartctl -a
>> and it
>> reported nothing unusual. I also ran a self test and this was the
>> result:
>>
>> $ sudo smartctl -l selftest /dev/sdb
>> smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+] (local build)
>> Copyright (C) 2002-20, Bruce Allen, Christian Franke,
>> www.smartmontools.org <http://wwwsmartmontools.org>
>>
>> === START OF READ SMART DATA SECTION ===
>> SMART Self-test log structure revision number 1
>> Num Test_Description Status Remaining
>> LifeTime(hours) LBA_of_first_error
>> # 1 Short offline Completed without error 00% 6625
>> -
>>
>> If the drive or the USB hub was overheating would that cause IO
>> errors?
>> I should also mention that the OS seems to remount the drive a
>> read only
>> once this happens.
>>
>> Chris.
>>
>>
>> On 20/03/2025 08:09, Chris Hoy Poy wrote:
>> > Hi Chris
>> >
>> > If it's timing out with IO errors, you probably have a hardware
>> problem
>> > somewhere in the mix.
>> >
>> > Cable , connector, hub or USB hard drive (probably the
>> destination drive
>> > by the sounds of it).
>> >
>> > Shouldn't matter which software you use to copy (dd, rsync etc)
>> - though
>> > very different technical processes, if it's dropping data
>> somewhere then
>> > they will bomb out.
>> >
>> > Cheers
>> > /Chris
>> >
>> >
>> > On Thu, 20 Mar 2025, 8:03 am Chris McCormick, <chris at mccormick.cx
>> > <mailto:chris at mccormick.cx>> wrote:
>> >
>> > Hey all,
>> >
>> > Unfortunately I couldn't make it to the recent RPi meetup,
>> otherwise
>> > that would have been a good place to ask about this
>> >
>> > Anyway, I have an RPi with a powered USB hub and two USB
>> hard drives
>> > plugged in doing backups. The second drive is a mirror of
>> the first
>> > which I have traditionally done with rsync. I recently
>> replaced one of
>> > the drives and I now need to mirror the full drive to the
>> new drive.
>> > I've been using rsync to do this but it takes ages just to
>> do the
>> > initial diff and craps out after a while with IO errors and
>> unmounts
>> > the
>> > destination drive. The total size of data is 2.3Tb.
>> >
>> > What is a good way to do a fast copy of all of the data on
>> one USB
>> > drive
>> > to the other? I'd prefer not to have to stuff around with
>> partitions if
>> > possible as they have slightly different partitions on each
>> drive and
>> > different labels etc.
>> >
>> > LLMs were no help at all with this so I'm relying on the
>> PLUG brains
>> > trust! 😅 Thanks for any suggestions.
>> >
>> > Chris.
>> >
>> > --
>> > https://mccormick.cx/ <https://mccormick.cx/>
>> > _______________________________________________
>> > PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>> > http://lists.plug.org.au/mailman/listinfo/plug
>> > <http://lists.plug.org.au/mailman/listinfo/plug>
>> > Committee e-mail: committee at plug.org.au
>> <mailto:committee at plug.org.au>
>> > PLUG Membership: http://www.plugorg.au/membership
>> <http://www.plug.org.au/membership>
>> > <http://www.plug.org.au/membership>
>> >
>>
>> --
>> https://mccormick.cx/
>>
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
--
Gregory Orange
More information about the plug
mailing list