<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thinking outside the box: Unison is often somewhat to much much
faster than rsync (depends on content - uses different heuristics
with stored information)</p>
<p>I use now use DRFB for mirroring - very fast on usb attached
storage (odroid HC2's - 32bit arm systems - better than most older
pi's)<br>
</p>
<p>I also use it for lxc images with duplicated hardware including
storage - stop on one host, make destination primary (microseconds
to switch unless not synced) then instantly start image on new
host.</p>
<p>Think of it as a permanent tunnel continuously transferring low
level file system changes between two hosts - the primary end of
the tunnel is the one you attach the host to as info is
unidirectional (primary->secondary). I usually have only only
one host running - starting the other when I need a sync (which
may need some manual intervention unfortunately if there are a lot
of changes).</p>
<p>Another suggestion is use a backups system like borgbackup or
perhaps dirvish (makes a hardlinked versioned copy) may be better
for your use case - both also much faster than rsync. dirvish
makes a versioned mirror when run so restore is just a copy away.<br>
</p>
<p>BillK</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 20/3/25 10:33, Chris Hoy Poy wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGNDYRK6Trwvd=N4dmioXpk2Q0bCALOTSS=KYLrZO4wrZ+EcCQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">Hi Chris,
<div dir="auto"><br>
</div>
<div dir="auto">Remounting the drive read only is a standard
error handling process for a disk experiencing certain errors </div>
<div dir="auto"><br>
</div>
<div dir="auto">The smartctl test is performed by the drive
itself, good news : it's probably not the drive, however I've
had drives that were clearly failing happily do these tests
(probably an issue on their connector etc)</div>
<div dir="auto"><br>
</div>
<div dir="auto">USB cables are pretty notorious for being flakey
etc</div>
<div dir="auto">Have had them work with some drives and not
others, total pita </div>
<div dir="auto"><br>
</div>
<div dir="auto">Rsync over a large volume , particularly with
lots of small files and directories, is a pretty brutal stress
test of several subsystems. </div>
<div dir="auto"><br>
</div>
<div dir="auto">/Chris </div>
</div>
<br>
<div class="gmail_quote gmail_quote_container">
<div dir="ltr" class="gmail_attr">On Thu, 20 Mar 2025, 10:26 am
Chris McCormick, <<a href="mailto:chris@mccormick.cx"
moz-do-not-send="true" class="moz-txt-link-freetext">chris@mccormick.cx</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Thanks for your reply.<br>
<br>
smartctl seems to think the hardware is fine. I did smartctl
-a and it <br>
reported nothing unusual. I also ran a self test and this was
the result:<br>
<br>
$ sudo smartctl -l selftest /dev/sdb<br>
smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+]
(local build)<br>
Copyright (C) 2002-20, Bruce Allen, Christian Franke, <a
href="http://wwwsmartmontools.org"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">www.smartmontools.org</a><br>
<br>
=== START OF READ SMART DATA SECTION ===<br>
SMART Self-test log structure revision number 1<br>
Num Test_Description Status Remaining <br>
LifeTime(hours) LBA_of_first_error<br>
# 1 Short offline Completed without error 00%
6625 <br>
-<br>
<br>
If the drive or the USB hub was overheating would that cause
IO errors? <br>
I should also mention that the OS seems to remount the drive a
read only <br>
once this happens.<br>
<br>
Chris.<br>
<br>
<br>
On 20/03/2025 08:09, Chris Hoy Poy wrote:<br>
> Hi Chris<br>
> <br>
> If it's timing out with IO errors, you probably have a
hardware problem <br>
> somewhere in the mix.<br>
> <br>
> Cable , connector, hub or USB hard drive (probably the
destination drive <br>
> by the sounds of it).<br>
> <br>
> Shouldn't matter which software you use to copy (dd,
rsync etc) - though <br>
> very different technical processes, if it's dropping data
somewhere then <br>
> they will bomb out.<br>
> <br>
> Cheers<br>
> /Chris<br>
> <br>
> <br>
> On Thu, 20 Mar 2025, 8:03 am Chris McCormick, <<a
href="mailto:chris@mccormick.cx" target="_blank"
rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">chris@mccormick.cx</a> <br>
> <mailto:<a href="mailto:chris@mccormick.cx"
target="_blank" rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">chris@mccormick.cx</a>>>
wrote:<br>
> <br>
> Hey all,<br>
> <br>
> Unfortunately I couldn't make it to the recent RPi
meetup, otherwise<br>
> that would have been a good place to ask about this<br>
> <br>
> Anyway, I have an RPi with a powered USB hub and two
USB hard drives<br>
> plugged in doing backups. The second drive is a
mirror of the first<br>
> which I have traditionally done with rsync. I
recently replaced one of<br>
> the drives and I now need to mirror the full drive to
the new drive.<br>
> I've been using rsync to do this but it takes ages
just to do the<br>
> initial diff and craps out after a while with IO
errors and unmounts<br>
> the<br>
> destination drive. The total size of data is 2.3Tb.<br>
> <br>
> What is a good way to do a fast copy of all of the
data on one USB<br>
> drive<br>
> to the other? I'd prefer not to have to stuff around
with partitions if<br>
> possible as they have slightly different partitions
on each drive and<br>
> different labels etc.<br>
> <br>
> LLMs were no help at all with this so I'm relying on
the PLUG brains<br>
> trust! 😅 Thanks for any suggestions.<br>
> <br>
> Chris.<br>
> <br>
> -- <br>
> <a href="https://mccormick.cx/"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://mccormick.cx/</a>
<<a href="https://mccormick.cx/"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://mccormick.cx/</a>><br>
> _______________________________________________<br>
> PLUG discussion list: <a
href="mailto:plug@plug.org.au" target="_blank"
rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">plug@plug.org.au</a>
<mailto:<a href="mailto:plug@plug.org.au" target="_blank"
rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">plug@plug.org.au</a>><br>
> <a
href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
> <<a
href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.plug.org.au/mailman/listinfo/plug</a>><br>
> Committee e-mail: <a
href="mailto:committee@plug.org.au" target="_blank"
rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">committee@plug.org.au</a>
<mailto:<a href="mailto:committee@plug.org.au"
target="_blank" rel="noreferrer" moz-do-not-send="true"
class="moz-txt-link-freetext">committee@plug.org.au</a>><br>
> PLUG Membership: <a
href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://www.plugorg.au/membership</a><br>
> <<a href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">http://www.plug.org.au/membership</a>><br>
> <br>
<br>
-- <br>
<a href="https://mccormick.cx/" rel="noreferrer noreferrer"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://mccormick.cx/</a><br>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre wrap="" class="moz-quote-pre">_______________________________________________
PLUG discussion list: <a class="moz-txt-link-abbreviated" href="mailto:plug@plug.org.au">plug@plug.org.au</a>
<a class="moz-txt-link-freetext" href="http://lists.plug.org.au/mailman/listinfo/plug">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a class="moz-txt-link-abbreviated" href="mailto:committee@plug.org.au">committee@plug.org.au</a>
PLUG Membership: <a class="moz-txt-link-freetext" href="http://www.plug.org.au/membership">http://www.plug.org.au/membership</a></pre>
</blockquote>
</body>
</html>