<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Thinking outside the box: Unison is often somewhat to much much
      faster than rsync (depends on content - uses different heuristics
      with stored information)</p>
    <p>I use now use DRFB for mirroring - very fast on usb attached
      storage (odroid HC2's - 32bit arm systems - better than most older
      pi's)<br>
    </p>
    <p>I also use it for lxc images with duplicated hardware including
      storage - stop on one host, make destination primary (microseconds
      to switch unless not synced) then instantly start image on new
      host.</p>
    <p>Think of it as a permanent tunnel continuously transferring low
      level file system changes between two hosts - the primary end of
      the tunnel is the one you attach the host to as info is
      unidirectional (primary->secondary).  I usually have only only
      one host running - starting the other when I need a sync (which
      may need some manual intervention unfortunately if there are a lot
      of changes).</p>
    <p>Another suggestion is use a backups system like borgbackup or
      perhaps dirvish (makes a hardlinked versioned copy) may be better
      for your use case - both also much faster than rsync.  dirvish
      makes a versioned mirror when run so restore is just a copy away.<br>
    </p>
    <p>BillK</p>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 20/3/25 10:33, Chris Hoy Poy wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAGNDYRK6Trwvd=N4dmioXpk2Q0bCALOTSS=KYLrZO4wrZ+EcCQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="auto">Hi Chris,
        <div dir="auto"><br>
        </div>
        <div dir="auto">Remounting the drive read only is a standard
          error handling process for a disk experiencing certain errors </div>
        <div dir="auto"><br>
        </div>
        <div dir="auto">The smartctl test is performed by the drive
          itself, good news : it's probably not the drive, however I've
          had drives that were clearly failing happily do these tests
          (probably an issue on their connector etc)</div>
        <div dir="auto"><br>
        </div>
        <div dir="auto">USB cables are pretty notorious for being flakey
          etc</div>
        <div dir="auto">Have had them work with some drives and not
          others, total pita </div>
        <div dir="auto"><br>
        </div>
        <div dir="auto">Rsync over a large volume , particularly with
          lots of small files and directories, is a pretty brutal stress
          test of several subsystems. </div>
        <div dir="auto"><br>
        </div>
        <div dir="auto">/Chris </div>
      </div>
      <br>
      <div class="gmail_quote gmail_quote_container">
        <div dir="ltr" class="gmail_attr">On Thu, 20 Mar 2025, 10:26 am
          Chris McCormick, <<a href="mailto:chris@mccormick.cx"
            moz-do-not-send="true" class="moz-txt-link-freetext">chris@mccormick.cx</a>>
          wrote:<br>
        </div>
        <blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
          <br>
          Thanks for your reply.<br>
          <br>
          smartctl seems to think the hardware is fine. I did smartctl
          -a and it <br>
          reported nothing unusual. I also ran a self test and this was
          the result:<br>
          <br>
          $ sudo smartctl -l selftest /dev/sdb<br>
          smartctl 7.2 2020-12-30 r5155 [aarch64-linux-6.1.21-v8+]
          (local build)<br>
          Copyright (C) 2002-20, Bruce Allen, Christian Franke, <a
            href="http://wwwsmartmontools.org"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true">www.smartmontools.org</a><br>
          <br>
          === START OF READ SMART DATA SECTION ===<br>
          SMART Self-test log structure revision number 1<br>
          Num  Test_Description    Status                  Remaining <br>
          LifeTime(hours)  LBA_of_first_error<br>
          # 1  Short offline       Completed without error       00%   
            6625 <br>
               -<br>
          <br>
          If the drive or the USB hub was overheating would that cause
          IO errors? <br>
          I should also mention that the OS seems to remount the drive a
          read only <br>
          once this happens.<br>
          <br>
          Chris.<br>
          <br>
          <br>
          On 20/03/2025 08:09, Chris Hoy Poy wrote:<br>
          > Hi Chris<br>
          > <br>
          > If it's timing out with IO errors, you probably have a
          hardware problem <br>
          > somewhere in the mix.<br>
          > <br>
          > Cable , connector, hub or USB hard drive (probably the
          destination drive <br>
          > by the sounds of it).<br>
          > <br>
          > Shouldn't matter which software you use to copy (dd,
          rsync etc) - though <br>
          > very different technical processes, if it's dropping data
          somewhere then <br>
          > they will bomb out.<br>
          > <br>
          > Cheers<br>
          > /Chris<br>
          > <br>
          > <br>
          > On Thu, 20 Mar 2025, 8:03 am Chris McCormick, <<a
            href="mailto:chris@mccormick.cx" target="_blank"
            rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">chris@mccormick.cx</a> <br>
          > <mailto:<a href="mailto:chris@mccormick.cx"
            target="_blank" rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">chris@mccormick.cx</a>>>
          wrote:<br>
          > <br>
          >     Hey all,<br>
          > <br>
          >     Unfortunately I couldn't make it to the recent RPi
          meetup, otherwise<br>
          >     that would have been a good place to ask about this<br>
          > <br>
          >     Anyway, I have an RPi with a powered USB hub and two
          USB hard drives<br>
          >     plugged in doing backups. The second drive is a
          mirror of the first<br>
          >     which I have traditionally done with rsync. I
          recently replaced one of<br>
          >     the drives and I now need to mirror the full drive to
          the new drive.<br>
          >     I've been using rsync to do this but it takes ages
          just to do the<br>
          >     initial diff and craps out after a while with IO
          errors and unmounts<br>
          >     the<br>
          >     destination drive. The total size of data is 2.3Tb.<br>
          > <br>
          >     What is a good way to do a fast copy of all of the
          data on one USB<br>
          >     drive<br>
          >     to the other? I'd prefer not to have to stuff around
          with partitions if<br>
          >     possible as they have slightly different partitions
          on each drive and<br>
          >     different labels etc.<br>
          > <br>
          >     LLMs were no help at all with this so I'm relying on
          the PLUG brains<br>
          >     trust! 😅 Thanks for any suggestions.<br>
          > <br>
          >     Chris.<br>
          > <br>
          >     -- <br>
          >     <a href="https://mccormick.cx/"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true" class="moz-txt-link-freetext">https://mccormick.cx/</a>
          <<a href="https://mccormick.cx/"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true" class="moz-txt-link-freetext">https://mccormick.cx/</a>><br>
          >     _______________________________________________<br>
          >     PLUG discussion list: <a
            href="mailto:plug@plug.org.au" target="_blank"
            rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">plug@plug.org.au</a>
          <mailto:<a href="mailto:plug@plug.org.au" target="_blank"
            rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">plug@plug.org.au</a>><br>
          >     <a
            href="http://lists.plug.org.au/mailman/listinfo/plug"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
          >     <<a
            href="http://lists.plug.org.au/mailman/listinfo/plug"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true" class="moz-txt-link-freetext">http://lists.plug.org.au/mailman/listinfo/plug</a>><br>
          >     Committee e-mail: <a
            href="mailto:committee@plug.org.au" target="_blank"
            rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">committee@plug.org.au</a>
          <mailto:<a href="mailto:committee@plug.org.au"
            target="_blank" rel="noreferrer" moz-do-not-send="true"
            class="moz-txt-link-freetext">committee@plug.org.au</a>><br>
          >     PLUG Membership: <a
            href="http://www.plug.org.au/membership"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true">http://www.plugorg.au/membership</a><br>
          >     <<a href="http://www.plug.org.au/membership"
            rel="noreferrer noreferrer" target="_blank"
            moz-do-not-send="true" class="moz-txt-link-freetext">http://www.plug.org.au/membership</a>><br>
          > <br>
          <br>
          -- <br>
          <a href="https://mccormick.cx/" rel="noreferrer noreferrer"
            target="_blank" moz-do-not-send="true"
            class="moz-txt-link-freetext">https://mccormick.cx/</a><br>
        </blockquote>
      </div>
      <br>
      <fieldset class="moz-mime-attachment-header"></fieldset>
      <pre wrap="" class="moz-quote-pre">_______________________________________________
PLUG discussion list: <a class="moz-txt-link-abbreviated" href="mailto:plug@plug.org.au">plug@plug.org.au</a>
<a class="moz-txt-link-freetext" href="http://lists.plug.org.au/mailman/listinfo/plug">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a class="moz-txt-link-abbreviated" href="mailto:committee@plug.org.au">committee@plug.org.au</a>
PLUG Membership: <a class="moz-txt-link-freetext" href="http://www.plug.org.au/membership">http://www.plug.org.au/membership</a></pre>
    </blockquote>
  </body>
</html>