<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi Ben, <br>
</p>
<p>I have heard of shucking drives but have not gone down that
route as it seems too fraught - I do suspect they might be
inferior (software, hardware?) from the cheap prices some of them
are alone. I am using USB because hardware like odroid xu4, c4
only have usb connectors (no SATA, no IDE etc.) ... even the HC2
uses an admittedly good quality internal USB3 to SATA. There are
some interesting threads on the xu4 forums on why linux UAS and
USB in general performs so badly (very common, poor quality
adaptors often using Seagate firmware that may not be standards
compliant) compared to other systems as people were blaming the
hardware - in LinuxLand sometimes the pursuit of perfection gets
in the way of just getting the job done at all.<br>
</p>
<p>BillK<br>
</p>
<div class="moz-cite-prefix">On 26/4/21 8:46 am, Benjamin wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGwEZWK-1t6ZOPzfsBTn+ePXtyKCTG_FJr8GjLm0+Z9PvFZwAA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">Ah. Well, the reason to get externals is because
they're trivial to "shuck", turning them into internal drives.
It becomes difficult to warranty them, and the reliability is
obviously not as good as "real NAS drives", but if you're
getting more than 3 drives they definitely become worth it as
you essentially get 50% more TB/$. If you only have 1-2 drives,
it's not worth the risks.
<div dir="auto"><br>
</div>
<div dir="auto">mfspro becomes worth it once the cost of the
licencing is low enough to offset the equivalent cost of
drives due to the erasure coding which is a really, really
awesome implementation. The break even point is 50TiB mostly
because I think that's the minimum they sell. I personally use
it to great effect, but YMMV.</div>
<div dir="auto"><br>
</div>
<div dir="auto">18TB and up CMR drives are fine, I haven't
noticed any latency issues with any of my use cases.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, 25 Apr 2021, 19:57
William Kenworthy, <<a href="mailto:billk@iinet.net.au"
moz-do-not-send="true">billk@iinet.net.au</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>Thanks Shaun,</p>
<p> good points. I have MFS set up as a single disk per
host except in one case where there are two matched WD
Greens. Data flows are via a VLAN segmented network
between hosts so are isolated. MFS is a chunkserver
system and isn't even close to RAID in concept so avoids
its issues. The MFS recommendations which I largely
followed are to use raw disks with XFS - if you have a
JBOD, LVM or RAID underlying a storage area it will defeat
some of the failure detection and mitigation strategies
MFS uses. With MFS a complete host failure will only take
down its storage and the MFS will completely self-heal
without data loss around the failure as long as there is
spare space and enough recovery time. Currently I can
take any two or three smaller chunkservers completely
off-line at the same time with no lost data or effect on
users and once healed the data redundancy is restored.<br>
</p>
<p>I have a habit of collecting castoff's and re-purposing
hardware so very little of my gear is a similar purchase
in timing or type - something that MFS deals with quite
elegantly as its mostly independent of operating
systems/hardware - I am even mixing 32bit and 64bit
operating systems on arm, arm64 and intel and while I
currently use Gentoo/openrc there is no reason I cant use
a different linux on each host :)</p>
<p>I would think the response times for 8TB and above are
because they are mostly SMR, not the data density per se?
Can you confirm as I don't think its a problem with CMR
drives? WD and Seagate have been caught out sneaking SMR
drives into NAS (where resilvering and SMR are a real
problem) and other product lines and have suffered some
consumer backlash because of it - Both companies now have
lists of which drive and type are SMR or CMR.<br>
</p>
<p>One point I would highlight is USB connected disks can be
a problem (reliability of the connection, throughput is
fine), particularly if UAS is involved and no-name
adaptors. Unfortunately for me all bar the Intel host
with an M.2NVME drive they are either builtin USB3 or
no-name USB3 adaptors so I can speak to experience ...</p>
<p>BillK</p>
<p><br>
</p>
<div>On 25/4/21 6:38 pm, <a
href="mailto:plug_list@holoarc.net" target="_blank"
rel="noreferrer" moz-do-not-send="true">plug_list@holoarc.net</a>
wrote:<br>
</div>
<blockquote type="cite">
<p>My 2 cents (and apologies if this has been covered
already):</p>
<p>I went the other route of building a NAS and having
storage off the NAS instead of vSAN or Distributed File
system approach. My experience/thoughts with consumer
grade hardware on my NAS (using mdadm and ZFS):<br>
</p>
<ol>
<li>Run the same speed etc ideally in the same RAID
group (not sure if mooseFS counters this with using
RAM as cache?). I have been caught out with thinking I
was getting 7.2K RPM drive just find the manufacture
changed drive speeds between different sizes in the
same series of drives (e.g. WD Red I think).
Personally I dislike 5.9k RPM drives...unless they're
in big Enterprise SAN/S3 solution.<br>
</li>
<li>Uses different brands and <b>batch numbers - </b>last
thing you want is have bad batch and they all start
failing around the same time - e.g. buying 5 x WD
blues from same store at the same time is bad idea
(and yes its pain).<br>
</li>
<li>8 TB and above drives have long response latency
(due to density) and thus be careful what
configuration you use and make sure it can handle long
build time</li>
<li>I have had drives die from HGST, Seagate and WD over
the years...HGST died the quickly and were pain to
replace under warranty from memory. <br>
</li>
</ol>
<p>-Shaun<br>
</p>
<div>On 25/04/2021 3:26 pm, Benjamin wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">It's not worth getting anything other
than cheapest non-SMR drives IMO for nearly any use
case... you can get performance by aggregating enough
drives anyways</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Apr 25, 2021
at 3:25 PM Benjamin <<a
href="mailto:zorlin@gmail.com" target="_blank"
rel="noreferrer" moz-do-not-send="true">zorlin@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">LizardFS is a bag of hurt with dead
development. Proceed with hella caution if you go
that route. I hope it changes and becomes worth
pursuing though.
<div><br>
</div>
<div>MFSpro is justifiable around 50TiB and up,
until then it's not really worth it.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Apr 25,
2021 at 3:22 PM William Kenworthy <<a
href="mailto:billk@iinet.net.au"
target="_blank" rel="noreferrer"
moz-do-not-send="true">billk@iinet.net.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Thanks Ben and Paul - this backs up my
readings/experience.</p>
<p>I will shortly need a new archive drive
because I have lest than 80Gb left on the
2Tb WD green I have been using for a few
years. As performance isn't an issue I will
likely go with a Seagate Barracuda this time
(still debating shingled or not because this
use is more cost sensitive than performance
on writing new data across a network - so
low priority, busy, but not excessively so
when in use - I am happy to allow time for
the shingling resilvering to complete as
long as it doesn't impact time to actually
backup the data too much.)</p>
<p>Moosefs is more difficult to quantify whats
needed - currently:</p>
<p>8 hosts (8 HDD, 1x M2.SSD, 6x arm32, 1x
arm64 and 1x intel - all odroid using
gentoo)</p>
<p>~21Tb space, 3/4 in use. I could delete
some as there is duplicate data stored so if
I lose a drive I can reclaim space easily as
well as decrease the goal in some places.</p>
<p>As well, I am using storage classes. High
use data has mostly 1 chunk on the intel/SSD
for performance and others on HDD's. I have
sc's ranging from 1 to 4 copies with 2, 3
and 4 in common use ... for example things
like VM's where there are hot spots with
temp file creation I have 2 copies (2SH)
whereas backups and user data have 4 copies
4HHHH or 4SHHH depending on priority (eg,
/home). Currently I have one WD Green drive
I would already toss if in a commercial
system, and two Seagate NAS drives I am not
totally happy with.<br>
</p>
<p>For these, definitely non-shingled (CMR)
7200rpm around 4TB seems ideal - but is a
NAS optimised drive useful or a waste for
moosefs? - vibration of nearby drives is the
only thing I can think of. Some are bound
together (5x odroid HC2) and some are in
pairs in relatively heavy PC case baymounts
(removed/pinched - from my sons ongoing
gaming PC build :) placed on a desk. I am
staring to lean towards the WD blacks for
this, but the HGST lines WD are starting to
integrate are interesting though more
expensive ... <br>
</p>
<p>I would love to have MFSpro but cant
justify it as super uptime isn't necessary,
EC isn't really attractive at my scale and
multiple masters isn't essential as I have
plenty of alternative systems I could bring
in quickly ... though I am watching lizardfs
and just might jump to it to get the
multiple masters that is in the free tier.<br>
</p>
<p>BillK</p>
<p><br>
</p>
<div>On 25/4/21 1:19 pm, Benjamin wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">+1 to all of it, cheers
Paul.
<div dir="auto"><br>
</div>
<div dir="auto">I think it's worth going
for the cheapest externals you can get,
shucking them, then using MooseFS since
you're already planning to.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I'd use copies=3 and if
you're storing more than 50TB talk to me
about mfspro.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun,
25 Apr 2021, 13:03 Paul Del, <<a
href="mailto:p@delfante.it"
target="_blank" rel="noreferrer"
moz-do-not-send="true">p@delfante.it</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello Bill
<div><br>
</div>
<div>My 2 cents worth<br>
<div><br>
</div>
<div>I am sure you know the common
things that can increase your hard
drives life and performance:</div>
<div>Temperature</div>
<div>Humidity</div>
<div>VIbration</div>
<div>Heavy Writes</div>
<div>Heaving Logging</div>
<div>Clean/Reliable power</div>
<div>Data throughput</div>
<div><br>
</div>
<div>The rust hard drives I have
seen the most failures with are:
(I recommend avoiding)</div>
<div>WD Green</div>
<div>WD Blue</div>
<div>Hitachi Deskstar</div>
<div>(Not The server drives)</div>
<div>
<div><br>
</div>
<div>The rust hard drives I
recommend the most are:</div>
<div>WD Black 7200rpm or better</div>
<div>Seagate 7200pm or better</div>
<div>(Not Red, Blue, Green,
Purple)</div>
<div><br>
</div>
<div>If you are doing the moose
distribute setup</div>
<div>You could always choose two
different brands/types</div>
<div><br>
</div>
<div>if you want to know more
specific things about which hard
drive failures. Check out this
from backblaze, I am sure
there's more around. Which is
one Benjamin sent around ages
ago.</div>
<div><a
href="https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/"
rel="noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/</a><br>
</div>
<div><a
href="https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/"
rel="noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/</a><br>
</div>
<div><br>
</div>
<div>Thanks Paul</div>
<div><br>
</div>
<div><br>
</div>
<div>On Sat, 24 Apr 2021, 09:02
William Kenworthy, <<a
href="mailto:billk@iinet.net.au"
rel="noreferrer noreferrer"
target="_blank"
moz-do-not-send="true">billk@iinet.net.au</a>>
wrote:<br>
<br>
> Just musing on what changes
I could make to streamline my
systems:<br>
><br>
> After a recent stray "r m
- r f " with a space in it I
ended up<br>
> removing both most of my
active data files, VM's etc ...
and the online<br>
> backups - ouch!<br>
><br>
> I have restored from
offline backups and have noticed
a ~10years old WD<br>
> green drive showing a few
early symptoms of failing
(SMART).<br>
><br>
> With the plethora of
colours now available (!) now
what drive is best for<br>
> a:<br>
><br>
> 1. moosefs chunkserver
(stores files for VM's, data
including the<br>
> mail servers user files,
home directories and of course
the online<br>
> borgbackup archives - the
disks are basically hammered all
the time.)<br>
><br>
> 2. offline backups
(~2tb data using borgbackup to
backup the online<br>
> borgbackup repo, used twice
a week for a few minutes at a
time.)<br>
><br>
> My longest serving drives
are WD greens 2Tb which until
now have just<br>
> keep ticking along. The
failing drive is a WD Green - I
have run<br>
> badblocks on it overnight
with no errors so far so it
might have<br>
> internally remapped the
failed sectors ok - I am using
xfs which does<br>
> not have badblock support.
Most drives spent previous years
in btrfs<br>
> raid 10's or ceph so they
have had a hard life!<br>
><br>
> Newer WD Reds and a Red pro
have failed over the years but I
still have<br>
> two in the mix (6tb and
2tb)<br>
><br>
> Some Seagate Ironwolfs that
show some SMART errors Backblaze
correlate<br>
> with drive failure and
throw an occasional USB
interface error but<br>
> otherwise seem OK.<br>
><br>
> There are shingled,
non-shingled drives,
surveillance, NAS flavours etc.<br>
> - but what have people had
success with? - or should I just
choose my<br>
> favourite colour and run
with it?<br>
><br>
> Thoughts?<br>
><br>
> BillK<br>
</div>
</div>
</div>
<img alt="" style="display:flex"
src="https://mailtrack.io/trace/mail/68659f62480c3e88ffa2f3ae2fde66bb4c88a16f.png?u=7051199"
moz-do-not-send="true" width="0"
height="0"></div>
_______________________________________________<br>
PLUG discussion list: <a
href="mailto:plug@plug.org.au"
rel="noreferrer noreferrer"
target="_blank" moz-do-not-send="true">plug@plug.org.au</a><br>
<a
href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a
href="mailto:committee@plug.org.au"
rel="noreferrer noreferrer"
target="_blank" moz-do-not-send="true">committee@plug.org.au</a><br>
PLUG Membership: <a
href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer noreferrer"
target="_blank" moz-do-not-send="true">http://www.plug.org.au/membership</a></blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">plug@plug.org.au</a>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank" rel="noreferrer" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">committee@plug.org.au</a>
PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank" rel="noreferrer" moz-do-not-send="true">http://www.plug.org.au/membership</a></pre>
</blockquote>
</div>
_______________________________________________<br>
PLUG discussion list: <a
href="mailto:plug@plug.org.au" target="_blank"
rel="noreferrer" moz-do-not-send="true">plug@plug.org.au</a><br>
<a
href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a
href="mailto:committee@plug.org.au"
target="_blank" rel="noreferrer"
moz-do-not-send="true">committee@plug.org.au</a><br>
PLUG Membership: <a
href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://www.plug.org.au/membership</a></blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">plug@plug.org.au</a>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank" rel="noreferrer" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">committee@plug.org.au</a>
PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank" rel="noreferrer" moz-do-not-send="true">http://www.plug.org.au/membership</a></pre>
</blockquote>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">plug@plug.org.au</a>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank" rel="noreferrer" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" rel="noreferrer" moz-do-not-send="true">committee@plug.org.au</a>
PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank" rel="noreferrer" moz-do-not-send="true">http://www.plug.org.au/membership</a></pre>
</blockquote>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au"
target="_blank" rel="noreferrer" moz-do-not-send="true">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a href="mailto:committee@plug.org.au"
target="_blank" rel="noreferrer" moz-do-not-send="true">committee@plug.org.au</a><br>
PLUG Membership: <a href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://www.plug.org.au/membership</a></blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
PLUG discussion list: <a class="moz-txt-link-abbreviated" href="mailto:plug@plug.org.au">plug@plug.org.au</a>
<a class="moz-txt-link-freetext" href="http://lists.plug.org.au/mailman/listinfo/plug">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a class="moz-txt-link-abbreviated" href="mailto:committee@plug.org.au">committee@plug.org.au</a>
PLUG Membership: <a class="moz-txt-link-freetext" href="http://www.plug.org.au/membership">http://www.plug.org.au/membership</a></pre>
</blockquote>
</body>
</html>