<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Thanks Shaun,</p>
<p> good points. I have MFS set up as a single disk per host
except in one case where there are two matched WD Greens. Data
flows are via a VLAN segmented network between hosts so are
isolated. MFS is a chunkserver system and isn't even close to
RAID in concept so avoids its issues. The MFS recommendations
which I largely followed are to use raw disks with XFS - if you
have a JBOD, LVM or RAID underlying a storage area it will defeat
some of the failure detection and mitigation strategies MFS uses.
With MFS a complete host failure will only take down its storage
and the MFS will completely self-heal without data loss around the
failure as long as there is spare space and enough recovery time.
Currently I can take any two or three smaller chunkservers
completely off-line at the same time with no lost data or effect
on users and once healed the data redundancy is restored.<br>
</p>
<p>I have a habit of collecting castoff's and re-purposing hardware
so very little of my gear is a similar purchase in timing or type
- something that MFS deals with quite elegantly as its mostly
independent of operating systems/hardware - I am even mixing 32bit
and 64bit operating systems on arm, arm64 and intel and while I
currently use Gentoo/openrc there is no reason I cant use a
different linux on each host :)</p>
<p>I would think the response times for 8TB and above are because
they are mostly SMR, not the data density per se? Can you confirm
as I don't think its a problem with CMR drives? WD and Seagate
have been caught out sneaking SMR drives into NAS (where
resilvering and SMR are a real problem) and other product lines
and have suffered some consumer backlash because of it - Both
companies now have lists of which drive and type are SMR or CMR.<br>
</p>
<p>One point I would highlight is USB connected disks can be a
problem (reliability of the connection, throughput is fine),
particularly if UAS is involved and no-name adaptors.
Unfortunately for me all bar the Intel host with an M.2NVME drive
they are either builtin USB3 or no-name USB3 adaptors so I can
speak to experience ...</p>
<p>BillK</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 25/4/21 6:38 pm,
<a class="moz-txt-link-abbreviated" href="mailto:plug_list@holoarc.net">plug_list@holoarc.net</a> wrote:<br>
</div>
<blockquote type="cite"
cite="mid:22f110c8-2080-28a9-10ab-087ff35e5952@holoarc.net">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>My 2 cents (and apologies if this has been covered already):</p>
<p>I went the other route of building a NAS and having storage off
the NAS instead of vSAN or Distributed File system approach. My
experience/thoughts with consumer grade hardware on my NAS
(using mdadm and ZFS):<br>
</p>
<ol>
<li>Run the same speed etc ideally in the same RAID group (not
sure if mooseFS counters this with using RAM as cache?). I
have been caught out with thinking I was getting 7.2K RPM
drive just find the manufacture changed drive speeds between
different sizes in the same series of drives (e.g. WD Red I
think). Personally I dislike 5.9k RPM drives...unless they're
in big Enterprise SAN/S3 solution.<br>
</li>
<li>Uses different brands and <b>batch numbers - </b>last
thing you want is have bad batch and they all start failing
around the same time - e.g. buying 5 x WD blues from same
store at the same time is bad idea (and yes its pain).<br>
</li>
<li>8 TB and above drives have long response latency (due to
density) and thus be careful what configuration you use and
make sure it can handle long build time</li>
<li>I have had drives die from HGST, Seagate and WD over the
years...HGST died the quickly and were pain to replace under
warranty from memory. <br>
</li>
</ol>
<p>-Shaun<br>
</p>
<div class="moz-cite-prefix">On 25/04/2021 3:26 pm, Benjamin
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGwEZWJCbAWLRMsSusxPJRvKM6a95R2rgEiYiHj_FNzX0bLMmg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">It's not worth getting anything other than
cheapest non-SMR drives IMO for nearly any use case... you can
get performance by aggregating enough drives anyways</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Apr 25, 2021 at 3:25
PM Benjamin <<a href="mailto:zorlin@gmail.com"
moz-do-not-send="true">zorlin@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">LizardFS is a bag of hurt with dead
development. Proceed with hella caution if you go that
route. I hope it changes and becomes worth pursuing
though.
<div><br>
</div>
<div>MFSpro is justifiable around 50TiB and up, until then
it's not really worth it.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Apr 25, 2021 at
3:22 PM William Kenworthy <<a
href="mailto:billk@iinet.net.au" target="_blank"
moz-do-not-send="true">billk@iinet.net.au</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Thanks Ben and Paul - this backs up my
readings/experience.</p>
<p>I will shortly need a new archive drive because I
have lest than 80Gb left on the 2Tb WD green I have
been using for a few years. As performance isn't
an issue I will likely go with a Seagate Barracuda
this time (still debating shingled or not because
this use is more cost sensitive than performance on
writing new data across a network - so low priority,
busy, but not excessively so when in use - I am
happy to allow time for the shingling resilvering to
complete as long as it doesn't impact time to
actually backup the data too much.)</p>
<p>Moosefs is more difficult to quantify whats needed
- currently:</p>
<p>8 hosts (8 HDD, 1x M2.SSD, 6x arm32, 1x arm64 and
1x intel - all odroid using gentoo)</p>
<p>~21Tb space, 3/4 in use. I could delete some as
there is duplicate data stored so if I lose a drive
I can reclaim space easily as well as decrease the
goal in some places.</p>
<p>As well, I am using storage classes. High use data
has mostly 1 chunk on the intel/SSD for performance
and others on HDD's. I have sc's ranging from 1 to
4 copies with 2, 3 and 4 in common use ... for
example things like VM's where there are hot spots
with temp file creation I have 2 copies (2SH)
whereas backups and user data have 4 copies 4HHHH or
4SHHH depending on priority (eg, /home). Currently
I have one WD Green drive I would already toss if in
a commercial system, and two Seagate NAS drives I am
not totally happy with.<br>
</p>
<p>For these, definitely non-shingled (CMR) 7200rpm
around 4TB seems ideal - but is a NAS optimised
drive useful or a waste for moosefs? - vibration of
nearby drives is the only thing I can think of.
Some are bound together (5x odroid HC2) and some are
in pairs in relatively heavy PC case baymounts
(removed/pinched - from my sons ongoing gaming PC
build :) placed on a desk. I am staring to lean
towards the WD blacks for this, but the HGST lines
WD are starting to integrate are interesting though
more expensive ... <br>
</p>
<p>I would love to have MFSpro but cant justify it as
super uptime isn't necessary, EC isn't really
attractive at my scale and multiple masters isn't
essential as I have plenty of alternative systems I
could bring in quickly ... though I am watching
lizardfs and just might jump to it to get the
multiple masters that is in the free tier.<br>
</p>
<p>BillK</p>
<p><br>
</p>
<div>On 25/4/21 1:19 pm, Benjamin wrote:<br>
</div>
<blockquote type="cite">
<div dir="auto">+1 to all of it, cheers Paul.
<div dir="auto"><br>
</div>
<div dir="auto">I think it's worth going for the
cheapest externals you can get, shucking them,
then using MooseFS since you're already planning
to.</div>
<div dir="auto"><br>
</div>
<div dir="auto">I'd use copies=3 and if you're
storing more than 50TB talk to me about mfspro.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, 25 Apr
2021, 13:03 Paul Del, <<a
href="mailto:p@delfante.it" target="_blank"
moz-do-not-send="true">p@delfante.it</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Hello Bill
<div><br>
</div>
<div>My 2 cents worth<br>
<div><br>
</div>
<div>I am sure you know the common things
that can increase your hard drives life
and performance:</div>
<div>Temperature</div>
<div>Humidity</div>
<div>VIbration</div>
<div>Heavy Writes</div>
<div>Heaving Logging</div>
<div>Clean/Reliable power</div>
<div>Data throughput</div>
<div><br>
</div>
<div>The rust hard drives I have seen the
most failures with are: (I recommend
avoiding)</div>
<div>WD Green</div>
<div>WD Blue</div>
<div>Hitachi Deskstar</div>
<div>(Not The server drives)</div>
<div>
<div><br>
</div>
<div>The rust hard drives I recommend the
most are:</div>
<div>WD Black 7200rpm or better</div>
<div>Seagate 7200pm or better</div>
<div>(Not Red, Blue, Green, Purple)</div>
<div><br>
</div>
<div>If you are doing the moose distribute
setup</div>
<div>You could always choose two different
brands/types</div>
<div><br>
</div>
<div>if you want to know more specific
things about which hard drive failures.
Check out this from backblaze, I am sure
there's more around. Which is one
Benjamin sent around ages ago.</div>
<div><a
href="https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/</a><br>
</div>
<div><a
href="https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/</a><br>
</div>
<div><br>
</div>
<div>Thanks Paul</div>
<div><br>
</div>
<div><br>
</div>
<div>On Sat, 24 Apr 2021, 09:02 William
Kenworthy, <<a
href="mailto:billk@iinet.net.au"
rel="noreferrer" target="_blank"
moz-do-not-send="true">billk@iinet.net.au</a>>
wrote:<br>
<br>
> Just musing on what changes I could
make to streamline my systems:<br>
><br>
> After a recent stray "r m - r f "
with a space in it I ended up<br>
> removing both most of my active
data files, VM's etc ... and the online<br>
> backups - ouch!<br>
><br>
> I have restored from offline
backups and have noticed a ~10years old
WD<br>
> green drive showing a few early
symptoms of failing (SMART).<br>
><br>
> With the plethora of colours now
available (!) now what drive is best for<br>
> a:<br>
><br>
> 1. moosefs chunkserver (stores
files for VM's, data including the<br>
> mail servers user files, home
directories and of course the online<br>
> borgbackup archives - the disks are
basically hammered all the time.)<br>
><br>
> 2. offline backups (~2tb data
using borgbackup to backup the online<br>
> borgbackup repo, used twice a week
for a few minutes at a time.)<br>
><br>
> My longest serving drives are WD
greens 2Tb which until now have just<br>
> keep ticking along. The failing
drive is a WD Green - I have run<br>
> badblocks on it overnight with no
errors so far so it might have<br>
> internally remapped the failed
sectors ok - I am using xfs which does<br>
> not have badblock support. Most
drives spent previous years in btrfs<br>
> raid 10's or ceph so they have had
a hard life!<br>
><br>
> Newer WD Reds and a Red pro have
failed over the years but I still have<br>
> two in the mix (6tb and 2tb)<br>
><br>
> Some Seagate Ironwolfs that show
some SMART errors Backblaze correlate<br>
> with drive failure and throw an
occasional USB interface error but<br>
> otherwise seem OK.<br>
><br>
> There are shingled, non-shingled
drives, surveillance, NAS flavours etc.<br>
> - but what have people had success
with? - or should I just choose my<br>
> favourite colour and run with it?<br>
><br>
> Thoughts?<br>
><br>
> BillK<br>
</div>
</div>
</div>
<img alt="" style="display: flex;"
src="https://mailtrack.io/trace/mail/68659f62480c3e88ffa2f3ae2fde66bb4c88a16f.png?u=7051199"
moz-do-not-send="true" width="0" height="0"></div>
_______________________________________________<br>
PLUG discussion list: <a
href="mailto:plug@plug.org.au"
rel="noreferrer" target="_blank"
moz-do-not-send="true">plug@plug.org.au</a><br>
<a
href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a
href="mailto:committee@plug.org.au"
rel="noreferrer" target="_blank"
moz-do-not-send="true">committee@plug.org.au</a><br>
PLUG Membership: <a
href="http://www.plug.org.au/membership"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">http://www.plug.org.au/membership</a></blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
PLUG discussion list: <a href="mailto:plug@plug.org.au" target="_blank" moz-do-not-send="true">plug@plug.org.au</a>
<a href="http://lists.plug.org.au/mailman/listinfo/plug" target="_blank" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a href="mailto:committee@plug.org.au" target="_blank" moz-do-not-send="true">committee@plug.org.au</a>
PLUG Membership: <a href="http://www.plug.org.au/membership" target="_blank" moz-do-not-send="true">http://www.plug.org.au/membership</a></pre>
</blockquote>
</div>
_______________________________________________<br>
PLUG discussion list: <a href="mailto:plug@plug.org.au"
target="_blank" moz-do-not-send="true">plug@plug.org.au</a><br>
<a href="http://lists.plug.org.au/mailman/listinfo/plug"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a><br>
Committee e-mail: <a
href="mailto:committee@plug.org.au" target="_blank"
moz-do-not-send="true">committee@plug.org.au</a><br>
PLUG Membership: <a
href="http://www.plug.org.au/membership"
rel="noreferrer" target="_blank"
moz-do-not-send="true">http://www.plug.org.au/membership</a></blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
PLUG discussion list: <a class="moz-txt-link-abbreviated" href="mailto:plug@plug.org.au" moz-do-not-send="true">plug@plug.org.au</a>
<a class="moz-txt-link-freetext" href="http://lists.plug.org.au/mailman/listinfo/plug" moz-do-not-send="true">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a class="moz-txt-link-abbreviated" href="mailto:committee@plug.org.au" moz-do-not-send="true">committee@plug.org.au</a>
PLUG Membership: <a class="moz-txt-link-freetext" href="http://www.plug.org.au/membership" moz-do-not-send="true">http://www.plug.org.au/membership</a></pre>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
PLUG discussion list: <a class="moz-txt-link-abbreviated" href="mailto:plug@plug.org.au">plug@plug.org.au</a>
<a class="moz-txt-link-freetext" href="http://lists.plug.org.au/mailman/listinfo/plug">http://lists.plug.org.au/mailman/listinfo/plug</a>
Committee e-mail: <a class="moz-txt-link-abbreviated" href="mailto:committee@plug.org.au">committee@plug.org.au</a>
PLUG Membership: <a class="moz-txt-link-freetext" href="http://www.plug.org.au/membership">http://www.plug.org.au/membership</a></pre>
</blockquote>
</body>
</html>