[plug] musing on HDD types (William Kenworthy)

Benjamin zorlin at gmail.com
Mon Apr 26 08:46:47 AWST 2021 

Ah. Well, the reason to get externals is because they're trivial to
"shuck", turning them into internal drives. It becomes difficult to
warranty them, and the reliability is obviously not as good as "real NAS
drives", but if you're getting more than 3 drives they definitely become
worth it as you essentially get 50% more TB/$. If you only have 1-2 drives,
it's not worth the risks.

mfspro becomes worth it once the cost of the licencing is low enough to
offset the equivalent cost of drives due to the erasure coding which is a
really, really awesome implementation. The break even point is 50TiB mostly
because I think that's the minimum they sell. I personally use it to great
effect, but YMMV.

18TB and up CMR drives are fine, I haven't noticed any latency issues with
any of my use cases.

On Sun, 25 Apr 2021, 19:57 William Kenworthy, <billk at iinet.net.au> wrote:

> Thanks Shaun,
>
>     good points.  I have MFS set up as a single disk per host except in
> one case where there are two matched WD Greens.  Data flows are via a VLAN
> segmented network between hosts so are isolated.  MFS is a chunkserver
> system and isn't even close to RAID in concept so avoids its issues.  The
> MFS recommendations which I largely followed are to use raw disks with XFS
> - if you have a JBOD, LVM or RAID underlying a storage area it will defeat
> some of the failure detection and mitigation strategies MFS uses.  With MFS
> a complete host failure will only take down its storage and the MFS will
> completely self-heal without data loss around the failure as long as there
> is spare space and enough recovery time.  Currently I can take any two or
> three smaller chunkservers completely off-line at the same time with no
> lost data or effect on users and once healed the data redundancy is
> restored.
>
> I have a habit of collecting castoff's and re-purposing hardware so very
> little of my gear is a similar purchase in timing or type - something that
> MFS deals with quite elegantly as its mostly independent of operating
> systems/hardware - I am even mixing 32bit and 64bit operating systems on
> arm, arm64 and intel and while I currently use Gentoo/openrc there is no
> reason I cant use a different linux on each host :)
>
> I would think the response times for 8TB and above are because they are
> mostly SMR, not the data density per se?  Can you confirm as I don't think
> its a problem with CMR drives?  WD and Seagate have been caught out
> sneaking SMR drives into NAS (where resilvering and SMR are a real problem)
> and other product lines and have suffered some consumer backlash because of
> it - Both companies now have lists of which drive and type are SMR or CMR.
>
> One point I would highlight is USB connected disks can be a problem
> (reliability of the connection, throughput is fine), particularly if UAS is
> involved and no-name adaptors.  Unfortunately for me all bar the Intel host
> with an M.2NVME drive they are either builtin USB3 or no-name USB3 adaptors
> so I can speak to experience ...
>
> BillK
>
>
> On 25/4/21 6:38 pm, plug_list at holoarc.net wrote:
>
> My 2 cents (and apologies if this has been covered already):
>
> I went the other route of building a NAS and having storage off the NAS
> instead of vSAN or Distributed File system approach. My experience/thoughts
> with consumer grade hardware on my NAS (using mdadm and ZFS):
>
>    1. Run the same speed etc ideally in the same RAID group (not sure if
>    mooseFS counters this with using RAM as cache?). I have been caught out
>    with thinking I was getting 7.2K RPM drive just find the manufacture
>    changed drive speeds between different sizes in the same series of drives
>    (e.g. WD Red I think). Personally I dislike 5.9k RPM drives...unless
>    they're in big Enterprise SAN/S3 solution.
>    2. Uses different brands and *batch numbers - *last thing you want is
>    have bad batch and they all start failing around the same time - e.g.
>    buying 5 x WD blues from same store at the same time is bad idea (and yes
>    its pain).
>    3. 8 TB and above drives have long response latency (due to density)
>    and thus be careful what configuration you use and make sure it can handle
>    long build time
>    4. I have had drives die from HGST, Seagate and WD over the
>    years...HGST died the quickly and were pain to replace under warranty from
>    memory.
>
> -Shaun
> On 25/04/2021 3:26 pm, Benjamin wrote:
>
> It's not worth getting anything other than cheapest non-SMR drives IMO for
> nearly any use case... you can get performance by aggregating enough drives
> anyways
>
> On Sun, Apr 25, 2021 at 3:25 PM Benjamin <zorlin at gmail.com> wrote:
>
>> LizardFS is a bag of hurt with dead development. Proceed with hella
>> caution if you go that route. I hope it changes and becomes worth pursuing
>> though.
>>
>> MFSpro is justifiable around 50TiB and up, until then it's not really
>> worth it.
>>
>> On Sun, Apr 25, 2021 at 3:22 PM William Kenworthy <billk at iinet.net.au>
>> wrote:
>>
>>> Thanks Ben and Paul - this backs up my readings/experience.
>>>
>>> I will shortly need a new archive drive because I have lest than 80Gb
>>> left on the 2Tb WD green I have been using for a  few years.  As
>>> performance isn't an issue I will likely go with a Seagate Barracuda this
>>> time (still debating shingled or not because this use is more cost
>>> sensitive than performance on writing new data across a network - so low
>>> priority, busy, but not excessively so when in use - I am happy to allow
>>> time for the shingling resilvering to complete as long as it doesn't impact
>>> time to actually backup the data too much.)
>>>
>>> Moosefs is more difficult to quantify whats needed - currently:
>>>
>>> 8 hosts (8 HDD, 1x M2.SSD, 6x arm32, 1x arm64 and 1x intel - all odroid
>>> using gentoo)
>>>
>>> ~21Tb space, 3/4 in use. I could delete some as there is duplicate data
>>> stored so if I lose a drive I can reclaim space easily as well as decrease
>>> the goal in some places.
>>>
>>> As well, I am using storage classes.  High use data has mostly 1 chunk
>>> on the intel/SSD for performance and others on HDD's.  I have sc's ranging
>>> from 1 to 4 copies with 2, 3 and 4 in common use ... for example things
>>> like VM's where there are hot spots with temp file creation I have 2 copies
>>> (2SH) whereas backups and user data have 4 copies 4HHHH or 4SHHH depending
>>> on priority (eg, /home).  Currently I have one WD Green drive I would
>>> already toss if in a commercial system, and two Seagate NAS drives I am not
>>> totally happy with.
>>>
>>> For these, definitely non-shingled (CMR) 7200rpm around 4TB seems ideal
>>> - but is a NAS optimised drive useful or a waste for moosefs? - vibration
>>> of nearby drives is the only thing I can think of.  Some are bound together
>>> (5x odroid HC2) and some are in pairs in relatively heavy PC case baymounts
>>> (removed/pinched - from my sons ongoing gaming PC build :) placed on a
>>> desk.  I am staring to lean towards the WD blacks for this, but the HGST
>>> lines WD are starting to integrate are interesting though more expensive
>>> ...
>>>
>>> I would love to have MFSpro but cant justify it as super uptime isn't
>>> necessary, EC isn't really attractive at my scale and multiple masters
>>> isn't essential as I have plenty of alternative systems I could bring in
>>> quickly ... though I am watching lizardfs and just might jump to it to get
>>> the multiple masters that is in the free tier.
>>>
>>> BillK
>>>
>>>
>>> On 25/4/21 1:19 pm, Benjamin wrote:
>>>
>>> +1 to all of it, cheers Paul.
>>>
>>> I think it's worth going for the cheapest externals you can get,
>>> shucking them, then using MooseFS since you're already planning to.
>>>
>>> I'd use copies=3 and if you're storing more than 50TB talk to me about
>>> mfspro.
>>>
>>> On Sun, 25 Apr 2021, 13:03 Paul Del, <p at delfante.it> wrote:
>>>
>>>> Hello Bill
>>>>
>>>> My 2 cents worth
>>>>
>>>> I am sure you know the common things that can increase your hard drives
>>>> life and performance:
>>>> Temperature
>>>> Humidity
>>>> VIbration
>>>> Heavy Writes
>>>> Heaving Logging
>>>> Clean/Reliable power
>>>> Data throughput
>>>>
>>>> The rust hard drives I have seen the most failures with are: (I
>>>> recommend avoiding)
>>>> WD Green
>>>> WD Blue
>>>> Hitachi Deskstar
>>>> (Not The server drives)
>>>>
>>>> The rust hard drives I recommend the most are:
>>>> WD Black 7200rpm or better
>>>> Seagate 7200pm or better
>>>> (Not Red, Blue, Green, Purple)
>>>>
>>>> If you are doing the moose distribute setup
>>>> You could always choose two different brands/types
>>>>
>>>> if you want to know more specific things about which hard drive
>>>> failures. Check out this from backblaze, I am sure there's more around.
>>>> Which is one Benjamin sent around ages ago.
>>>> https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/
>>>> https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/
>>>>
>>>> Thanks Paul
>>>>
>>>>
>>>> On Sat, 24 Apr 2021, 09:02 William Kenworthy, <billk at iinet.net.au>
>>>> wrote:
>>>>
>>>> > Just musing on what changes I could make to streamline my systems:
>>>> >
>>>> > After a recent stray "r m  - r f " with a space in it I ended up
>>>> > removing both most of my active data files, VM's etc ... and the
>>>> online
>>>> > backups - ouch!
>>>> >
>>>> > I have restored from offline backups and have noticed a ~10years old
>>>> WD
>>>> > green drive showing a few early symptoms of failing (SMART).
>>>> >
>>>> > With the plethora of colours now available (!) now what drive is best
>>>> for
>>>> > a:
>>>> >
>>>> >     1. moosefs chunkserver (stores files for VM's, data including the
>>>> > mail servers user files, home directories and of course the online
>>>> > borgbackup archives - the disks are basically hammered all the time.)
>>>> >
>>>> >     2. offline backups (~2tb data using borgbackup to backup the
>>>> online
>>>> > borgbackup repo, used twice a week for a few minutes at a time.)
>>>> >
>>>> > My longest serving drives are WD greens 2Tb which until now have just
>>>> > keep ticking along.  The failing drive is a WD Green - I have run
>>>> > badblocks on it overnight with no errors so far so it might have
>>>> > internally remapped the failed sectors ok - I am using xfs which does
>>>> > not have badblock support.  Most drives spent previous years in btrfs
>>>> > raid 10's or ceph so they have had a hard life!
>>>> >
>>>> > Newer WD Reds and a Red pro have failed over the years but I still
>>>> have
>>>> > two in the mix (6tb and 2tb)
>>>> >
>>>> > Some Seagate Ironwolfs that show some SMART errors Backblaze correlate
>>>> > with drive failure and throw an occasional USB interface error but
>>>> > otherwise seem OK.
>>>> >
>>>> > There are shingled, non-shingled drives, surveillance, NAS flavours
>>>> etc.
>>>> > - but what have people had success with? - or should I just choose my
>>>> > favourite colour and run with it?
>>>> >
>>>> > Thoughts?
>>>> >
>>>> > BillK
>>>> _______________________________________________
>>>> PLUG discussion list: plug at plug.org.au
>>>> http://lists.plug.org.au/mailman/listinfo/plug
>>>> Committee e-mail: committee at plug.org.au
>>>> PLUG Membership: http://www.plug.org.au/membership
>>>
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.auhttp://lists.plug.org.au/mailman/listinfo/plug
>>> Committee e-mail: committee at plug.org.au
>>> PLUG Membership: http://www.plug.org.au/membership
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au
>>> http://lists.plug.org.au/mailman/listinfo/plug
>>> Committee e-mail: committee at plug.org.au
>>> PLUG Membership: http://www.plug.org.au/membership
>>
>>
> _______________________________________________
> PLUG discussion list: plug at plug.org.auhttp://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.auhttp://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20210426/37ac7b54/attachment.html>

More information about the plug mailing list