[plug] musing on HDD types (William Kenworthy)
William Kenworthy
billk at iinet.net.au
Sun Apr 25 19:55:56 AWST 2021
Thanks Shaun,
good points. I have MFS set up as a single disk per host except in
one case where there are two matched WD Greens. Data flows are via a
VLAN segmented network between hosts so are isolated. MFS is a
chunkserver system and isn't even close to RAID in concept so avoids its
issues. The MFS recommendations which I largely followed are to use raw
disks with XFS - if you have a JBOD, LVM or RAID underlying a storage
area it will defeat some of the failure detection and mitigation
strategies MFS uses. With MFS a complete host failure will only take
down its storage and the MFS will completely self-heal without data loss
around the failure as long as there is spare space and enough recovery
time. Currently I can take any two or three smaller chunkservers
completely off-line at the same time with no lost data or effect on
users and once healed the data redundancy is restored.
I have a habit of collecting castoff's and re-purposing hardware so very
little of my gear is a similar purchase in timing or type - something
that MFS deals with quite elegantly as its mostly independent of
operating systems/hardware - I am even mixing 32bit and 64bit operating
systems on arm, arm64 and intel and while I currently use Gentoo/openrc
there is no reason I cant use a different linux on each host :)
I would think the response times for 8TB and above are because they are
mostly SMR, not the data density per se? Can you confirm as I don't
think its a problem with CMR drives? WD and Seagate have been caught
out sneaking SMR drives into NAS (where resilvering and SMR are a real
problem) and other product lines and have suffered some consumer
backlash because of it - Both companies now have lists of which drive
and type are SMR or CMR.
One point I would highlight is USB connected disks can be a problem
(reliability of the connection, throughput is fine), particularly if UAS
is involved and no-name adaptors. Unfortunately for me all bar the
Intel host with an M.2NVME drive they are either builtin USB3 or no-name
USB3 adaptors so I can speak to experience ...
BillK
On 25/4/21 6:38 pm, plug_list at holoarc.net wrote:
>
> My 2 cents (and apologies if this has been covered already):
>
> I went the other route of building a NAS and having storage off the
> NAS instead of vSAN or Distributed File system approach. My
> experience/thoughts with consumer grade hardware on my NAS (using
> mdadm and ZFS):
>
> 1. Run the same speed etc ideally in the same RAID group (not sure if
> mooseFS counters this with using RAM as cache?). I have been
> caught out with thinking I was getting 7.2K RPM drive just find
> the manufacture changed drive speeds between different sizes in
> the same series of drives (e.g. WD Red I think). Personally I
> dislike 5.9k RPM drives...unless they're in big Enterprise SAN/S3
> solution.
> 2. Uses different brands and *batch numbers - *last thing you want is
> have bad batch and they all start failing around the same time -
> e.g. buying 5 x WD blues from same store at the same time is bad
> idea (and yes its pain).
> 3. 8 TB and above drives have long response latency (due to density)
> and thus be careful what configuration you use and make sure it
> can handle long build time
> 4. I have had drives die from HGST, Seagate and WD over the
> years...HGST died the quickly and were pain to replace under
> warranty from memory.
>
> -Shaun
>
> On 25/04/2021 3:26 pm, Benjamin wrote:
>> It's not worth getting anything other than cheapest non-SMR drives
>> IMO for nearly any use case... you can get performance by aggregating
>> enough drives anyways
>>
>> On Sun, Apr 25, 2021 at 3:25 PM Benjamin <zorlin at gmail.com
>> <mailto:zorlin at gmail.com>> wrote:
>>
>> LizardFS is a bag of hurt with dead development. Proceed with
>> hella caution if you go that route. I hope it changes and becomes
>> worth pursuing though.
>>
>> MFSpro is justifiable around 50TiB and up, until then it's not
>> really worth it.
>>
>> On Sun, Apr 25, 2021 at 3:22 PM William Kenworthy
>> <billk at iinet.net.au <mailto:billk at iinet.net.au>> wrote:
>>
>> Thanks Ben and Paul - this backs up my readings/experience.
>>
>> I will shortly need a new archive drive because I have lest
>> than 80Gb left on the 2Tb WD green I have been using for a
>> few years. As performance isn't an issue I will likely go
>> with a Seagate Barracuda this time (still debating shingled
>> or not because this use is more cost sensitive than
>> performance on writing new data across a network - so low
>> priority, busy, but not excessively so when in use - I am
>> happy to allow time for the shingling resilvering to complete
>> as long as it doesn't impact time to actually backup the data
>> too much.)
>>
>> Moosefs is more difficult to quantify whats needed - currently:
>>
>> 8 hosts (8 HDD, 1x M2.SSD, 6x arm32, 1x arm64 and 1x intel -
>> all odroid using gentoo)
>>
>> ~21Tb space, 3/4 in use. I could delete some as there is
>> duplicate data stored so if I lose a drive I can reclaim
>> space easily as well as decrease the goal in some places.
>>
>> As well, I am using storage classes. High use data has
>> mostly 1 chunk on the intel/SSD for performance and others on
>> HDD's. I have sc's ranging from 1 to 4 copies with 2, 3 and
>> 4 in common use ... for example things like VM's where there
>> are hot spots with temp file creation I have 2 copies (2SH)
>> whereas backups and user data have 4 copies 4HHHH or 4SHHH
>> depending on priority (eg, /home). Currently I have one WD
>> Green drive I would already toss if in a commercial system,
>> and two Seagate NAS drives I am not totally happy with.
>>
>> For these, definitely non-shingled (CMR) 7200rpm around 4TB
>> seems ideal - but is a NAS optimised drive useful or a waste
>> for moosefs? - vibration of nearby drives is the only thing I
>> can think of. Some are bound together (5x odroid HC2) and
>> some are in pairs in relatively heavy PC case baymounts
>> (removed/pinched - from my sons ongoing gaming PC build :)
>> placed on a desk. I am staring to lean towards the WD blacks
>> for this, but the HGST lines WD are starting to integrate are
>> interesting though more expensive ...
>>
>> I would love to have MFSpro but cant justify it as super
>> uptime isn't necessary, EC isn't really attractive at my
>> scale and multiple masters isn't essential as I have plenty
>> of alternative systems I could bring in quickly ... though I
>> am watching lizardfs and just might jump to it to get the
>> multiple masters that is in the free tier.
>>
>> BillK
>>
>>
>> On 25/4/21 1:19 pm, Benjamin wrote:
>>> +1 to all of it, cheers Paul.
>>>
>>> I think it's worth going for the cheapest externals you can
>>> get, shucking them, then using MooseFS since you're already
>>> planning to.
>>>
>>> I'd use copies=3 and if you're storing more than 50TB talk
>>> to me about mfspro.
>>>
>>> On Sun, 25 Apr 2021, 13:03 Paul Del, <p at delfante.it
>>> <mailto:p at delfante.it>> wrote:
>>>
>>> Hello Bill
>>>
>>> My 2 cents worth
>>>
>>> I am sure you know the common things that can increase
>>> your hard drives life and performance:
>>> Temperature
>>> Humidity
>>> VIbration
>>> Heavy Writes
>>> Heaving Logging
>>> Clean/Reliable power
>>> Data throughput
>>>
>>> The rust hard drives I have seen the most failures with
>>> are: (I recommend avoiding)
>>> WD Green
>>> WD Blue
>>> Hitachi Deskstar
>>> (Not The server drives)
>>>
>>> The rust hard drives I recommend the most are:
>>> WD Black 7200rpm or better
>>> Seagate 7200pm or better
>>> (Not Red, Blue, Green, Purple)
>>>
>>> If you are doing the moose distribute setup
>>> You could always choose two different brands/types
>>>
>>> if you want to know more specific things about which
>>> hard drive failures. Check out this from backblaze, I am
>>> sure there's more around. Which is one Benjamin sent
>>> around ages ago.
>>> https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/
>>> <https://www.backblaze.com/blog/backblaze-hard-drive-stats-for-2020/>
>>> https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/
>>> <https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/>
>>>
>>> Thanks Paul
>>>
>>>
>>> On Sat, 24 Apr 2021, 09:02 William Kenworthy,
>>> <billk at iinet.net.au <mailto:billk at iinet.net.au>> wrote:
>>>
>>> > Just musing on what changes I could make to streamline
>>> my systems:
>>> >
>>> > After a recent stray "r m - r f " with a space in it
>>> I ended up
>>> > removing both most of my active data files, VM's etc
>>> ... and the online
>>> > backups - ouch!
>>> >
>>> > I have restored from offline backups and have noticed
>>> a ~10years old WD
>>> > green drive showing a few early symptoms of failing
>>> (SMART).
>>> >
>>> > With the plethora of colours now available (!) now
>>> what drive is best for
>>> > a:
>>> >
>>> > 1. moosefs chunkserver (stores files for VM's,
>>> data including the
>>> > mail servers user files, home directories and of
>>> course the online
>>> > borgbackup archives - the disks are basically hammered
>>> all the time.)
>>> >
>>> > 2. offline backups (~2tb data using borgbackup to
>>> backup the online
>>> > borgbackup repo, used twice a week for a few minutes
>>> at a time.)
>>> >
>>> > My longest serving drives are WD greens 2Tb which
>>> until now have just
>>> > keep ticking along. The failing drive is a WD Green -
>>> I have run
>>> > badblocks on it overnight with no errors so far so it
>>> might have
>>> > internally remapped the failed sectors ok - I am using
>>> xfs which does
>>> > not have badblock support. Most drives spent previous
>>> years in btrfs
>>> > raid 10's or ceph so they have had a hard life!
>>> >
>>> > Newer WD Reds and a Red pro have failed over the years
>>> but I still have
>>> > two in the mix (6tb and 2tb)
>>> >
>>> > Some Seagate Ironwolfs that show some SMART errors
>>> Backblaze correlate
>>> > with drive failure and throw an occasional USB
>>> interface error but
>>> > otherwise seem OK.
>>> >
>>> > There are shingled, non-shingled drives, surveillance,
>>> NAS flavours etc.
>>> > - but what have people had success with? - or should I
>>> just choose my
>>> > favourite colour and run with it?
>>> >
>>> > Thoughts?
>>> >
>>> > BillK
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au
>>> <mailto:plug at plug.org.au>
>>> http://lists.plug.org.au/mailman/listinfo/plug
>>> <http://lists.plug.org.au/mailman/listinfo/plug>
>>> Committee e-mail: committee at plug.org.au
>>> <mailto:committee at plug.org.au>
>>> PLUG Membership: http://www.plug.org.au/membership
>>> <http://www.plug.org.au/membership>
>>>
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>>> http://lists.plug.org.au/mailman/listinfo/plug <http://lists.plug.org.au/mailman/listinfo/plug>
>>> Committee e-mail: committee at plug.org.au <mailto:committee at plug.org.au>
>>> PLUG Membership: http://www.plug.org.au/membership <http://www.plug.org.au/membership>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>> http://lists.plug.org.au/mailman/listinfo/plug
>> <http://lists.plug.org.au/mailman/listinfo/plug>
>> Committee e-mail: committee at plug.org.au
>> <mailto:committee at plug.org.au>
>> PLUG Membership: http://www.plug.org.au/membership
>> <http://www.plug.org.au/membership>
>>
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20210425/1d6dd6bd/attachment.html>
More information about the plug
mailing list