[plug] Filesystems for lots of inodes
Bill Kenworthy
billk at iinet.net.au
Tue Jan 7 09:30:13 AWST 2020
Hi Stephen,
I have 4 HC2's with another four disks on an old mini itx and a
further two added to the HC2's via USB3 (10 disks of various sizes
overall). Chunked systems like ceph and moosefs work with multiple
cpu's/disks as their strength is scattering copies of the chunks across
the different storage areas minimising risk. You can simulate it with
VM's but performance (and risk to data) is "not good". A lightly loaded
moosefs media server could in theory run on a single HC2 (with no
redundancy - basicly a waste!) - ceph has a much higher minimum
requirement in number of management machines and chnunkservers and
networking before you can actually use it. Its also a lot more
complex/steeper learning curve so I would strongly suggest starting with
moosefs to gain experience then move to ceph if that's the way you see
forward.
Also worth noting if you are thinking multiple usb2/3 drives: a single
USB3 runs fine on any of the odroid variants I have (H2, HC2 and n2) -
two usb2 or usb3 drives and moosefs=disaster! (the odroids (H2 and n2)
use a single internal hub for the usb connections creating a choke point
for data flow that generates a lot of errors. The most valuable data is
using goal 5 so I can lose a number of disks simultaneously without
losing data if the array is in balance - simulations (shutting down
cpu's/drives at random) bear this out.
My use case is the drives are getting old, being repurposed from the two
BTRFS raid10 arrays I was running. After surviving a bcache ssd failure
then a backing store failure (WD Red drive) in quick succession I
decided moosefs might give me a better chance of the data surviving when
I start the inevitable old disks dying dance that I know is coming.
BillK
'
On 7/1/20 8:35 am, Stephen Price wrote:
> Hi all,
>
> I've been lurking here for a while and lately have gotten myself an
> Odroid HC2 (https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/)
> I'm primarily a Windows user (and .Net Developer) so my Linux exposure
> is limited (but I do love it. Still trying to hit that critical mass
> in my knowledge where I don't feel like a gumbie)
>
> I was a Server support guy a long time ago (again, Windows) and this
> thread has me interested (given my HC2 purchase for tinkering) and so
> I looked up Ceph to see if it would be a good fit for tinkering
> with/learning. I had a look at the Ceph docs and the Installation page
> launches right into the installation but It seems to start well past
> the OS. So I'm assuming this can be installed on whatever linux distro
> you like/have?
>
> Just wanted to drop a reply and thanks for the interesting thread.
> Hope no one minds if I post questions if I get stuck with anything. I
> usually find my way eventually, its a useful IT skill. Love having a
> job where you get paid to "try shit until it works".
>
> cheers
> Stephen
>
> On Mon, Jan 6, 2020 at 9:30 PM Gregory Orange <home at oranges.id.au
> <mailto:home at oranges.id.au>> wrote:
>
> Yep, Ceph certainly has an enterprisey scent about it. We have
> dozens of nodes, and we have just moved from 2x10Gb to 2x100Gb for
> the storage network, although we're sticking with Gb for management.
>
> Choosing the right horse for the right course is part of the
> wisdom required to deal with IT in a functional way, at any scale.
>
>
> --
> Gregory Orange
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, 6 January 2020 5:49 PM, Bill Kenworthy
> <billk at iinet.net.au <mailto:billk at iinet.net.au>> wrote:
>
> > On 6/1/20 11:41 am, Gregory Orange wrote:
> >
> > > We've been using CephFS for over a year for our pilot sync
> system. It
> > > also seems to refuse to lose data, although it's hardly busy
> with any
> > > sort of production load.
> > > -- Gregory Orange
> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > > On Sunday, 5 January 2020 8:31 PM, Chris Hoy Poy
> chris at hoypoy.id.au <mailto:chris at hoypoy.id.au>
> > > wrote:
> > >
> > > > You would end up running xfs on top of ceph anyway :-)
> (unless you
> > > > don't care about your data, then you could give cephfs a try !)
> > > > /Chris
> > > > On Sun, 5 Jan 2020, 8:29 pm Gregory Orange,
> <home at oranges.id.au <mailto:home at oranges.id.au>
> > > > mailto:home at oranges.id.au <mailto:home at oranges.id.au>> wrote:
> > > >
> > > > I suppose I should mention Ceph since we're talking about
> > > > resilient storage systems, but it's likely out of scope
> here.
> > > > Bare minimum of three physical nodes, scales up real
> big. Refuses
> > > > to lose data, despite our valiant attempts over the past
> three
> > > > years. Mimic version is probably better suited to production
> > > > loads than Nautilus given our recent experiences. It's
> an object
> > > > store, so if you want file, that's at least one more
> layer on top.
> > > >
> > > >
> > > > -- Gregory Orange
> > > >
> > > > Sent from phone
> > > >
> > > >
> > > >
> > > > -------- Original Message --------
> > > > On 4 Jan 2020, 13:20, Brad Campbell <
> brad at fnarfbargle.com <mailto:brad at fnarfbargle.com>
> > > > <mailto:brad at fnarfbargle.com
> <mailto:brad at fnarfbargle.com>>> wrote:
> > > >
> > > >
> > > > On 4/1/20 1:01 pm, Bill Kenworthy wrote:
> > > >
> > > >
> > > > > Hi Brad,
> > > > >
> > > > > I have had a lot of pain from ext4 over the
> years and
> > > > have really
> > > > > only started using it again seriously recently ...
> and I
> > > > must admit, its
> > > > > a lot better than it was but I will move off it
> when I get
> > > > time - been
> > > > > burnt by it too often.
> > > > >
> > > > > reiserfs3 was my goto for inode problems in the
> past (its
> > > > still there,
> > > > > and I think maintained) but I moved to btrfs after
> the Hans
> > > > Reiser saga
> > > > > and while it has its ups and downs, stability under
> > > > punishment that
> > > > > kills ext3/4 with live scrub and snapshots made it
> great.
> > > > >
> > > > > Currently I am moving to moosefs on xfs and am
> impressed -
> > > > particularly
> > > > > with xfs so far. Live power off, various failure
> tests etc.
> > > > and I have
> > > > > not lost any data.
> > > > >
> > > > > For backup I use moosefs snapshots and borgbackup
> (main
> > > > repository is
> > > > > also on moosefs - daily + some data is 10
> minutely, as well
> > > > as an
> > > > > offline borgbackup on btrfs removable drive, this
> once a
> > > > week or so) as
> > > > > the backup software. I previously used dirvish
> for many
> > > > years though it
> > > > > had a tendency to eat ext4 file systems, it was
> great on
> > > > reiserfs and
> > > > > btrfs.
> > > > >
> > > > > Hope this helps with ideas.
> > > >
> > > >
> > > > G'day Bill,
> > > >
> > > >
> > > > It does. Thanks. Interesting how peoples experiences
> differ.
> > > > I've always
> > > > used ext[234], abused them severely and never lost a
> byte.
> > > >
> > > >
> > > >
> > > > My only foray into an alternative filesystem was
> helping a
> > > > mate with a
> > > > large btrfs layout, but after it "ran out of space"
> and ate
> > > > about 13T of
> > > > his data, and the response from the developers was
> "yeah, it
> > > > can do
> > > > that" we never looked at it again. A bit like bcache, it
> > > > always seemed
> > > > to be "almost there as long as you only use it in
> certain
> > > > circumstances
> > > > that never expose the corner cases".
> > > >
> > > >
> > > >
> > > > I'll have a serious play with xfs and see how it
> performs. I
> > > > know all
> > > > the little NAS WD Mybooks I've bought over the years
> have all
> > > > had xfs as
> > > > their main storage pool, but I've always converted
> them to
> > > > ext[234].
> > > >
> > > >
> > > >
> > > > I'll add moosefs and borgbackup to my long list of
> "must take
> > > > a look at
> > > > that one day".
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Brad
> > > > --
> > > > An expert is a person who has found out by his own
> painful
> > > > experience all the mistakes that one can make in a very
> > > > narrow field. - Niels Bohr
> > > > _______________________________________________
> > > > PLUG discussion list: plug at plug.org.au
> <mailto:plug at plug.org.au> <mailto:plug at plug.org.au
> <mailto:plug at plug.org.au>>
> > > > http://lists.plug.org.au/mailman/listinfo/plug
> > > > Committee e-mail: committee at plug.org.au
> <mailto:committee at plug.org.au>
> > > > <mailto:committee at plug.org.au
> <mailto:committee at plug.org.au>>
> > > > PLUG Membership: http://www.plug.org.au/membership
> > > >
> > > >
> > > > _______________________________________________
> > > > PLUG discussion list: plug at plug.org.au
> <mailto:plug at plug.org.au> <mailto:plug at plug.org.au
> <mailto:plug at plug.org.au>>
> > > > http://lists.plug.org.au/mailman/listinfo/plug
> > > > Committee e-mail: committee at plug.org.au
> <mailto:committee at plug.org.au>
> > > > <mailto:committee at plug.org.au
> <mailto:committee at plug.org.au>>
> > > > PLUG Membership: http://www.plug.org.au/membership
> > > >
> > >
> > > PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
> > > http://lists.plug.org.au/mailman/listinfo/plug
> > > Committee e-mail: committee at plug.org.au
> <mailto:committee at plug.org.au>
> > > PLUG Membership: http://www.plug.org.au/membership
> >
> > I did try ceph (admittedly a few years ago) and gave it up as a
> bad joke
> > after losing the entire system numerous times as soon as I tried to
> > stress it. Check google, its still happening!
> >
> > I did have to go down the path of getting more ram/split
> networks etc
> > with moosefs to get the performance I wanted, but it is stable.
> Ceph
> > requires this upfront as a minimum (despite what the docs were
> saying)
> >
> > Ceph does have a better design in that chunk placement works on a
> > formula (and vm images can use RDB) while moosefs keeps their
> location
> > in memory (proportional to the number of files stored - turns
> out be a
> > lot of memory!) - this does not scale as well, but with ceph its
> > redundant if the system crashes and isn't recoverable.
> >
> > BillK
> >
> > The problem is resources - moosfs can work if lightly loaded but
> ceph
> > requires full up front investment of at least two high speed
> networks,
> > and reasonably powerful servers.
> >
> > PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
> > http://lists.plug.org.au/mailman/listinfo/plug
> > Committee e-mail: committee at plug.org.au
> <mailto:committee at plug.org.au>
> > PLUG Membership: http://www.plug.org.au/membership
>
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au <mailto:committee at plug.org.au>
> PLUG Membership: http://www.plug.org.au/membership
>
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
More information about the plug
mailing list