[plug] Filesystems for lots of inodes

Bill Kenworthy billk at iinet.net.au
Mon Jan 6 17:49:03 AWST 2020


On 6/1/20 11:41 am, Gregory Orange wrote:
> We've been using CephFS for over a year for our pilot sync system. It 
> also seems to refuse to lose data, although it's hardly busy with any 
> sort of production load.
>
> -- Gregory Orange
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Sunday, 5 January 2020 8:31 PM, Chris Hoy Poy <chris at hoypoy.id.au> 
> wrote:
>
>> You would end up running xfs on top of ceph anyway :-) (unless you 
>> don't care about your data, then you could give cephfs a try !)
>>
>> /Chris
>>
>> On Sun, 5 Jan 2020, 8:29 pm Gregory Orange, <home at oranges.id.au 
>> <mailto:home at oranges.id.au>> wrote:
>>
>>     I suppose I should mention Ceph since we're talking about
>>     resilient storage systems, but it's likely out of scope here.
>>     Bare minimum of three physical nodes, scales up real big. Refuses
>>     to lose data, despite our valiant attempts over the past three
>>     years. Mimic version is probably better suited to production
>>     loads than Nautilus given our recent experiences. It's an object
>>     store, so if you want file, that's at least one more layer on top.
>>
>>
>>     -- Gregory Orange
>>
>>     Sent from phone
>>
>>
>>
>>     -------- Original Message --------
>>     On 4 Jan 2020, 13:20, Brad Campbell < brad at fnarfbargle.com
>>     <mailto:brad at fnarfbargle.com>> wrote:
>>
>>
>>         On 4/1/20 1:01 pm, Bill Kenworthy wrote:
>>
>>
>>         > Hi Brad,
>>         >
>>         >     I have had a lot of pain from ext4 over the years and
>>         have really
>>         > only started using it again seriously recently ... and I
>>         must admit, its
>>         > a lot better than it was but I will move off it when I get
>>         time - been
>>         > burnt by it too often.
>>         >
>>         > reiserfs3 was my goto for inode problems in the past (its
>>         still there,
>>         > and I think maintained) but I moved to btrfs after the Hans
>>         Reiser saga
>>         > and while it has its ups and downs, stability under
>>         punishment that
>>         > kills ext3/4 with live scrub and snapshots made it great.
>>         >
>>         > Currently I am moving to moosefs on xfs and am impressed -
>>         particularly
>>         > with xfs so far. Live power off, various failure tests etc.
>>         and I have
>>         > not lost any data.
>>         >
>>         > For backup I use moosefs snapshots and borgbackup (main
>>         repository is
>>         > also on moosefs - daily + some data is 10 minutely, as well
>>         as an
>>         > offline borgbackup on btrfs removable drive, this once a
>>         week or so) as
>>         > the backup software.  I previously used dirvish for many
>>         years though it
>>         > had a tendency to eat ext4 file systems, it was great on
>>         reiserfs and
>>         > btrfs.
>>         >
>>         > Hope this helps with ideas.
>>
>>
>>         G'day Bill,
>>
>>
>>         It does. Thanks. Interesting how peoples experiences differ.
>>         I've always
>>         used ext[234], abused them severely and never lost a byte.
>>
>>
>>
>>         My only foray into an alternative filesystem was helping a
>>         mate with a
>>         large btrfs layout, but after it "ran out of space" and ate
>>         about 13T of
>>         his data, and the response from the developers was "yeah, it
>>         can do
>>         that" we never looked at it again. A bit like bcache, it
>>         always seemed
>>         to be "almost there as long as you only use it in certain
>>         circumstances
>>         that never expose the corner cases".
>>
>>
>>
>>         I'll have a serious play with xfs and see how it performs. I
>>         know all
>>         the little NAS WD Mybooks I've bought over the years have all
>>         had xfs as
>>         their main storage pool, but I've always converted them to
>>         ext[234].
>>
>>
>>
>>         I'll add moosefs and borgbackup to my long list of "must take
>>         a look at
>>         that one day".
>>
>>
>>
>>         Regards,
>>         Brad
>>         --
>>         An expert is a person who has found out by his own painful
>>         experience all the mistakes that one can make in a very
>>         narrow field. - Niels Bohr
>>         _______________________________________________
>>         PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>>         http://lists.plug.org.au/mailman/listinfo/plug
>>         Committee e-mail: committee at plug.org.au
>>         <mailto:committee at plug.org.au>
>>         PLUG Membership: http://www.plug.org.au/membership
>>
>>
>>         _______________________________________________
>>         PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>>         http://lists.plug.org.au/mailman/listinfo/plug
>>         Committee e-mail: committee at plug.org.au
>>         <mailto:committee at plug.org.au>
>>         PLUG Membership: http://www.plug.org.au/membership
>>
>
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership


I did try ceph (admittedly a few years ago) and gave it up as a bad joke 
after losing the entire system numerous times as soon as I tried to 
stress it.  Check google, its still happening!

I did have to go down the path of getting more ram/split networks etc 
with moosefs to get the performance I wanted, but it is stable.  Ceph 
requires this upfront as a minimum (despite what the docs were saying)

Ceph does have a better design in that chunk placement works on a 
formula (and vm images can use RDB) while moosefs keeps their location 
in memory (proportional to the number of files stored - turns out be a 
lot of memory!) - this does not scale as well, but with ceph its 
redundant if the system crashes and isn't recoverable.

BillK



The problem is resources - moosfs can work if lightly loaded but ceph 
requires full up front investment of at least two high speed networks, 
and reasonably powerful servers.




More information about the plug mailing list