[plug] Filesystems for lots of inodes
Benjamin
zorlin at gmail.com
Tue Jan 7 09:18:48 AWST 2020
Yeah, Ceph is pretty hungry. I'm also not sure it's available on any 32-bit
platforms, especially ARM. You might be able to get it working on a HC2 or
similar but no-one would want to support it.
It's worth spinning up a virtual Ceph lab if you've got oodles of RAM on a
laptop or something though.
~ B
On Tue, Jan 7, 2020 at 9:08 AM Chris Hoy Poy <chris at hoypoy.id.au> wrote:
> I would recommend Ubuntu for a pretty smooth Ceph experience. the issue is
> typically that it behaves very badly in small setups.
>
> Ceph really wants 3x storage nodes (if not 5-6, and multiple disks per
> storage node, don't raid those disks, it will do that!) and 2x management
> nodes to start with. Feel free to throw up any queries on it, I have a
> decade or something silly experience with it, and Greg as stated has been
> using it a fair while too. (We still have a bit of it in Portland but I
> don't touch on it much these days otherwise!)
>
> /Chris
>
>
> On Tue, 7 Jan 2020, 8:35 am Stephen Price, <stephen at perthprojects.com>
> wrote:
>
>> Hi all,
>>
>> I've been lurking here for a while and lately have gotten myself an
>> Odroid HC2 (https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/)
>> I'm primarily a Windows user (and .Net Developer) so my Linux exposure is
>> limited (but I do love it. Still trying to hit that critical mass in my
>> knowledge where I don't feel like a gumbie)
>>
>> I was a Server support guy a long time ago (again, Windows) and this
>> thread has me interested (given my HC2 purchase for tinkering) and so I
>> looked up Ceph to see if it would be a good fit for tinkering
>> with/learning. I had a look at the Ceph docs and the Installation page
>> launches right into the installation but It seems to start well past the
>> OS. So I'm assuming this can be installed on whatever linux distro you
>> like/have?
>>
>> Just wanted to drop a reply and thanks for the interesting thread. Hope
>> no one minds if I post questions if I get stuck with anything. I usually
>> find my way eventually, its a useful IT skill. Love having a job where you
>> get paid to "try shit until it works".
>>
>> cheers
>> Stephen
>>
>> On Mon, Jan 6, 2020 at 9:30 PM Gregory Orange <home at oranges.id.au> wrote:
>>
>>> Yep, Ceph certainly has an enterprisey scent about it. We have dozens of
>>> nodes, and we have just moved from 2x10Gb to 2x100Gb for the storage
>>> network, although we're sticking with Gb for management.
>>>
>>> Choosing the right horse for the right course is part of the wisdom
>>> required to deal with IT in a functional way, at any scale.
>>>
>>>
>>> --
>>> Gregory Orange
>>>
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Monday, 6 January 2020 5:49 PM, Bill Kenworthy <billk at iinet.net.au>
>>> wrote:
>>>
>>> > On 6/1/20 11:41 am, Gregory Orange wrote:
>>> >
>>> > > We've been using CephFS for over a year for our pilot sync system. It
>>> > > also seems to refuse to lose data, although it's hardly busy with any
>>> > > sort of production load.
>>> > > -- Gregory Orange
>>> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> > > On Sunday, 5 January 2020 8:31 PM, Chris Hoy Poy chris at hoypoy.id.au
>>> > > wrote:
>>> > >
>>> > > > You would end up running xfs on top of ceph anyway :-) (unless you
>>> > > > don't care about your data, then you could give cephfs a try !)
>>> > > > /Chris
>>> > > > On Sun, 5 Jan 2020, 8:29 pm Gregory Orange, <home at oranges.id.au
>>> > > > mailto:home at oranges.id.au> wrote:
>>> > > >
>>> > > > I suppose I should mention Ceph since we're talking about
>>> > > > resilient storage systems, but it's likely out of scope here.
>>> > > > Bare minimum of three physical nodes, scales up real big.
>>> Refuses
>>> > > > to lose data, despite our valiant attempts over the past three
>>> > > > years. Mimic version is probably better suited to production
>>> > > > loads than Nautilus given our recent experiences. It's an
>>> object
>>> > > > store, so if you want file, that's at least one more layer on
>>> top.
>>> > > >
>>> > > >
>>> > > > -- Gregory Orange
>>> > > >
>>> > > > Sent from phone
>>> > > >
>>> > > >
>>> > > >
>>> > > > -------- Original Message --------
>>> > > > On 4 Jan 2020, 13:20, Brad Campbell < brad at fnarfbargle.com
>>> > > > <mailto:brad at fnarfbargle.com>> wrote:
>>> > > >
>>> > > >
>>> > > > On 4/1/20 1:01 pm, Bill Kenworthy wrote:
>>> > > >
>>> > > >
>>> > > > > Hi Brad,
>>> > > > >
>>> > > > > I have had a lot of pain from ext4 over the years and
>>> > > > have really
>>> > > > > only started using it again seriously recently ... and I
>>> > > > must admit, its
>>> > > > > a lot better than it was but I will move off it when I
>>> get
>>> > > > time - been
>>> > > > > burnt by it too often.
>>> > > > >
>>> > > > > reiserfs3 was my goto for inode problems in the past (its
>>> > > > still there,
>>> > > > > and I think maintained) but I moved to btrfs after the
>>> Hans
>>> > > > Reiser saga
>>> > > > > and while it has its ups and downs, stability under
>>> > > > punishment that
>>> > > > > kills ext3/4 with live scrub and snapshots made it great.
>>> > > > >
>>> > > > > Currently I am moving to moosefs on xfs and am impressed
>>> -
>>> > > > particularly
>>> > > > > with xfs so far. Live power off, various failure tests
>>> etc.
>>> > > > and I have
>>> > > > > not lost any data.
>>> > > > >
>>> > > > > For backup I use moosefs snapshots and borgbackup (main
>>> > > > repository is
>>> > > > > also on moosefs - daily + some data is 10 minutely, as
>>> well
>>> > > > as an
>>> > > > > offline borgbackup on btrfs removable drive, this once a
>>> > > > week or so) as
>>> > > > > the backup software. I previously used dirvish for many
>>> > > > years though it
>>> > > > > had a tendency to eat ext4 file systems, it was great on
>>> > > > reiserfs and
>>> > > > > btrfs.
>>> > > > >
>>> > > > > Hope this helps with ideas.
>>> > > >
>>> > > >
>>> > > > G'day Bill,
>>> > > >
>>> > > >
>>> > > > It does. Thanks. Interesting how peoples experiences
>>> differ.
>>> > > > I've always
>>> > > > used ext[234], abused them severely and never lost a byte.
>>> > > >
>>> > > >
>>> > > >
>>> > > > My only foray into an alternative filesystem was helping a
>>> > > > mate with a
>>> > > > large btrfs layout, but after it "ran out of space" and ate
>>> > > > about 13T of
>>> > > > his data, and the response from the developers was "yeah,
>>> it
>>> > > > can do
>>> > > > that" we never looked at it again. A bit like bcache, it
>>> > > > always seemed
>>> > > > to be "almost there as long as you only use it in certain
>>> > > > circumstances
>>> > > > that never expose the corner cases".
>>> > > >
>>> > > >
>>> > > >
>>> > > > I'll have a serious play with xfs and see how it performs.
>>> I
>>> > > > know all
>>> > > > the little NAS WD Mybooks I've bought over the years have
>>> all
>>> > > > had xfs as
>>> > > > their main storage pool, but I've always converted them to
>>> > > > ext[234].
>>> > > >
>>> > > >
>>> > > >
>>> > > > I'll add moosefs and borgbackup to my long list of "must
>>> take
>>> > > > a look at
>>> > > > that one day".
>>> > > >
>>> > > >
>>> > > >
>>> > > > Regards,
>>> > > > Brad
>>> > > > --
>>> > > > An expert is a person who has found out by his own painful
>>> > > > experience all the mistakes that one can make in a very
>>> > > > narrow field. - Niels Bohr
>>> > > > _______________________________________________
>>> > > > PLUG discussion list: plug at plug.org.au <mailto:
>>> plug at plug.org.au>
>>> > > > http://lists.plug.org.au/mailman/listinfo/plug
>>> > > > Committee e-mail: committee at plug.org.au
>>> > > > <mailto:committee at plug.org.au>
>>> > > > PLUG Membership: http://www.plug.org.au/membership
>>> > > >
>>> > > >
>>> > > > _______________________________________________
>>> > > > PLUG discussion list: plug at plug.org.au <mailto:
>>> plug at plug.org.au>
>>> > > > http://lists.plug.org.au/mailman/listinfo/plug
>>> > > > Committee e-mail: committee at plug.org.au
>>> > > > <mailto:committee at plug.org.au>
>>> > > > PLUG Membership: http://www.plug.org.au/membership
>>> > > >
>>> > >
>>> > > PLUG discussion list: plug at plug.org.au
>>> > > http://lists.plug.org.au/mailman/listinfo/plug
>>> > > Committee e-mail: committee at plug.org.au
>>> > > PLUG Membership: http://www.plug.org.au/membership
>>> >
>>> > I did try ceph (admittedly a few years ago) and gave it up as a bad
>>> joke
>>> > after losing the entire system numerous times as soon as I tried to
>>> > stress it. Check google, its still happening!
>>> >
>>> > I did have to go down the path of getting more ram/split networks etc
>>> > with moosefs to get the performance I wanted, but it is stable. Ceph
>>> > requires this upfront as a minimum (despite what the docs were saying)
>>> >
>>> > Ceph does have a better design in that chunk placement works on a
>>> > formula (and vm images can use RDB) while moosefs keeps their location
>>> > in memory (proportional to the number of files stored - turns out be a
>>> > lot of memory!) - this does not scale as well, but with ceph its
>>> > redundant if the system crashes and isn't recoverable.
>>> >
>>> > BillK
>>> >
>>> > The problem is resources - moosfs can work if lightly loaded but ceph
>>> > requires full up front investment of at least two high speed networks,
>>> > and reasonably powerful servers.
>>> >
>>> > PLUG discussion list: plug at plug.org.au
>>> > http://lists.plug.org.au/mailman/listinfo/plug
>>> > Committee e-mail: committee at plug.org.au
>>> > PLUG Membership: http://www.plug.org.au/membership
>>>
>>>
>>> _______________________________________________
>>> PLUG discussion list: plug at plug.org.au
>>> http://lists.plug.org.au/mailman/listinfo/plug
>>> Committee e-mail: committee at plug.org.au
>>> PLUG Membership: http://www.plug.org.au/membership
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20200107/b4df73c3/attachment.html>
More information about the plug
mailing list