[plug] Filesystems for lots of inodes
Chris Hoy Poy
chris at hoypoy.id.au
Tue Jan 7 09:08:22 AWST 2020
I would recommend Ubuntu for a pretty smooth Ceph experience. the issue is
typically that it behaves very badly in small setups.
Ceph really wants 3x storage nodes (if not 5-6, and multiple disks per
storage node, don't raid those disks, it will do that!) and 2x management
nodes to start with. Feel free to throw up any queries on it, I have a
decade or something silly experience with it, and Greg as stated has been
using it a fair while too. (We still have a bit of it in Portland but I
don't touch on it much these days otherwise!)
/Chris
On Tue, 7 Jan 2020, 8:35 am Stephen Price, <stephen at perthprojects.com>
wrote:
> Hi all,
>
> I've been lurking here for a while and lately have gotten myself an Odroid
> HC2 (https://www.hardkernel.com/shop/odroid-hc2-home-cloud-two/)
> I'm primarily a Windows user (and .Net Developer) so my Linux exposure is
> limited (but I do love it. Still trying to hit that critical mass in my
> knowledge where I don't feel like a gumbie)
>
> I was a Server support guy a long time ago (again, Windows) and this
> thread has me interested (given my HC2 purchase for tinkering) and so I
> looked up Ceph to see if it would be a good fit for tinkering
> with/learning. I had a look at the Ceph docs and the Installation page
> launches right into the installation but It seems to start well past the
> OS. So I'm assuming this can be installed on whatever linux distro you
> like/have?
>
> Just wanted to drop a reply and thanks for the interesting thread. Hope no
> one minds if I post questions if I get stuck with anything. I usually find
> my way eventually, its a useful IT skill. Love having a job where you get
> paid to "try shit until it works".
>
> cheers
> Stephen
>
> On Mon, Jan 6, 2020 at 9:30 PM Gregory Orange <home at oranges.id.au> wrote:
>
>> Yep, Ceph certainly has an enterprisey scent about it. We have dozens of
>> nodes, and we have just moved from 2x10Gb to 2x100Gb for the storage
>> network, although we're sticking with Gb for management.
>>
>> Choosing the right horse for the right course is part of the wisdom
>> required to deal with IT in a functional way, at any scale.
>>
>>
>> --
>> Gregory Orange
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Monday, 6 January 2020 5:49 PM, Bill Kenworthy <billk at iinet.net.au>
>> wrote:
>>
>> > On 6/1/20 11:41 am, Gregory Orange wrote:
>> >
>> > > We've been using CephFS for over a year for our pilot sync system. It
>> > > also seems to refuse to lose data, although it's hardly busy with any
>> > > sort of production load.
>> > > -- Gregory Orange
>> > > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> > > On Sunday, 5 January 2020 8:31 PM, Chris Hoy Poy chris at hoypoy.id.au
>> > > wrote:
>> > >
>> > > > You would end up running xfs on top of ceph anyway :-) (unless you
>> > > > don't care about your data, then you could give cephfs a try !)
>> > > > /Chris
>> > > > On Sun, 5 Jan 2020, 8:29 pm Gregory Orange, <home at oranges.id.au
>> > > > mailto:home at oranges.id.au> wrote:
>> > > >
>> > > > I suppose I should mention Ceph since we're talking about
>> > > > resilient storage systems, but it's likely out of scope here.
>> > > > Bare minimum of three physical nodes, scales up real big.
>> Refuses
>> > > > to lose data, despite our valiant attempts over the past three
>> > > > years. Mimic version is probably better suited to production
>> > > > loads than Nautilus given our recent experiences. It's an object
>> > > > store, so if you want file, that's at least one more layer on
>> top.
>> > > >
>> > > >
>> > > > -- Gregory Orange
>> > > >
>> > > > Sent from phone
>> > > >
>> > > >
>> > > >
>> > > > -------- Original Message --------
>> > > > On 4 Jan 2020, 13:20, Brad Campbell < brad at fnarfbargle.com
>> > > > <mailto:brad at fnarfbargle.com>> wrote:
>> > > >
>> > > >
>> > > > On 4/1/20 1:01 pm, Bill Kenworthy wrote:
>> > > >
>> > > >
>> > > > > Hi Brad,
>> > > > >
>> > > > > I have had a lot of pain from ext4 over the years and
>> > > > have really
>> > > > > only started using it again seriously recently ... and I
>> > > > must admit, its
>> > > > > a lot better than it was but I will move off it when I get
>> > > > time - been
>> > > > > burnt by it too often.
>> > > > >
>> > > > > reiserfs3 was my goto for inode problems in the past (its
>> > > > still there,
>> > > > > and I think maintained) but I moved to btrfs after the
>> Hans
>> > > > Reiser saga
>> > > > > and while it has its ups and downs, stability under
>> > > > punishment that
>> > > > > kills ext3/4 with live scrub and snapshots made it great.
>> > > > >
>> > > > > Currently I am moving to moosefs on xfs and am impressed -
>> > > > particularly
>> > > > > with xfs so far. Live power off, various failure tests
>> etc.
>> > > > and I have
>> > > > > not lost any data.
>> > > > >
>> > > > > For backup I use moosefs snapshots and borgbackup (main
>> > > > repository is
>> > > > > also on moosefs - daily + some data is 10 minutely, as
>> well
>> > > > as an
>> > > > > offline borgbackup on btrfs removable drive, this once a
>> > > > week or so) as
>> > > > > the backup software. I previously used dirvish for many
>> > > > years though it
>> > > > > had a tendency to eat ext4 file systems, it was great on
>> > > > reiserfs and
>> > > > > btrfs.
>> > > > >
>> > > > > Hope this helps with ideas.
>> > > >
>> > > >
>> > > > G'day Bill,
>> > > >
>> > > >
>> > > > It does. Thanks. Interesting how peoples experiences differ.
>> > > > I've always
>> > > > used ext[234], abused them severely and never lost a byte.
>> > > >
>> > > >
>> > > >
>> > > > My only foray into an alternative filesystem was helping a
>> > > > mate with a
>> > > > large btrfs layout, but after it "ran out of space" and ate
>> > > > about 13T of
>> > > > his data, and the response from the developers was "yeah, it
>> > > > can do
>> > > > that" we never looked at it again. A bit like bcache, it
>> > > > always seemed
>> > > > to be "almost there as long as you only use it in certain
>> > > > circumstances
>> > > > that never expose the corner cases".
>> > > >
>> > > >
>> > > >
>> > > > I'll have a serious play with xfs and see how it performs. I
>> > > > know all
>> > > > the little NAS WD Mybooks I've bought over the years have
>> all
>> > > > had xfs as
>> > > > their main storage pool, but I've always converted them to
>> > > > ext[234].
>> > > >
>> > > >
>> > > >
>> > > > I'll add moosefs and borgbackup to my long list of "must
>> take
>> > > > a look at
>> > > > that one day".
>> > > >
>> > > >
>> > > >
>> > > > Regards,
>> > > > Brad
>> > > > --
>> > > > An expert is a person who has found out by his own painful
>> > > > experience all the mistakes that one can make in a very
>> > > > narrow field. - Niels Bohr
>> > > > _______________________________________________
>> > > > PLUG discussion list: plug at plug.org.au <mailto:
>> plug at plug.org.au>
>> > > > http://lists.plug.org.au/mailman/listinfo/plug
>> > > > Committee e-mail: committee at plug.org.au
>> > > > <mailto:committee at plug.org.au>
>> > > > PLUG Membership: http://www.plug.org.au/membership
>> > > >
>> > > >
>> > > > _______________________________________________
>> > > > PLUG discussion list: plug at plug.org.au <mailto:
>> plug at plug.org.au>
>> > > > http://lists.plug.org.au/mailman/listinfo/plug
>> > > > Committee e-mail: committee at plug.org.au
>> > > > <mailto:committee at plug.org.au>
>> > > > PLUG Membership: http://www.plug.org.au/membership
>> > > >
>> > >
>> > > PLUG discussion list: plug at plug.org.au
>> > > http://lists.plug.org.au/mailman/listinfo/plug
>> > > Committee e-mail: committee at plug.org.au
>> > > PLUG Membership: http://www.plug.org.au/membership
>> >
>> > I did try ceph (admittedly a few years ago) and gave it up as a bad joke
>> > after losing the entire system numerous times as soon as I tried to
>> > stress it. Check google, its still happening!
>> >
>> > I did have to go down the path of getting more ram/split networks etc
>> > with moosefs to get the performance I wanted, but it is stable. Ceph
>> > requires this upfront as a minimum (despite what the docs were saying)
>> >
>> > Ceph does have a better design in that chunk placement works on a
>> > formula (and vm images can use RDB) while moosefs keeps their location
>> > in memory (proportional to the number of files stored - turns out be a
>> > lot of memory!) - this does not scale as well, but with ceph its
>> > redundant if the system crashes and isn't recoverable.
>> >
>> > BillK
>> >
>> > The problem is resources - moosfs can work if lightly loaded but ceph
>> > requires full up front investment of at least two high speed networks,
>> > and reasonably powerful servers.
>> >
>> > PLUG discussion list: plug at plug.org.au
>> > http://lists.plug.org.au/mailman/listinfo/plug
>> > Committee e-mail: committee at plug.org.au
>> > PLUG Membership: http://www.plug.org.au/membership
>>
>>
>> _______________________________________________
>> PLUG discussion list: plug at plug.org.au
>> http://lists.plug.org.au/mailman/listinfo/plug
>> Committee e-mail: committee at plug.org.au
>> PLUG Membership: http://www.plug.org.au/membership
>
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.plug.org.au/pipermail/plug/attachments/20200107/655bf904/attachment.html>
More information about the plug
mailing list