[plug] XtreemFS (was Re: Syncing live Data)

[Daniel Pittman]

William Kenworthy <billk at iinet.net.au> writes:
> On Sat, 2010-12-04 at 11:36 +1100, Daniel Pittman wrote:
>> Patrick Coleman <blinken at gmail.com> writes:
>> > On Fri, Dec 3, 2010 at 9:26 PM, Daniel Pittman <daniel at rimspace.net> wrote:
>> >>
>> >> Given that I am not aware of any cluster file systems that will work
>> >> across both Windows and Linux systems (and given this is a Linux list) I
>> >> suggested some solutions that would work between a pair of Linux
>> >> servers.
>> >
>> > I would look seriously at Windows DFS. But, this is a Linux list.
>> > Google is your friend.
>> 
>> I don't believe it would fix the OPs problem anyhow: unless Samba 4 will
>> participate in a DFS replication agreement, that will give a unified
>> namespace but not replication, which the OP stated was a requirement.
>> 
>> You *really* don't want to do CIFS over a WAN link - the protocol is hugely
>> latency sensitive, and some of the common client tools *cough* office
>> *cough* make literally thousands of random, single-byte synchronous
>> requests while they open a document.  Non-replicated DFS would make some
>> files super-painful for the clients to open.
>
> This is turning into a general discussion on file systems!

I was trying to keep it on topic (albeit "what won't work and why"), but since
you ask and I just spent a while investigating...

> xtreemfs has been mentioned - I am in the process of setting this up for
> testing and would be interested in knowing if anyone is using it
> successfully and if there are any gotchas?

Well, replication is a PITA, so you are now dependent for the operation of
your filesystem on the continuous running of all the relevant nodes.  They
promise they are improving replication support, but for now it hurts.

It is passably robust against brief outages, though, and lives up to the
promises it made on other topics.

> I am replicating an approximately 14GB home directory (actually a
> subdirectory of /home/user) using unison across 4 machines/sites with
> dirvish backup to read only copies for versioning.  Xtreemfs looks like a
> viable alternative to the unison part of keeping everything consistent but I
> am unsure how it work with a laptop that is often isolated from the network,
> and will reconnect from various sites/places.

It will not work at all while you are offline, but performance is acceptable
over the WAN.  It is not hugely latency sensitive, but doesn't do all that
much local caching.

Think of it as a version of an NFS export that can be formed from multiple
machines, but otherwise not substantially different to that in terms of
failure handling behaviour.


I don't think it will be a great solution to your problem, although there are
definitely far worse answers.  (glusterfs is not any better, FWIW.)

        Daniel
-- 
✣ Daniel Pittman            ✉ daniel at rimspace.net            ☎ +61 401 155 707
               ♽ made with 100 percent post-consumer electrons