[plug] Point to point fast network

[Rusty Ramser]

Very interesting description of your solution, Brad.  Thanks for that.

(I guess just putting one of the drives directly into the other machine
wasn't technically a possibility?)

Cheers.


-----Original Message-----
From: plug [mailto:plug-bounces at plug.org.au] On Behalf Of Brad Campbell
Sent: Friday, 2 August 2013 14:11
To: plug at plug.org.au
Subject: [plug] Point to point fast network

G'day Guys,

I've had to shuffle some data around recently between machines, and I've
been playing with bonded Gig-E to do it. I post it here in case someone
finds it useful or interesting.

Because of an error on an Ext4 filesystem I've been unable to rectify nor
explain, I've had to move 8TB of data from one machine to another, recreate
the filesystem and then move it back again.

In the interest of making this process as quick as possible I looked at a
couple of local network options. The machines are sat next to each other, so
distance is not an issue.

The obvious solution was some 10G cards and a patch lead, but I was unable
to find cards that were affordable, and could conclusively be shown to work
well with Linux.

I tried using some cheap Realtek based Gig-E cards (I had them lying
around). I found that bonded they scaled pretty well, but I was pushing to
get more than about 400M per card (far, far short of the theoretical
maximum).

I happened to have a Quad port Intel Gig-E adapter in one of the machines,
so I jumped on fleabay and picked up a second card for just under $100
shipped. When that arrived I trunked 3 of the ports and configured them in a
balance-rr bonded connection.

All conventional wisdom says I was likely to get close to the theoretical
maximum for about 2 ports as packet re-ordering and other issues would start
making life difficult.

In reality, once I turned the mtu up to 9000, set
/proc/sys/net/ipv4/tcp_reordering to 127 (default 3) and set the txqueuelen
on the bond device to 10000, I found I could stream tar over netcat and
comfortably sustain 300 megabytes/s to and from the disks.

It sped the process up no end. If I can free up the 4th port on the first
machine, I might even try 4 ports and see if I can make it go faster.

It's nice if you need a big hose.
_______________________________________________
PLUG discussion list: plug at plug.org.au
http://lists.plug.org.au/mailman/listinfo/plug
Committee e-mail: committee at plug.org.au
PLUG Membership: http://www.plug.org.au/membership