[plug] Point to point fast network

[Brad Campbell]

G'day Guys,

I've had to shuffle some data around recently between machines, and I've 
been playing with bonded Gig-E to do it. I post it here in case someone 
finds it useful or interesting.

Because of an error on an Ext4 filesystem I've been unable to rectify 
nor explain, I've had to move 8TB of data from one machine to another, 
recreate the filesystem and then move it back again.

In the interest of making this process as quick as possible I looked at 
a couple of local network options. The machines are sat next to each 
other, so distance is not an issue.

The obvious solution was some 10G cards and a patch lead, but I was 
unable to find cards that were affordable, and could conclusively be 
shown to work well with Linux.

I tried using some cheap Realtek based Gig-E cards (I had them lying 
around). I found that bonded they scaled pretty well, but I was pushing 
to get more than about 400M per card (far, far short of the theoretical 
maximum).

I happened to have a Quad port Intel Gig-E adapter in one of the 
machines, so I jumped on fleabay and picked up a second card for just 
under $100 shipped. When that arrived I trunked 3 of the ports and 
configured them in a balance-rr bonded connection.

All conventional wisdom says I was likely to get close to the 
theoretical maximum for about 2 ports as packet re-ordering and other 
issues would start making life difficult.

In reality, once I turned the mtu up to 9000, set 
/proc/sys/net/ipv4/tcp_reordering to 127 (default 3) and set the 
txqueuelen on the bond device to 10000, I found I could stream tar over 
netcat and comfortably sustain 300 megabytes/s to and from the disks.

It sped the process up no end. If I can free up the 4th port on the 
first machine, I might even try 4 ports and see if I can make it go faster.

It's nice if you need a big hose.