[plug] Point to point fast network

[Andrew Cooks]

8TB of storage with 300MB/s sustained write throughput sounds... loud,
hot and... expensive.

Thanks for sharing.

a.

On Fri, Aug 2, 2013 at 2:10 PM, Brad Campbell <brad at fnarfbargle.com> wrote:
> G'day Guys,
>
> I've had to shuffle some data around recently between machines, and I've
> been playing with bonded Gig-E to do it. I post it here in case someone
> finds it useful or interesting.
>
> Because of an error on an Ext4 filesystem I've been unable to rectify nor
> explain, I've had to move 8TB of data from one machine to another, recreate
> the filesystem and then move it back again.
>
> In the interest of making this process as quick as possible I looked at a
> couple of local network options. The machines are sat next to each other, so
> distance is not an issue.
>
> The obvious solution was some 10G cards and a patch lead, but I was unable
> to find cards that were affordable, and could conclusively be shown to work
> well with Linux.
>
> I tried using some cheap Realtek based Gig-E cards (I had them lying
> around). I found that bonded they scaled pretty well, but I was pushing to
> get more than about 400M per card (far, far short of the theoretical
> maximum).
>
> I happened to have a Quad port Intel Gig-E adapter in one of the machines,
> so I jumped on fleabay and picked up a second card for just under $100
> shipped. When that arrived I trunked 3 of the ports and configured them in a
> balance-rr bonded connection.
>
> All conventional wisdom says I was likely to get close to the theoretical
> maximum for about 2 ports as packet re-ordering and other issues would start
> making life difficult.
>
> In reality, once I turned the mtu up to 9000, set
> /proc/sys/net/ipv4/tcp_reordering to 127 (default 3) and set the txqueuelen
> on the bond device to 10000, I found I could stream tar over netcat and
> comfortably sustain 300 megabytes/s to and from the disks.
>
> It sped the process up no end. If I can free up the 4th port on the first
> machine, I might even try 4 ports and see if I can make it go faster.
>
> It's nice if you need a big hose.
> _______________________________________________
> PLUG discussion list: plug at plug.org.au
> http://lists.plug.org.au/mailman/listinfo/plug
> Committee e-mail: committee at plug.org.au
> PLUG Membership: http://www.plug.org.au/membership