[plug] gigabit switch recommendations

[Patrick Coleman]

Hey Paul,

It's been a while since I configured this, but I wanted to elaborate
on what Brad was saying before you go out and spend any money -

On Mon, Nov 17, 2014 at 10:55 AM, Paul Del <p at delfante.it> wrote:
> I kept it pretty simple because as long as it's faster than a single port I
> would be happy
> If I can get the speed of 2x gigabit ports that would be excellent

You won't get this with balance-rr and a cheap switch. balance-rr != LACP:

- balance-rr just blats _outgoing_ packets from the server round-robin
over the bonded link; LACP is not used. For 2x1Gb/s links you will
achieve 2Gb/s _outbound_; the switch will be terribly confused by this
as it will see the same MAC on several interfaces, and cheaper models
will probably just dump all traffic destined for the server down one
of the bonded links only. Even cheaper switches (and some expensive
ones) will get really confused and you'll start losing packets.

On a decent managed switch you should be able to configure some sort
of algorithm based on a hash of the source/destination and TCP ports
so (for traffic towards the server) each _TCP session_ is switched
down a different bonded link.

You will need to configure this manually on both the server and (if
you require inbound traffic balancing) the switch. The key advantage
to balance-rr however is that the switch doesn't have to be involved;
if you only require outbound traffic balancing it sometimes works well
even with unmanaged switches.

If a link fails and one side doesn't notice - the classic case being a
fibre media converter between the two ends - traffic will continue to
be blatted out that interface and every second outbound packet will be
lost.

This gets you 2Gb/s outbound from the server, and at most 1Gb/s
inbound for a given TCP session (eg. file download from a NAS, but
you'll be able to run 2x 1Gb/s downloads simultaneously).

- LACP is the Link Aggregation Control Protocol, key word here being
protocol. The idea here is that you put the Linux bonding driver into
802.3ad mode[1] and then it sends discovery packets down the bonded
interfaces. The managed switch on the other end detects this, and if
LACP is enabled an aggregate link will be negotiated between the
server and the switch. You don't need to manually specify the
interfaces on the switch end, and if one link fails both sides will
agree to not transmit data down that link. This is convenient if
you're cleaning up the server room and need to reroute cables (I kid,
I kid, that's what active/backup application failover was invented
for).

The algorithm on both ends is generally a hash on the
source/destination and TCP ports as above, so in both directions you
will get 1Gb/s for any given TCP session, but if you have several TCP
sessions they will in most cases be balanced across both links.

So, the key point here is that the only thing that will get you 2Gb/s
in every case is a 10Gb/s network adapter :) LACP, balance-rr etc were
designed for a server with several hundred clients - with a few
hundred un-natted TCP connections LACP will (generally) balance
traffic nicely across a pair of 1Gb/s links in both directions (lots
of qualifiers in that sentence ;).

I wasn't kidding on the 10Gb/s adapter front either - a pair of PCIe
adapters (one for your desktop, one for the NAS) + transceivers + a
fibre lead may well come in under $500 if you shop around[2].

Cheers,

Patrick


1. https://www.kernel.org/doc/Documentation/networking/bonding.txt
2. http://www.ebay.com/bhp/10gb-network-card