[plug] gigabit switch recommendations

[Brad Campbell]

On 17/11/14 20:02, Patrick Coleman wrote:
> Hey Paul,
>
> It's been a while since I configured this, but I wanted to elaborate
> on what Brad was saying before you go out and spend any money -
>
> On Mon, Nov 17, 2014 at 10:55 AM, Paul Del <p at delfante.it> wrote:
>> I kept it pretty simple because as long as it's faster than a single port I
>> would be happy
>> If I can get the speed of 2x gigabit ports that would be excellent
>
> You won't get this with balance-rr and a cheap switch. balance-rr != LACP:
>
> - balance-rr just blats _outgoing_ packets from the server round-robin
> over the bonded link; LACP is not used. For 2x1Gb/s links you will
> achieve 2Gb/s _outbound_; the switch will be terribly confused by this
> as it will see the same MAC on several interfaces, and cheaper models
> will probably just dump all traffic destined for the server down one
> of the bonded links only. Even cheaper switches (and some expensive
> ones) will get really confused and you'll start losing packets.
>

The _only_ way I've been able to make this work through a switch is to 
break the switch into 2 vlans. Each port on all bonded machines goes to 
a separate vlan, so you effectively switch each port separately. It 
works, but it's ugly.

Otherwise, what Patrick said stands. If you plug bonded machines into a 
switch the machine will blast packets down both interfaces, but the 
switch will aggregate these and jam them down the first interface it 
hits in its MAC table. As we all know, 2 into one does not fit.

Now, on top of all the switching heartache you need to seriously tweak 
the kernels to be able to make this all work and utilise the bandwidth 
available. What you have is massive potential for out of order packets, 
and lots of protocols don't like that, so you have to tweak the network 
settings in the kernel to queue and re-order packets to avoid a flurry 
of retries. Once you start putting a switch in the loop you have another 
device that will queue and re-transmit packets, you've just multiplied 
the problem significantly.

I get great results locally because at each end I'm using 3 ports from a 
4 port Intel PCIe NIC, so the latencies are predictable and tweakable. 
When I tried to use a combination of on-board and PCIe NICs the whole 
thing turned to mush. I've even tried bonding on my iMac with the 
on-board NIC and the NIC built into the Thunderbolt external monitor, 
and the extra latency that involves matters.

I got better and more predictable results using 3 ports in my server 
(bridged) and breaking the network into three physical segments. Each of 
my two high bandwidth consumers effectively gets its own pipe, and then 
one more for all the other stuff (printers, CCTV, WAP, backup NAS).