[plug] Bandwidth limiting

[Craig Ringer]

Mark O'Shea wrote:

> Well it's a little pointless throttling input on the endpoint.  The
> packets are still being sent to you at the same rate, your machine has
> received them and started to process them, so what's the gain?

Quite a lot - it's the only control you have. What you do is drop 
packets as the data rate exceeds a certain threshold. This causes the 
TCP/IP send algorithm at the sending end to throttle back sending (as it 
will do whenever packets start getting dropped). Because TCP/IP doesn't 
have any side-channels for bandwidth negotiation etc, this is the only 
way you can communicate with the remote host to ask them to slow down.

So, while you loose the odd packet (just like you will in any normal 
TCP/IP connection), you gain control and often a smoother traffic flow.

It would be very nice if my ISP would give me administration rights on 
the router at the other end of my ADSL link, permitting me to impose a 
more ideal solution. As I don't see this happening, I'm very happy to 
make use of the control provided by QoS under Linux.

[A quick explanation, if it's useful, follows. I'm no expert in this, so 
be aware that this is just my own understanding of the situation.]

One thing that helps make this more effective, by the way, is to cap the 
total incoming and outgoing data rates at 95% of the maximum supported 
by your line. It sounds crazy, but this can make your 'net connection 
faster in one measure - one that matters a lot. What you're doing is 
sacrificing throughput for control over latency, something that matters 
a lot if you use ssh, IRC, IM, IMAP mail, games, or anything else 
interactive. Even normal web browsing and FTP can benefit. The reason 
capping your connection helps is twofold.

For one, your ISP's router will usually keep very deep packet queue so 
that stalls or gaps in upstream transmission don't interrupt 
transmission down your link. It's much the same principle as buffering 
in streming media applications. The cost of this is that a packet for 
some interactive service can sit for quite a while (relatively) in the 
queue behind packets from some bulk data transfer, resulting in long 
packet round trip delays and awful latency.

Also, the chances are that your ISP's router doesn't attempt to 
prioritise packets. It probably doesn't send interactive traffic to the 
front of the queue, for example. For one thing, it's hard for it to know 
what traffic /is/ interactive - all it has to go by is the ToS (Type of 
Service) flag in the packet, and that's not reliably set by lots of 
programs. Second, it might just not care.

If you cap your connection at about 95% of max speed, then there will be 
no deep queue at the router, because you're moving the bottleneck down 
the line to yourself. The ISP's router will almost always be able to 
forward the packet on as soon as it arrives. This, by its self, helps 
improve latency, because you can choose to maintain much shallower 
queues on your system if you prefer.

You can go further than that, though. If you control the bottleneck, you 
can do things like putting some types of traffic to the back of the 
queue, so they're only sent if nothing else more important is waiting. 
You can also make sure that certain high-priority traffic gets sent as 
soon as possible, even if there's lots of other traffic waiting. If 
you're trying to use SSH or a remote X11 application while lobbing huge 
chunks of data between networks, this is bliss. Note that you can't 
control when you receive traffic - you have to use the trick with 
dropping packets to get the other end to slow down, instead. This is 
usually done by dropping them as they're to be sent (forwarded) out of 
the "LAN" network interface on your firewall/gateway.

So ... in conclusion, I think there are a lot of good reasons to 
throttle input at the endpoint, mostly thanks to the wonderful 
flexibility of TCP/IP.

If anybody can point out a nice way to use the same QoS tools (tc, HTB, 
SFQ, etc) to control input rates directly, I'll be happy. Suggestions 
involving forwarding packets between virtual interfaces created with 
iproute2 will result in a sound (if remote) whack with a spade - QoS is 
already quite complicated enough to set up and debug.

--
Craig Ringer