Lesson 20 advanced

Congestion Control and Quality of Service

Why networks congest, bufferbloat, open-loop and closed-loop congestion control, QoS requirements of voice, video and bulk traffic, classification and DSCP marking, queuing, policing and shaping, the leaky bucket and token bucket algorithms, and shaping and testing with Linux tc.

A link can only send so many bits per second. When more traffic arrives than it can carry, packets wait in a queue, and when the queue is full they are dropped. That is congestion. Lesson 14 showed how TCP senders react to it; this lesson looks at the network side: how routers and hosts manage queues, give important traffic priority and smooth bursts, which together are called Quality of Service (QoS).

Why congestion happens

  • Speed mismatch: a 10 Gbit/s LAN feeding a 500 Mbit/s internet link.
  • Fan-in: many senders converge on one receiver or uplink (common in data centers: “incast”).
  • Bursts: traffic arrives in clumps even when the average is low.
  • Slow devices: a router’s CPU or buffers cannot keep up.

Symptoms: latency rises first (packets wait in queues), then loss starts (queues overflow), then throughput collapses if senders keep retransmitting.

Bufferbloat

You might think bigger buffers avoid drops, so they must be better. They are not. A queue that holds seconds of traffic turns a busy link into a slow link: every packet, including your video call and DNS lookups, waits behind a huge upload. That is bufferbloat, and it is why a home connection becomes unusable while someone uploads a video.

Test it by measuring latency while the link is busy:

$ ping -c 5 1.1.1.1 | tail -1
rtt min/avg/max/mdev = 11.2/11.6/12.3/0.4 ms
$ # now start a big upload in another terminal and ping again
$ ping -c 5 1.1.1.1 | tail -1
rtt min/avg/max/mdev = 380.1/612.4/901.7/190.2 ms

Latency jumping from 12 ms to over 600 ms under load is classic bufferbloat. The fix is active queue management (AQM), below, usually combined with shaping slightly under the link speed so the queue builds on a device you control.

Congestion control approaches

Open loop (prevent it):

  • Admission control: refuse new flows when capacity is booked (telephone networks, some 5G slices).
  • Traffic shaping at the edge so senders never exceed their contract.
  • Good retransmission and acknowledgment policies.

Closed loop (detect and react):

  • Implicit signals: loss and rising delay, which TCP and QUIC senders react to.
  • Explicit signals: ECN marks in the IP header, set by routers before queues overflow, telling senders to slow down without losing any packet.
  • Backpressure and choke packets: a congested node asks upstream nodes to slow down (Ethernet pause frames, priority flow control in data center storage networks).

What traffic needs

TrafficBandwidthLatencyJitterLoss tolerance
Voice (VoIP)Low (about 100 kbit/s per call)Under 150 ms one wayUnder 30 msVery low
Video callsMedium to highUnder 150 to 200 msLowLow
Streaming videoHighSeconds are fine (buffered)TolerantMedium (retransmitted)
Online gamesLowAs low as possibleLowLow
Web and APIsMediumModerateTolerantRetransmitted
Backups and bulk transfersAs much as possibleTolerantTolerantRetransmitted

QoS does not create bandwidth. It decides who waits and who is dropped when there is not enough.

The QoS toolkit

1. Classification and marking

Identify traffic (by port, address, application) and mark it so every later device can act quickly. At layer 3 the mark is the 6-bit DSCP field in the IP header; at layer 2 it is the 3-bit PCP in the 802.1Q VLAN tag.

DSCP nameValueTypical use
EF (expedited forwarding)46Voice media
AF4134Interactive video
CS3 / AF3124 / 26Call signaling, important business apps
CS648Routing protocols
Default (best effort)0Everything else
CS1 / LE8 / 1Scavenger: backups, updates

Marks are only trusted inside your own network; ISPs usually reset them at the border.

$ sudo tcpdump -i enp3s0 -nn -v -c 1 udp portrange 16384-32767 | grep -o 'tos 0x[0-9a-f]*'
tos 0xb8

tos 0xb8 is DSCP 46 (0xb8 = 184, and 184 ÷ 4 = 46): this voice packet is marked EF.

2. Queuing and scheduling

The router keeps several queues and decides which to serve next:

  • FIFO: one queue, first come first served. Simple; no protection.
  • Priority queuing: always serve the high queue first. Great for voice, but it can starve everything else, so it is capped.
  • Weighted fair queuing / class-based queuing: each class gets a guaranteed share of the link.
  • Fair queuing per flow (fq, fq_codel, cake): one heavy flow cannot crowd out the others.

3. Active queue management

Instead of waiting for the queue to overflow, drop or ECN-mark a few packets early so TCP senders slow down. RED did this based on queue length; CoDel and its siblings do it based on how long packets wait, which works much better. fq_codel is the default queuing discipline on most Linux distributions.

4. Policing and shaping

Both enforce a rate, differently:

  • Policing drops (or re-marks) traffic above the rate immediately. No added delay, but drops hurt TCP. ISPs police at their edge.
  • Shaping queues the excess and sends it later at the allowed rate. Smooths traffic and avoids drops, at the cost of some delay.

Leaky bucket and token bucket

These two classic algorithms are how shapers and policers decide what conforms.

Leaky bucket: packets pour into a bucket that leaks at a constant rate. No matter how bursty the input, the output is perfectly smooth. If the bucket (queue) is full, new packets are dropped. Good for strict smoothing, but it never lets a burst through even when the link is idle.

Token bucket: tokens drip into a bucket at rate r, up to a maximum of b tokens. Sending a byte costs a token. If tokens are available, traffic goes out immediately, so bursts up to b are allowed after quiet periods; over the long run the average cannot exceed r.

Leaky bucketToken bucket
OutputConstant rateAverage rate r with bursts up to b
Idle timeWastedSaved up as tokens
Typical useStrict smoothingMost real shapers and policers

Worked example: a token bucket with r = 1 MB/s and b = 2 MB on a link that can do 10 MB/s. After two idle seconds the bucket is full. A burst can run at 10 MB/s while tokens last: it gains 1 MB/s and spends 10 MB/s, so it lasts 2 ÷ (10 − 1) ≈ 0.22 s and moves about 2.2 MB. After that the sender is held to 1 MB/s.

QoS on Linux with tc

tc (traffic control) configures queuing disciplines (qdiscs).

$ tc qdisc show dev enp3s0
qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
$ sudo tc qdisc replace dev enp3s0 root tbf rate 20mbit burst 32kbit latency 50ms
$ sudo tc qdisc replace dev enp3s0 root cake bandwidth 480mbit
$ sudo tc qdisc del dev enp3s0 root

The tbf line is a token bucket shaper at 20 Mbit/s. The cake line shapes to just under a 500 Mbit/s uplink with fair queuing and AQM in one step, which is the usual cure for home bufferbloat (OpenWrt’s “SQM” uses it).

tc with netem also creates bad conditions, ideal for testing how an application behaves on a poor network:

$ sudo tc qdisc add dev enp3s0 root netem delay 100ms 20ms loss 1%
$ ping -c 3 192.168.1.1
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=91.3 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=118.7 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=104.2 ms
$ sudo tc qdisc del dev enp3s0 root netem

Measure real throughput between two machines with iperf3:

$ iperf3 -s                               # on the server
$ iperf3 -c 192.168.1.50 -t 10 -P 4       # on the client, 4 parallel streams
[SUM]   0.00-10.00  sec  1.09 GBytes   938 Mbits/sec                  receiver

938 Mbit/s is what a healthy 1 Gbit/s link delivers after header overhead.

Exercise

Ping to your router is 2 ms when idle and 700 ms while a large upload runs. What is the best fix?

  1. Shape the uplink slightly below its real speed with a fair-queuing AQM such as CAKE or fq_codel
  2. Increase the router’s buffer size
  3. Switch the upload from TCP to UDP
  4. Lower the MTU to 576
Show answer

Answer: A. The delay comes from an oversized queue (bufferbloat). Shaping just under the link rate keeps the queue on your device, where AQM keeps it short and fair.

Exercise

What is the main difference between policing and shaping?

  1. Policing works on UDP only; shaping works on TCP only
  2. Shaping drops excess traffic; policing delays it
  3. Policing drops or re-marks excess traffic immediately; shaping queues it and sends it later
  4. They are two names for the same thing
Show answer

Answer: C. Policing enforces the rate by discarding; shaping buffers the excess, trading some delay for fewer drops.

Exercise

Which property lets a token bucket send a burst after an idle period?

  1. It drops packets when idle
  2. Unused tokens accumulate up to the bucket size b and can be spent at once
  3. It doubles the rate every second
  4. It ignores the average rate r
Show answer

Answer: B. Tokens build up while the sender is quiet, so a burst of up to b bytes can go out at full link speed. The long-term average still cannot exceed r.

Key takeaways

  • Congestion shows up as rising latency first, then loss.
  • Oversized queues cause bufferbloat; AQM (CoDel, fq_codel, CAKE) keeps queues short.
  • QoS classifies and marks traffic (DSCP), then queues, polices or shapes it.
  • The leaky bucket gives a constant output; the token bucket allows bursts up to b at average rate r.
  • tc shapes and simulates bad networks; iperf3 measures real throughput.

FAQ

Does QoS help on my internet connection?

Only on the traffic you send, and only if your device holds the queue. Shape your uplink (and optionally downlink) slightly below the provider’s rate so your router, not the modem, decides what waits.

Will marking my packets EF make them faster across the internet?

No. Providers generally ignore or reset DSCP marks from customers. Marks work inside networks you control and on links where you have an agreement with the provider.

Is QoS still needed on fast networks?

Less often in LANs with plenty of capacity, but yes at bottlenecks: WAN links, Wi-Fi, home uplinks and data center links carrying storage traffic. Speed reduces how often congestion happens, not what it does when it happens.

What is ECN and should I enable it?

Explicit Congestion Notification lets routers mark packets instead of dropping them. Linux enables it for connections that request it by default, and modern AQMs use it. It is safe and helpful where the path supports it.

↑ ↓ navigate Enter open Esc close