A link can only send so many bits per second. When more traffic arrives than it can carry, packets wait in a queue, and when the queue is full they are dropped. That is congestion. Lesson 14 showed how TCP senders react to it; this lesson looks at the network side: how routers and hosts manage queues, give important traffic priority and smooth bursts, which together are called Quality of Service (QoS).
Why congestion happens
- Speed mismatch: a 10 Gbit/s LAN feeding a 500 Mbit/s internet link.
- Fan-in: many senders converge on one receiver or uplink (common in data centers: “incast”).
- Bursts: traffic arrives in clumps even when the average is low.
- Slow devices: a router’s CPU or buffers cannot keep up.
Symptoms: latency rises first (packets wait in queues), then loss starts (queues overflow), then throughput collapses if senders keep retransmitting.
Bufferbloat
You might think bigger buffers avoid drops, so they must be better. They are not. A queue that holds seconds of traffic turns a busy link into a slow link: every packet, including your video call and DNS lookups, waits behind a huge upload. That is bufferbloat, and it is why a home connection becomes unusable while someone uploads a video.
Test it by measuring latency while the link is busy:
$ ping -c 5 1.1.1.1 | tail -1
rtt min/avg/max/mdev = 11.2/11.6/12.3/0.4 ms
$ # now start a big upload in another terminal and ping again
$ ping -c 5 1.1.1.1 | tail -1
rtt min/avg/max/mdev = 380.1/612.4/901.7/190.2 ms
Latency jumping from 12 ms to over 600 ms under load is classic bufferbloat. The fix is active queue management (AQM), below, usually combined with shaping slightly under the link speed so the queue builds on a device you control.
Congestion control approaches
Open loop (prevent it):
- Admission control: refuse new flows when capacity is booked (telephone networks, some 5G slices).
- Traffic shaping at the edge so senders never exceed their contract.
- Good retransmission and acknowledgment policies.
Closed loop (detect and react):
- Implicit signals: loss and rising delay, which TCP and QUIC senders react to.
- Explicit signals: ECN marks in the IP header, set by routers before queues overflow, telling senders to slow down without losing any packet.
- Backpressure and choke packets: a congested node asks upstream nodes to slow down (Ethernet pause frames, priority flow control in data center storage networks).
What traffic needs
| Traffic | Bandwidth | Latency | Jitter | Loss tolerance |
|---|---|---|---|---|
| Voice (VoIP) | Low (about 100 kbit/s per call) | Under 150 ms one way | Under 30 ms | Very low |
| Video calls | Medium to high | Under 150 to 200 ms | Low | Low |
| Streaming video | High | Seconds are fine (buffered) | Tolerant | Medium (retransmitted) |
| Online games | Low | As low as possible | Low | Low |
| Web and APIs | Medium | Moderate | Tolerant | Retransmitted |
| Backups and bulk transfers | As much as possible | Tolerant | Tolerant | Retransmitted |
QoS does not create bandwidth. It decides who waits and who is dropped when there is not enough.
The QoS toolkit
1. Classification and marking
Identify traffic (by port, address, application) and mark it so every later device can act quickly. At layer 3 the mark is the 6-bit DSCP field in the IP header; at layer 2 it is the 3-bit PCP in the 802.1Q VLAN tag.
| DSCP name | Value | Typical use |
|---|---|---|
| EF (expedited forwarding) | 46 | Voice media |
| AF41 | 34 | Interactive video |
| CS3 / AF31 | 24 / 26 | Call signaling, important business apps |
| CS6 | 48 | Routing protocols |
| Default (best effort) | 0 | Everything else |
| CS1 / LE | 8 / 1 | Scavenger: backups, updates |
Marks are only trusted inside your own network; ISPs usually reset them at the border.
$ sudo tcpdump -i enp3s0 -nn -v -c 1 udp portrange 16384-32767 | grep -o 'tos 0x[0-9a-f]*'
tos 0xb8
tos 0xb8 is DSCP 46 (0xb8 = 184, and 184 ÷ 4 = 46): this voice packet is marked EF.
2. Queuing and scheduling
The router keeps several queues and decides which to serve next:
- FIFO: one queue, first come first served. Simple; no protection.
- Priority queuing: always serve the high queue first. Great for voice, but it can starve everything else, so it is capped.
- Weighted fair queuing / class-based queuing: each class gets a guaranteed share of the link.
- Fair queuing per flow (
fq,fq_codel,cake): one heavy flow cannot crowd out the others.
3. Active queue management
Instead of waiting for the queue to overflow, drop or ECN-mark a few packets early so TCP senders slow down. RED did this based on queue length; CoDel and its siblings do it based on how long packets wait, which works much better. fq_codel is the default queuing discipline on most Linux distributions.
4. Policing and shaping
Both enforce a rate, differently:
- Policing drops (or re-marks) traffic above the rate immediately. No added delay, but drops hurt TCP. ISPs police at their edge.
- Shaping queues the excess and sends it later at the allowed rate. Smooths traffic and avoids drops, at the cost of some delay.
Leaky bucket and token bucket
These two classic algorithms are how shapers and policers decide what conforms.
Leaky bucket: packets pour into a bucket that leaks at a constant rate. No matter how bursty the input, the output is perfectly smooth. If the bucket (queue) is full, new packets are dropped. Good for strict smoothing, but it never lets a burst through even when the link is idle.
Token bucket: tokens drip into a bucket at rate r, up to a maximum of b tokens. Sending a byte costs a token. If tokens are available, traffic goes out immediately, so bursts up to b are allowed after quiet periods; over the long run the average cannot exceed r.
| Leaky bucket | Token bucket | |
|---|---|---|
| Output | Constant rate | Average rate r with bursts up to b |
| Idle time | Wasted | Saved up as tokens |
| Typical use | Strict smoothing | Most real shapers and policers |
Worked example: a token bucket with r = 1 MB/s and b = 2 MB on a link that can do 10 MB/s. After two idle seconds the bucket is full. A burst can run at 10 MB/s while tokens last: it gains 1 MB/s and spends 10 MB/s, so it lasts 2 ÷ (10 − 1) ≈ 0.22 s and moves about 2.2 MB. After that the sender is held to 1 MB/s.
QoS on Linux with tc
tc (traffic control) configures queuing disciplines (qdiscs).
$ tc qdisc show dev enp3s0
qdisc fq_codel 0: root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
$ sudo tc qdisc replace dev enp3s0 root tbf rate 20mbit burst 32kbit latency 50ms
$ sudo tc qdisc replace dev enp3s0 root cake bandwidth 480mbit
$ sudo tc qdisc del dev enp3s0 root
The tbf line is a token bucket shaper at 20 Mbit/s. The cake line shapes to just under a 500 Mbit/s uplink with fair queuing and AQM in one step, which is the usual cure for home bufferbloat (OpenWrt’s “SQM” uses it).
tc with netem also creates bad conditions, ideal for testing how an application behaves on a poor network:
$ sudo tc qdisc add dev enp3s0 root netem delay 100ms 20ms loss 1%
$ ping -c 3 192.168.1.1
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=91.3 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=118.7 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=104.2 ms
$ sudo tc qdisc del dev enp3s0 root netem
Measure real throughput between two machines with iperf3:
$ iperf3 -s # on the server
$ iperf3 -c 192.168.1.50 -t 10 -P 4 # on the client, 4 parallel streams
[SUM] 0.00-10.00 sec 1.09 GBytes 938 Mbits/sec receiver
938 Mbit/s is what a healthy 1 Gbit/s link delivers after header overhead.
Exercise
Ping to your router is 2 ms when idle and 700 ms while a large upload runs. What is the best fix?
Show answer
Answer: A. The delay comes from an oversized queue (bufferbloat). Shaping just under the link rate keeps the queue on your device, where AQM keeps it short and fair.
Exercise
What is the main difference between policing and shaping?
Show answer
Answer: C. Policing enforces the rate by discarding; shaping buffers the excess, trading some delay for fewer drops.
Exercise
Which property lets a token bucket send a burst after an idle period?
Show answer
Answer: B. Tokens build up while the sender is quiet, so a burst of up to b bytes can go out at full link speed. The long-term average still cannot exceed r.
Key takeaways
- Congestion shows up as rising latency first, then loss.
- Oversized queues cause bufferbloat; AQM (CoDel, fq_codel, CAKE) keeps queues short.
- QoS classifies and marks traffic (DSCP), then queues, polices or shapes it.
- The leaky bucket gives a constant output; the token bucket allows bursts up to b at average rate r.
tcshapes and simulates bad networks;iperf3measures real throughput.
FAQ
Does QoS help on my internet connection?
Only on the traffic you send, and only if your device holds the queue. Shape your uplink (and optionally downlink) slightly below the provider’s rate so your router, not the modem, decides what waits.
Will marking my packets EF make them faster across the internet?
No. Providers generally ignore or reset DSCP marks from customers. Marks work inside networks you control and on links where you have an agreement with the provider.
Is QoS still needed on fast networks?
Less often in LANs with plenty of capacity, but yes at bottlenecks: WAN links, Wi-Fi, home uplinks and data center links carrying storage traffic. Speed reduces how often congestion happens, not what it does when it happens.
What is ECN and should I enable it?
Explicit Congestion Notification lets routers mark packets instead of dropping them. Linux enables it for connections that request it by default, and modern AQMs use it. It is safe and helpful where the path supports it.