Lesson 14 intermediate

The Transport Layer and TCP

Ports, sockets and multiplexing, then TCP in depth: the segment header, the three-way handshake, sequence numbers and acknowledgments, retransmission, flow control with the receive window, connection teardown and TIME_WAIT, TCP states, and congestion control from slow start to CUBIC and BBR.

IP delivers packets to a host. The transport layer delivers data to the right process on that host and decides how much care to take on the way. The two classic choices are TCP (reliable, ordered, connection-oriented) and UDP (fast, minimal, connectionless). This lesson covers ports and TCP; lesson 15 covers UDP and the newer transports.

Ports and sockets

A port is a 16-bit number (0 to 65535) that identifies a process’s endpoint:

RangeNameExamples
0 to 1023Well-known (need root to bind on Linux)22 SSH, 53 DNS, 80 HTTP, 443 HTTPS
1024 to 49151Registered3306 MySQL, 5432 PostgreSQL, 6443 Kubernetes API
49152 to 65535Dynamic / ephemeralChosen by the OS for client connections

A socket is an address plus a port. A TCP connection is identified by the 4-tuple (source IP, source port, destination IP, destination port), plus the protocol. That is how one web server on port 443 can hold a hundred thousand connections at once: each one has a different client address or port. Using ports to share one network connection among many applications is called multiplexing.

$ ss -tlnp
State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096           0.0.0.0:22          0.0.0.0:*      users:(("sshd",pid=812,fd=3))
LISTEN  0       511          127.0.0.1:5432        0.0.0.0:*      users:(("postgres",pid=1044,fd=6))
LISTEN  0       511                  *:443               *:*      users:(("nginx",pid=1290,fd=7))
$ cat /proc/sys/net/ipv4/ip_local_port_range
32768	60999

127.0.0.1:5432 listens only on loopback, so PostgreSQL is unreachable from other machines. 0.0.0.0 or * means every address. On Windows: Get-NetTCPConnection -State Listen.

What TCP promises

  • Reliable: lost segments are retransmitted.
  • Ordered: the application reads bytes in the order they were sent.
  • Byte stream: no message boundaries; the application frames its own messages.
  • Flow controlled: the sender never overwhelms the receiver.
  • Congestion controlled: the sender backs off when the network is overloaded.
  • Full duplex and connection-oriented: both sides set up state first and can send at once.

The TCP header

FieldSizePurpose
Source / destination port16 bits eachWhich processes
Sequence number32 bitsPosition of the first data byte in the stream
Acknowledgment number32 bitsNext byte expected from the other side
Data offset4 bitsHeader length (20 to 60 bytes)
Flags9 bitsSYN, ACK, FIN, RST, PSH, URG, ECE, CWR
Window16 bitsHow many bytes the receiver can accept now
Checksum16 bitsHeader and data
Urgent pointer16 bitsRarely used
OptionsvariableMSS, window scale, SACK, timestamps

The three-way handshake

Client                                        Server (listening on 443)
  | --- SYN  seq=1000 ----------------------->  |   "I want to talk; my numbers start at 1000"
  | <-- SYN-ACK  seq=5000 ack=1001 ----------  |   "OK; mine start at 5000; I expect your 1001"
  | --- ACK  ack=5001 ---------------------->  |   "Got it; I expect your 5001"
  |            connection ESTABLISHED          |

Both sides pick a random initial sequence number (to defeat spoofing) and agree on options such as the MSS (maximum segment size: 1,460 bytes on a 1,500 MTU, after 40 bytes of IP and TCP headers), window scaling and SACK. You can watch it:

$ sudo tcpdump -i any -nn 'tcp port 443 and host 203.0.113.10' -c 3
11:05:02.100 IP 192.168.1.42.51644 > 203.0.113.10.443: Flags [S], seq 1000, win 64240, options [mss 1460,sackOK,TS val 1 ecr 0,nop,wscale 7], length 0
11:05:02.112 IP 203.0.113.10.443 > 192.168.1.42.51644: Flags [S.], seq 5000, ack 1001, win 65160, options [mss 1460,sackOK,TS val 9 ecr 1,nop,wscale 7], length 0
11:05:02.112 IP 192.168.1.42.51644 > 203.0.113.10.443: Flags [.], ack 5001, win 502, length 0

In tcpdump, S is SYN, . is ACK, F is FIN, R is RST and P is PSH. The handshake costs one full round trip before any data flows; that delay is why latency matters so much.

What different failures look like:

You seeMeaning
SYN, then nothing, repeated SYNsA firewall silently drops the traffic, or the host is down
SYN, then RSTThe host is up but nothing listens on that port (“connection refused”)
Handshake OK, then RST laterThe application or a middlebox killed the connection

Sequence numbers, ACKs and retransmission

Every byte has a sequence number. The receiver sends cumulative ACKs: “I have everything up to byte N−1, send N next”. If the sender does not get an ACK within the retransmission timeout (RTO, computed from measured round-trip times), it resends. Faster recovery comes from:

  • Fast retransmit: three duplicate ACKs for the same byte mean a segment is missing; resend it without waiting for the timer.
  • SACK (selective acknowledgment): the receiver lists the blocks it does have, so only the gaps are resent.
$ nstat -az | grep -E 'TcpRetransSegs|TcpOutSegs'
TcpOutSegs                      48210933           0.0
TcpRetransSegs                     71204           0.0

That is about 0.15% retransmitted, healthy. Several percent points at loss somewhere on the path.

Flow control

The window field tells the sender how much free buffer the receiver has. If the application reads slowly, the window shrinks, possibly to zero, and the sender pauses. Window scaling multiplies the 16-bit field (max 65,535) so windows can reach gigabytes, which long, fast paths need: filling a 1 Gbit/s path with 100 ms of RTT needs a window of about 12.5 MB (the bandwidth-delay product from lesson 5).

Closing a connection and TIME_WAIT

Each direction closes separately with a FIN and its ACK, so a graceful close usually takes four segments. The side that closes first enters TIME_WAIT for twice the maximum segment lifetime (60 seconds on Linux) so stray late segments cannot confuse a new connection with the same 4-tuple. An RST aborts a connection immediately with no goodbye.

$ ss -tan state time-wait | wc -l
1842
$ ss -s
TCP:   2391 (estab 412, closed 1852, orphaned 0, timewait 1842)

Thousands of TIME_WAIT sockets on a busy client are normal; they only become a problem when a client opens so many short connections to one destination that it runs out of ephemeral ports. Connection reuse (HTTP keep-alive, connection pools) is the fix.

The TCP states

StateMeaning
LISTENServer waiting for connections
SYN-SENT / SYN-RECEIVEDHandshake in progress
ESTABLISHEDData can flow
FIN-WAIT-1 / FIN-WAIT-2We closed, waiting for the other side
CLOSE-WAITThe other side closed; our application has not closed yet
LAST-ACKWe closed after them; waiting for the final ACK
TIME-WAITClosed first, waiting out stray segments

Many CLOSE-WAIT sockets are an application bug. The peer has hung up, but your program never called close(). They pile up until the process runs out of file descriptors. TIME-WAIT, by contrast, is normal.

Congestion control

Flow control protects the receiver; congestion control protects the network. The sender keeps a congestion window (cwnd) and may only have the smaller of cwnd and the receiver’s window in flight.

  1. Slow start: cwnd starts small (10 segments on Linux) and doubles every round trip.
  2. Congestion avoidance: after a threshold, cwnd grows roughly one segment per round trip.
  3. On loss: cwnd is cut (multiplicative decrease), then growth resumes. This “additive increase, multiplicative decrease” (AIMD) is what lets millions of flows share links fairly.

Algorithms differ in how they grow and react:

  • Reno / NewReno: the classic AIMD.
  • CUBIC: the Linux and Windows default; grows along a cubic curve to fill fast, long paths quickly.
  • BBR: models the path’s bandwidth and round trip time instead of reacting to loss; often much better on lossy or long links.
  • ECN: routers mark packets instead of dropping them, so senders slow down before loss happens.
$ sysctl net.ipv4.tcp_congestion_control net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_available_congestion_control = reno cubic bbr
$ ss -ti dst 203.0.113.10
ESTAB 0 0 192.168.1.42:51644 203.0.113.10:443
	 cubic wscale:7,7 rto:212 rtt:11.402/0.73 mss:1448 cwnd:42 bytes_acked:5310882 retrans:0/3

ss -ti shows the live congestion state of each connection: algorithm, round-trip time and current cwnd.

Exercise

curl https://10.0.0.5:8443 returns “Connection refused” immediately. What happened at the TCP level?

  1. The host answered the SYN with an RST because nothing listens on port 8443
  2. A firewall silently dropped the SYN
  3. The TLS certificate is invalid
  4. DNS failed to resolve the address
Show answer

Answer: A. “Connection refused” means the host is reachable and replied with RST. A silent drop by a firewall looks like a timeout instead.

Exercise

A server shows thousands of sockets in CLOSE-WAIT that never go away. Where is the bug?

  1. In the network, which is losing FIN segments
  2. In the client, which never sends FIN
  3. In the TCP congestion control setting
  4. In the server application, which does not close sockets after the peer hangs up
Show answer

Answer: D. CLOSE-WAIT means the remote side already sent FIN and the kernel is waiting for the local program to call close(). Only the application can fix it.

Exercise

What does TCP’s receive window protect against?

  1. Congestion in routers along the path
  2. A fast sender overflowing a slow receiver’s buffer
  3. Packets arriving out of order
  4. Spoofed SYN packets
Show answer

Answer: B. The window is flow control: the receiver advertises how much buffer it has free. Congestion is handled separately by the congestion window.

Key takeaways

  • Ports identify processes; a connection is a unique 4-tuple.
  • TCP’s handshake is SYN, SYN-ACK, ACK and costs one round trip.
  • Sequence numbers and ACKs drive retransmission; SACK and fast retransmit speed recovery.
  • The receive window is flow control; the congestion window protects the network.
  • TIME-WAIT is normal; piles of CLOSE-WAIT are an application bug.

FAQ

Why is it called a “three-way” handshake if closing takes four segments?

Opening combines the server’s SYN and ACK into one segment. On close, each side usually sends its FIN at a different time, so the ACK and FIN travel separately. Sometimes they are combined and the close also takes three.

What is a SYN flood?

An attack that sends many SYNs without finishing the handshake, filling the server’s queue of half-open connections. SYN cookies (on by default in Linux) let the server answer without storing state until the final ACK arrives.

Should I switch to BBR?

It often helps servers that send to distant or mobile clients. Test it: sysctl -w net.ipv4.tcp_congestion_control=bbr, usually together with the fq queuing discipline, and compare throughput and retransmissions.

What is Nagle’s algorithm?

A rule that holds back small segments until earlier data is acknowledged, to avoid sending many tiny packets. Interactive protocols disable it with the TCP_NODELAY socket option.

↑ ↓ navigate Enter open Esc close