Lesson 24 intermediate

Network Troubleshooting Playbook

A repeatable method for fixing network problems: define the problem, work bottom up through the layers with one command per question, read the evidence, capture packets with tcpdump, and solve ten common scenarios from "no internet" to "slow only over VPN", with Linux and Windows commands.

This last lesson turns the whole course into a playbook. When something “doesn’t work”, guessing wastes hours. A method finds the broken layer in minutes, and every step uses a command you have already met. Keep the Network Cheatsheet open beside it.

The method

  1. Define the problem precisely. Who is affected (one user, one subnet, everyone)? Which destination? Since when? What changed? “The app is down” becomes “users in VLAN 30 cannot reach api.example.com on 443 since 14:05, after the firewall change”.
  2. Gather facts before touching anything: addresses, routes, DNS answers, logs.
  3. Form one hypothesis and test it with one command.
  4. Change one thing at a time, and know how to undo it.
  5. Verify the fix from the user’s point of view, not just from the server.
  6. Write down cause and fix, so the next person (or you) is faster.

Divide and conquer: test from the middle of the path. If the user can reach the gateway but not the server, the problem is between them, and you have just ruled out half the network.

Bottom up, one question per layer

#QuestionLinuxWindows
1Is the link up?ip -brief link, ethtool <if>Get-NetAdapter
2Do I have the right address and mask?ip -brief addripconfig /all
3Is there a default route?ip route, ip route get <dst>Get-NetRoute -DestinationPrefix 0.0.0.0/0
4Can I reach the gateway?ping -c 3 <gateway>, ip neighTest-Connection <gateway>
5Can I reach the internet by IP?ping -c 3 1.1.1.1Test-Connection 1.1.1.1
6Does DNS resolve?dig <name>, resolvectl query <name>Resolve-DnsName <name>
7Where does the path stop?mtr -rwc 20 <dst>, tracepath <dst>tracert <dst>, pathping <dst>
8Is the port reachable?nc -zv <host> <port>Test-NetConnection <host> -Port <port>
9Is the service listening?ss -tlnp (on the server)Get-NetTCPConnection -State Listen
10Does the application answer correctly?curl -v https://<host>/curl.exe -v https://<host>/

Stop at the first “no”: that is your layer.

Reading the evidence

What you seeWhat it meansNext step
NO-CARRIER, link downLayer 1: cable, port, NIC, switch port shutSwap cable, check switch port status
Address 169.254.x.xNo DHCP answerVLAN, DHCP server, relay (lesson 10)
Gateway ping fails, ip neigh shows FAILEDLayer 2: wrong VLAN, gateway downCheck switch port VLAN, gateway interface
ping 1.1.1.1 works, names failDNSdig @1.1.1.1 <name> to compare resolvers
Network is unreachableNo route at allDefault route missing
Traceroute stops at a hop, then * * *Filtering or routing loss there, or just ICMP rate limitingTest the actual port with nc or curl
Connection refusedHost reachable, nothing listening (RST)ss -tlnp on the server, bind address
Connection timeoutSomething drops packets silentlyFirewalls, security groups, routes back
TLS errorsCertificate name, expiry, chain, or time skewopenssl s_client, check the clock
HTTP 502 / 504Proxy reached, backend failed or slowBackend health and logs

Refused versus timeout is the most valuable clue in networking. “Refused” means you reached the machine and it said no. “Timeout” means something in the path ate your packets.

Packet capture recipes

When the tables do not settle it, look at the packets. Capture on both ends if you can: if the client sends and the server never sees it, the problem is in between.

$ sudo tcpdump -i any -nn host 203.0.113.10 and port 443
$ sudo tcpdump -i enp3s0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
$ sudo tcpdump -i enp3s0 -nn port 53
$ sudo tcpdump -i enp3s0 -nn icmp
$ sudo tcpdump -i enp3s0 -w /tmp/case-123.pcap -c 5000 host 10.20.5.9
Pattern in the captureDiagnosis
SYN, SYN, SYN… no replyDropped by a firewall, or the reply takes another path
SYN, then RSTPort closed
SYN, SYN-ACK, ACK, then nothing when data is largeMTU or fragmentation problem (try ping -M do -s 1372)
Many retransmissions and duplicate ACKsPacket loss on the path
DNS query with no responseResolver unreachable or blocked
ICMP “fragmentation needed” or “packet too big”Path MTU is smaller than you think

Open saved .pcap files in Wireshark; its “Expert Information” and “Follow TCP stream” views find most problems quickly.

Ten common scenarios

1. “No internet” for one machine. Walk the table from step 1. Most often: Wi-Fi associated to the wrong network, a 169.254 address, or a stale static DNS server.

2. “No internet” for everyone. Check the edge: the router’s WAN status, the ISP, and whether the router itself can resolve and ping. Then check DNS forwarders.

3. Names fail, IPs work.

$ resolvectl status | grep 'DNS Servers'
$ dig @1.1.1.1 example.com +short
$ dig example.com +short

If the public resolver answers and yours does not, fix or replace your resolver.

4. One website fails, others work. Compare dig results from different resolvers, then curl -v to see if TCP, TLS or HTTP fails. Check whether only IPv6 is broken: curl -4 versus curl -6.

5. Works by IP, fails by name over VPN. The VPN did not push its DNS servers or search domains; internal names go to a public resolver. Check resolvectl status per interface.

6. Small pages load, large ones hang (often over VPN or tunnels). Classic MTU black hole: ICMP “fragmentation needed” is blocked.

$ ping -M do -s 1472 -c 2 203.0.113.10
$ ping -M do -s 1372 -c 2 203.0.113.10

If the smaller one works, lower the tunnel MTU or enable MSS clamping on the router (tcp flags syn tcp option maxseg size set rt mtu in nftables).

7. Server reachable from its own subnet but not from others. Usually the return path: the server has a wrong or missing default route, or a host firewall allows only local traffic. Check ip route on the server and capture there to see if replies leave.

8. Port open locally, closed from outside.

$ ss -tlnp | grep 8080
LISTEN 0 511 127.0.0.1:8080 0.0.0.0:* users:(("node",pid=2211,fd=19))

Bound to 127.0.0.1: only local clients can connect. Bind to 0.0.0.0 (or the right interface), then check host firewall and cloud security groups.

9. Intermittent drops or slowness. Run mtr for a few minutes and watch where loss starts and continues; check interface error counters (ip -s link, ethtool -S); look for duplex mismatch, Wi-Fi signal, or a saturated uplink (bufferbloat, lesson 20). Measure with iperf3 between two points to split the path.

10. Two hosts randomly lose connectivity. Duplicate IP or MAC. Look for flapping entries:

$ sudo arping -D -I enp3s0 -c 2 192.168.1.77
Unicast reply from 192.168.1.77 [52:54:00:7A:11:02]  0.61ms
Unicast reply from 192.168.1.77 [3C:2C:4A:88:01:5B]  0.74ms

Two different MACs answering for one IP means an address conflict.

Habits that save time

  • Keep a baseline: normal latency, throughput and error counts, so you can spot “not normal”.
  • Check what changed first: deploys, firewall rules, certificates, DNS records.
  • Test from the user’s side of the problem, and with the real protocol and port.
  • Remember: ping failing does not mean the service is down, and ping working does not mean it is up.
  • Look at both directions: many “routing” problems are return-path problems.
  • Write the incident down: symptom, cause, fix and how you found it.

Exercise

curl https://10.0.8.20 waits 30 seconds and then times out, while ping 10.0.8.20 works. What is the most likely cause?

  1. Nothing is listening on port 443
  2. A firewall or security group silently drops TCP 443
  3. DNS is not resolving the name
  4. The server’s TLS certificate expired
Show answer

Answer: B. A closed port answers immediately with “refused”. A long timeout with working ping points to something dropping TCP 443 on the way. DNS is not involved when you use the IP.

Exercise

Users can open small web pages through a new site-to-site VPN, but large downloads hang after the handshake. What do you test first?

  1. The DNS TTL
  2. The DHCP lease time
  3. The path MTU, with ping -M do -s at decreasing sizes
  4. The SNMP community
Show answer

Answer: C. A handshake and small replies fit in a packet; large segments exceed the tunnel MTU and the “fragmentation needed” messages are blocked. Lower the MTU or clamp the TCP MSS.

Exercise

A server answers from inside its own subnet but not from other subnets, and a capture on the server shows incoming SYNs. What should you check next?

  1. The server’s routes and host firewall, since replies may not be going back the right way
  2. The client’s Wi-Fi signal
  3. The server’s MX record
  4. The switch’s spanning tree root
Show answer

Answer: A. The requests arrive, so the forward path works. A wrong default route or a host firewall limiting non-local sources stops the replies.

Key takeaways

  • Define the problem precisely and ask what changed.
  • Work bottom up: link, address, route, gateway, internet, DNS, path, port, service, application.
  • “Refused” means reached and rejected; “timeout” means something dropped packets.
  • Capture on both ends; SYN patterns, retransmissions and ICMP messages tell the story.
  • MTU black holes, return paths, bind addresses and DNS cause most “mysterious” failures.

FAQ

What is the single most useful troubleshooting command?

There is no single one, but ip route get <destination> (which route?), dig (which address?) and curl -v (what does the application say?) answer the three questions behind most problems.

Why does traceroute show stars in the middle while the destination answers?

Some routers do not answer, or rate-limit, the ICMP messages traceroute relies on. If later hops and the destination answer, the stars are harmless.

Should I capture packets on production servers?

Yes, carefully: filter tightly (host and port), limit the count with -c or file size with -C, and remember captures may contain sensitive data, so store and delete them responsibly.

Where do I go after this course?

Practise: build labs with containerlab, VMs or a cheap cloud account, break things on purpose and fix them. Then go deeper in the area you need: the Kubernetes Tutorial for cluster networking, cloud networking for your provider, or a certification such as CCNA or Network+ for a structured path.

↑ ↓ navigate Enter open Esc close