This last lesson turns the whole course into a playbook. When something “doesn’t work”, guessing wastes hours. A method finds the broken layer in minutes, and every step uses a command you have already met. Keep the Network Cheatsheet open beside it.
The method
- Define the problem precisely. Who is affected (one user, one subnet, everyone)? Which destination? Since when? What changed? “The app is down” becomes “users in VLAN 30 cannot reach
api.example.comon 443 since 14:05, after the firewall change”. - Gather facts before touching anything: addresses, routes, DNS answers, logs.
- Form one hypothesis and test it with one command.
- Change one thing at a time, and know how to undo it.
- Verify the fix from the user’s point of view, not just from the server.
- Write down cause and fix, so the next person (or you) is faster.
Divide and conquer: test from the middle of the path. If the user can reach the gateway but not the server, the problem is between them, and you have just ruled out half the network.
Bottom up, one question per layer
| # | Question | Linux | Windows |
|---|---|---|---|
| 1 | Is the link up? | ip -brief link, ethtool <if> | Get-NetAdapter |
| 2 | Do I have the right address and mask? | ip -brief addr | ipconfig /all |
| 3 | Is there a default route? | ip route, ip route get <dst> | Get-NetRoute -DestinationPrefix 0.0.0.0/0 |
| 4 | Can I reach the gateway? | ping -c 3 <gateway>, ip neigh | Test-Connection <gateway> |
| 5 | Can I reach the internet by IP? | ping -c 3 1.1.1.1 | Test-Connection 1.1.1.1 |
| 6 | Does DNS resolve? | dig <name>, resolvectl query <name> | Resolve-DnsName <name> |
| 7 | Where does the path stop? | mtr -rwc 20 <dst>, tracepath <dst> | tracert <dst>, pathping <dst> |
| 8 | Is the port reachable? | nc -zv <host> <port> | Test-NetConnection <host> -Port <port> |
| 9 | Is the service listening? | ss -tlnp (on the server) | Get-NetTCPConnection -State Listen |
| 10 | Does the application answer correctly? | curl -v https://<host>/ | curl.exe -v https://<host>/ |
Stop at the first “no”: that is your layer.
Reading the evidence
| What you see | What it means | Next step |
|---|---|---|
NO-CARRIER, link down | Layer 1: cable, port, NIC, switch port shut | Swap cable, check switch port status |
Address 169.254.x.x | No DHCP answer | VLAN, DHCP server, relay (lesson 10) |
Gateway ping fails, ip neigh shows FAILED | Layer 2: wrong VLAN, gateway down | Check switch port VLAN, gateway interface |
ping 1.1.1.1 works, names fail | DNS | dig @1.1.1.1 <name> to compare resolvers |
Network is unreachable | No route at all | Default route missing |
Traceroute stops at a hop, then * * * | Filtering or routing loss there, or just ICMP rate limiting | Test the actual port with nc or curl |
Connection refused | Host reachable, nothing listening (RST) | ss -tlnp on the server, bind address |
| Connection timeout | Something drops packets silently | Firewalls, security groups, routes back |
| TLS errors | Certificate name, expiry, chain, or time skew | openssl s_client, check the clock |
| HTTP 502 / 504 | Proxy reached, backend failed or slow | Backend health and logs |
Refused versus timeout is the most valuable clue in networking. “Refused” means you reached the machine and it said no. “Timeout” means something in the path ate your packets.
Packet capture recipes
When the tables do not settle it, look at the packets. Capture on both ends if you can: if the client sends and the server never sees it, the problem is in between.
$ sudo tcpdump -i any -nn host 203.0.113.10 and port 443
$ sudo tcpdump -i enp3s0 -nn 'tcp[tcpflags] & (tcp-syn|tcp-rst) != 0'
$ sudo tcpdump -i enp3s0 -nn port 53
$ sudo tcpdump -i enp3s0 -nn icmp
$ sudo tcpdump -i enp3s0 -w /tmp/case-123.pcap -c 5000 host 10.20.5.9
| Pattern in the capture | Diagnosis |
|---|---|
| SYN, SYN, SYN… no reply | Dropped by a firewall, or the reply takes another path |
| SYN, then RST | Port closed |
| SYN, SYN-ACK, ACK, then nothing when data is large | MTU or fragmentation problem (try ping -M do -s 1372) |
| Many retransmissions and duplicate ACKs | Packet loss on the path |
| DNS query with no response | Resolver unreachable or blocked |
| ICMP “fragmentation needed” or “packet too big” | Path MTU is smaller than you think |
Open saved .pcap files in Wireshark; its “Expert Information” and “Follow TCP stream” views find most problems quickly.
Ten common scenarios
1. “No internet” for one machine. Walk the table from step 1. Most often: Wi-Fi associated to the wrong network, a 169.254 address, or a stale static DNS server.
2. “No internet” for everyone. Check the edge: the router’s WAN status, the ISP, and whether the router itself can resolve and ping. Then check DNS forwarders.
3. Names fail, IPs work.
$ resolvectl status | grep 'DNS Servers'
$ dig @1.1.1.1 example.com +short
$ dig example.com +short
If the public resolver answers and yours does not, fix or replace your resolver.
4. One website fails, others work. Compare dig results from different resolvers, then curl -v to see if TCP, TLS or HTTP fails. Check whether only IPv6 is broken: curl -4 versus curl -6.
5. Works by IP, fails by name over VPN. The VPN did not push its DNS servers or search domains; internal names go to a public resolver. Check resolvectl status per interface.
6. Small pages load, large ones hang (often over VPN or tunnels). Classic MTU black hole: ICMP “fragmentation needed” is blocked.
$ ping -M do -s 1472 -c 2 203.0.113.10
$ ping -M do -s 1372 -c 2 203.0.113.10
If the smaller one works, lower the tunnel MTU or enable MSS clamping on the router (tcp flags syn tcp option maxseg size set rt mtu in nftables).
7. Server reachable from its own subnet but not from others. Usually the return path: the server has a wrong or missing default route, or a host firewall allows only local traffic. Check ip route on the server and capture there to see if replies leave.
8. Port open locally, closed from outside.
$ ss -tlnp | grep 8080
LISTEN 0 511 127.0.0.1:8080 0.0.0.0:* users:(("node",pid=2211,fd=19))
Bound to 127.0.0.1: only local clients can connect. Bind to 0.0.0.0 (or the right interface), then check host firewall and cloud security groups.
9. Intermittent drops or slowness. Run mtr for a few minutes and watch where loss starts and continues; check interface error counters (ip -s link, ethtool -S); look for duplex mismatch, Wi-Fi signal, or a saturated uplink (bufferbloat, lesson 20). Measure with iperf3 between two points to split the path.
10. Two hosts randomly lose connectivity. Duplicate IP or MAC. Look for flapping entries:
$ sudo arping -D -I enp3s0 -c 2 192.168.1.77
Unicast reply from 192.168.1.77 [52:54:00:7A:11:02] 0.61ms
Unicast reply from 192.168.1.77 [3C:2C:4A:88:01:5B] 0.74ms
Two different MACs answering for one IP means an address conflict.
Habits that save time
- Keep a baseline: normal latency, throughput and error counts, so you can spot “not normal”.
- Check what changed first: deploys, firewall rules, certificates, DNS records.
- Test from the user’s side of the problem, and with the real protocol and port.
- Remember: ping failing does not mean the service is down, and ping working does not mean it is up.
- Look at both directions: many “routing” problems are return-path problems.
- Write the incident down: symptom, cause, fix and how you found it.
Exercise
curl https://10.0.8.20 waits 30 seconds and then times out, while ping 10.0.8.20 works. What is the most likely cause?
Show answer
Answer: B. A closed port answers immediately with “refused”. A long timeout with working ping points to something dropping TCP 443 on the way. DNS is not involved when you use the IP.
Exercise
Users can open small web pages through a new site-to-site VPN, but large downloads hang after the handshake. What do you test first?
Show answer
Answer: C. A handshake and small replies fit in a packet; large segments exceed the tunnel MTU and the “fragmentation needed” messages are blocked. Lower the MTU or clamp the TCP MSS.
Exercise
A server answers from inside its own subnet but not from other subnets, and a capture on the server shows incoming SYNs. What should you check next?
Show answer
Answer: A. The requests arrive, so the forward path works. A wrong default route or a host firewall limiting non-local sources stops the replies.
Key takeaways
- Define the problem precisely and ask what changed.
- Work bottom up: link, address, route, gateway, internet, DNS, path, port, service, application.
- “Refused” means reached and rejected; “timeout” means something dropped packets.
- Capture on both ends; SYN patterns, retransmissions and ICMP messages tell the story.
- MTU black holes, return paths, bind addresses and DNS cause most “mysterious” failures.
FAQ
What is the single most useful troubleshooting command?
There is no single one, but ip route get <destination> (which route?), dig (which address?) and curl -v (what does the application say?) answer the three questions behind most problems.
Why does traceroute show stars in the middle while the destination answers?
Some routers do not answer, or rate-limit, the ICMP messages traceroute relies on. If later hops and the destination answer, the stars are harmless.
Should I capture packets on production servers?
Yes, carefully: filter tightly (host and port), limit the count with -c or file size with -C, and remember captures may contain sensitive data, so store and delete them responsibly.
Where do I go after this course?
Practise: build labs with containerlab, VMs or a cheap cloud account, break things on purpose and fix them. Then go deeper in the area you need: the Kubernetes Tutorial for cluster networking, cloud networking for your provider, or a certification such as CCNA or Network+ for a structured path.