“The site is down” is rarely a code problem. Far more often a service listens on the wrong address, a DNS record points to an old server, a route is missing or a firewall drops the packets. From the outside all of these look identical: a browser spinning until it times out. A DevOps engineer who can tell them apart in five minutes saves hours of guessing and a lot of blame aimed at the wrong team.
In this lesson you read a server’s network configuration with the modern ip and ss tools, follow a name through DNS, test connections at each layer, get an overview of how Ubuntu and RHEL configure networks, and work through an “Orbit is unreachable” incident from the cable to the cloud. Use the orbit-lab VM: containers and WSL hide part of the network behind their own virtual layers.
If you skipped lesson 9, simulate the Orbit API with sudo mkdir -p /srv/orbit/public && python3 -m http.server 3000 --bind 127.0.0.1 --directory /srv/orbit/public &.
Interfaces and addresses
A network interface is a door between the machine and a network. A server usually has the loopback interface lo and one or more Ethernet interfaces with names like ens3, enp0s3 or eth0. Each interface can carry one or more IP addresses.
$ ip -br addr
lo UNKNOWN 127.0.0.1/8 ::1/128
ens3 UP 10.110.47.23/24 fe80::5054:ff:fe1a:2b3c/64
$ ip addr show ens3
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:1a:2b:3c brd ff:ff:ff:ff:ff:ff
inet 10.110.47.23/24 metric 100 brd 10.110.47.255 scope global dynamic ens3
valid_lft 3412sec preferred_lft 3412sec
inet6 fe80::5054:ff:fe1a:2b3c/64 scope link
valid_lft forever preferred_lft forever
-br(brief) prints one line per interface: name, state, addresses. Start every investigation with it.UP,LOWER_UPmeans the interface is enabled and has a link (the virtual cable is plugged in).UPwithoutLOWER_UPmeans enabled but no carrier.inet 10.110.47.23/24is the IPv4 address with its prefix: the first 24 bits name the network, so neighbours are10.110.47.0to10.110.47.255.dynamicmeans DHCP assigned it, with the remaining lease time below.link/etheris the MAC address;mtu 1500is the largest packet the link carries.
ip -br link shows only the link layer (state and MAC), and ip neigh shows the neighbours the machine has recently talked to. Your lab IP will differ; Multipass uses a different range on each host OS.
Private, public and loopback addresses
| Range | Kind | What it is for |
|---|---|---|
127.0.0.0/8, ::1 | Loopback | The machine talking to itself. Never leaves the host |
10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 | Private (RFC 1918) | Internal networks and cloud VPCs. Not routed on the internet |
100.64.0.0/10 | Carrier-grade NAT | Shared by ISPs and some overlay networks such as Tailscale |
169.254.0.0/16, fe80::/10 | Link-local | Only valid on one link. 169.254.169.254 is the cloud metadata service |
Everything else (for example 203.0.113.10) | Public | Reachable from the internet if routing and firewalls allow |
Orbit’s servers have private addresses in a VPC (orbit-web-01 is 10.0.1.11, orbit-db-01 is 10.0.2.10), and only the load balancer has a public address. A cloud VM rarely sees its public IP on an interface: the provider translates it at the edge, so ip addr shows only the private one.
Routes and the default gateway
An address says who you are; the routing table says where packets go. For each destination the kernel picks the most specific matching route.
$ ip route
default via 10.110.47.1 dev ens3 proto dhcp src 10.110.47.23 metric 100
10.110.47.0/24 dev ens3 proto kernel scope link src 10.110.47.23 metric 100
$ ip route get 1.1.1.1
1.1.1.1 via 10.110.47.1 dev ens3 src 10.110.47.23 uid 1000
cache
The second line says the local /24 network is reachable directly on ens3. The first line is the default gateway: anything not matched elsewhere goes to the router 10.110.47.1. No default route means no internet, even though local traffic works. ip route get answers “which way would a packet to this address go, and from which source IP?”, which is gold on servers with several interfaces or a VPN.
| Legacy (net-tools) | Modern (iproute2) |
|---|---|
ifconfig | ip addr, ip link |
route -n | ip route |
arp -n | ip neigh |
netstat -tulpn | ss -tulpn |
net-tools has been deprecated for years and is not installed on new Ubuntu or RHEL systems. Learn the right-hand column; old blog posts are the only place the left one still lives.
DNS: from name to address
Applications ask the C library to resolve a name, and /etc/nsswitch.conf decides the order. On Ubuntu the hosts: line reads roughly files ... dns: first /etc/hosts, then DNS.
/etc/hostsholds static entries such as10.0.2.10 orbit-db-01. Handy for a lab; in production, prefer real DNS so every host agrees./etc/resolv.confnames the DNS server. On Ubuntu it is a symlink to a stub file pointing at127.0.0.53, a local cache run by systemd-resolved, which forwards to the real servers learned from DHCP or netplan. Do not edit it by hand; your change is overwritten.resolvectlshows and tests what systemd-resolved does.
$ resolvectl status ens3
Link 2 (ens3)
Current Scopes: DNS
Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.110.47.1
DNS Servers: 10.110.47.1
$ getent hosts orbit-db-01
10.0.2.10 orbit-db-01
$ dig +short orbit.example.com
203.0.113.10
$ dig @1.1.1.1 +short orbit.example.com
203.0.113.10
The two tools answer different questions, and confusing them causes many false alarms:
| Tool | What it asks | Reads /etc/hosts? |
|---|---|---|
getent hosts name | “What would my application get?” (the full nsswitch path) | Yes |
resolvectl query name | “What does the local resolver answer, and from where?” | Yes, via resolved |
dig name | “What does the configured DNS server say?” On Ubuntu that is the resolved stub | Indirectly: the stub also serves /etc/hosts entries |
dig @1.1.1.1 name | “What does a public resolver say?” (bypasses every local layer) | No |
If dig @1.1.1.1 has the new address and getent still shows the old one, the cause is a local cache or /etc/hosts, not the DNS record; sudo resolvectl flush-caches clears resolved’s cache. dig comes from bind9-dnsutils (sudo apt install bind9-dnsutils if missing). In its full output, status: NXDOMAIN means the name does not exist and SERVER: shows who answered.
Ports and sockets
An IP address gets a packet to the machine; a port gets it to the right program. A program that waits for connections opens a listening socket on an address and port. ss lists them:
$ sudo ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=511,fd=14))
tcp LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=930,fd=6))
tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=930,fd=7))
tcp LISTEN 0 5 127.0.0.1:3000 0.0.0.0:* users:(("python3",pid=2875,fd=3))
tcp LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=790,fd=6))
tcp LISTEN 0 4096 *:22 *:* users:(("sshd",pid=701,fd=3),("systemd",pid=1,fd=60))
-t TCP, -u UDP, -l listening only, -p show the process (needs sudo for other users’ processes), -n numeric ports instead of names. Filters keep it short: ss -ltn 'sport = :443'. Without -l, ss -tn shows established connections, which tells you who is talking to the server right now.
The local address decides who can connect:
| Bound to | Who can reach it |
|---|---|
127.0.0.1:3000 | Only programs on this machine. Correct for the Orbit API behind nginx, and for Redis |
10.0.2.10:5432 | Anyone who can reach that private address. Right for PostgreSQL on orbit-db-01 |
0.0.0.0:80 or *:80 | Every IPv4 address on every interface (* and [::] include IPv6). Right for nginx |
This table explains a huge share of “it works on the server but not from my laptop” tickets. Note also that systemd (PID 1) appears on port 22: Ubuntu socket-activates SSH through ssh.socket, as you saw in lesson 9.
Testing connectivity
Each tool tests a different layer, so pick deliberately:
$ ping -c 3 10.0.2.10
64 bytes from 10.0.2.10: icmp_seq=1 ttl=64 time=0.412 ms
...
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
$ nc -zv orbit-db-01 5432
Connection to orbit-db-01 (10.0.2.10) 5432 port [tcp/postgresql] succeeded!
$ curl -I http://127.0.0.1:3000/
HTTP/1.0 200 OK
Content-type: text/html
...
$ curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://orbit.example.com/
200 0.184211s
ping -c 3sends three ICMP echo requests. It proves basic reachability, but many clouds and firewalls block ICMP, so a failed ping does not mean the host is down, and a working ping does not mean the service is up.nc -zv host portopens a TCP connection and closes it (-zzero I/O,-vverbose). It is the quickest “is this port reachable from here?” test. “Connection refused” means the host answered but nothing listens (or a firewall rejects); a hang until timeout means packets are being dropped on the way.curl -Ifetches only the response headers;curl -vshows every step: DNS result, TCP connect, TLS handshake, request and response headers.-wprints timing and status for scripts and health checks;--connect-timeout 5stops long hangs.tracepath host(installed by default) ortraceroute hostlists the routers on the path;mtr -rwc 20 hostcombines both with packet loss per hop, which is the evidence your network team or ISP will ask for.
Two neighbours belong here too. nmap scans many ports at once; use it only on systems you are allowed to scan. Remote logins and file copies with ssh, scp and rsync get their own lesson, lesson 16.
How the network is configured
Commands like ip addr add change the running state and are lost at reboot. Persistent configuration lives in files managed by the distribution’s tool.
Ubuntu: netplan. YAML files in /etc/netplan/ describe the network, and netplan renders them for systemd-networkd (servers) or NetworkManager (desktops). On cloud VMs, cloud-init writes 50-cloud-init.yaml from the provider’s data. A static address for orbit-web-01 looks like this:
network:
version: 2
ethernets:
ens3:
dhcp4: false
addresses: [10.0.1.11/24]
routes:
- to: default
via: 10.0.1.1
nameservers:
addresses: [10.0.0.2]
Apply it with sudo netplan try, which rolls back automatically after 120 seconds unless you confirm: a lifesaver when you are connected over SSH to the interface you are changing. sudo netplan apply applies without the safety net, and netplan status shows the result. Keep the files readable only by root (mode 600), because they can hold Wi-Fi or VPN secrets.
RHEL, Rocky, Alma: NetworkManager. Use nmcli: nmcli device status lists interfaces, nmcli connection show lists profiles, and sudo nmcli connection modify ens3 ipv4.method manual ipv4.addresses 10.0.3.5/24 ipv4.gateway 10.0.3.1 ipv4.dns 10.0.0.2 followed by sudo nmcli connection up ens3 applies a static address. Profiles are stored as keyfiles in /etc/NetworkManager/system-connections/.
Warning: On a cloud VM, change the network in the provider’s console or Terraform, not on the box. A wrong gateway over SSH locks you out, and cloud-init may rewrite your file at the next boot anyway.
Troubleshooting: “Orbit is unreachable”
Users report that https://orbit.example.com times out. Work from the bottom layer up and from the server outward. Each step either finds the fault or rules out a whole category.
| Step | Question | Command on the server | Bad sign |
|---|---|---|---|
| 1. Link | Is the interface up? | ip -br link | DOWN, or no LOWER_UP |
| 2. Address | Does it have the expected IP? | ip -br addr | No IPv4, or a 169.254.x.x address (DHCP failed) |
| 3. Route | Can it reach the gateway and beyond? | ip route, ping -c 2 <gateway> | No default via line |
| 4. DNS | Does the name point here? | dig +short orbit.example.com, getent hosts | Old IP, NXDOMAIN, or resolver timeouts |
| 5. Service | Is something listening, on the right address? | sudo ss -ltnp, curl -I http://127.0.0.1/ | Nothing on 80/443, or bound to 127.0.0.1 only |
| 6. Host firewall | Does the server drop the traffic? | sudo ufw status verbose, sudo nft list ruleset | Port 443 not allowed |
| 7. Cloud | Does a security group or network ACL block it? | Provider console or CLI; tcpdump on the server | No packets arrive at all |
On orbit-web-02 the walk stopped at step 5:
$ curl -sI http://127.0.0.1/ | head -1
HTTP/1.1 200 OK
$ sudo ss -ltnp 'sport = :443'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
nginx answers on port 80 locally, but nothing listens on 443: a configuration change had removed the HTTPS server block, and nginx -t plus a reload fixed it. Had 443 been listening, the next suspects would have been the host firewall (lesson 17) and the cloud security group, which sits in front of the VM and is invisible from inside it. Once the network checks pass, read the application logs (lesson 18): the network is fine and the problem is higher up.
Watching packets with tcpdump
When every check looks right and it still fails, look at the packets themselves. tcpdump shows what really arrives at and leaves the interface.
$ sudo tcpdump -ni any -c 3 tcp port 8080
tcpdump: data link type LINUX_SLL2
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:31:12.481902 ens3 In IP 10.110.47.1.52814 > 10.110.47.23.8080: Flags [S], seq 1840, win 64240, length 0
10:31:12.481950 ens3 Out IP 10.110.47.23.8080 > 10.110.47.1.52814: Flags [S.], seq 9921, ack 1841, win 65160, length 0
10:31:12.482301 ens3 In IP 10.110.47.1.52814 > 10.110.47.23.8080: Flags [.], ack 1, win 502, length 0
-n skips name lookups, -i any listens on all interfaces, -c 3 stops after three packets, and the final expression filters (port 443, host 10.0.2.10, tcp port 5432 and host 10.0.1.12). Read the flags: [S] is a connection request, [S.] the reply, [.] an acknowledgement, [R] a reset (nothing listening, or a rejecting firewall). Repeated [S] with no reply means the server is dropping them; no packets at all means they are blocked before they reach the VM, typically by a security group. Add -w capture.pcap to save a capture for Wireshark.
Red Hat and Alpine: ip, ss, ping and tcpdump work the same on Rocky Linux 10. dig lives in bind-utils (sudo dnf install bind-utils), networks are configured with nmcli, and the host firewall is firewalld (sudo firewall-cmd --list-all). Alpine uses /etc/network/interfaces, BusyBox versions of ip and ping with fewer options (apk add iproute2 for the full ip and ss), and apk add bind-tools for dig.
Practice lab
You want to understand why the Orbit API on orbit-lab cannot be reached from your own computer, and prove it with evidence.
- Find the VM’s interface name, IPv4 address, default gateway and DNS server.
- List the listening TCP sockets with their processes and explain why
curl http://<VM IP>:3000/from your host fails whilecurl http://127.0.0.1:3000/on the VM works. - Start a temporary server with
python3 -m http.server 8080(which binds to all addresses), reach it from your host, and capture the handshake withtcpdump. - Add
127.0.0.1 orbit.testto/etc/hosts, then comparegetent hosts orbit.test,dig +short orbit.testanddig @1.1.1.1 +short orbit.test. Explain each answer.
Show a solution
ip -br addr shows the interface (for example ens3) and its address, ip route the default via gateway and resolvectl status the DNS server. sudo ss -ltnp shows python3 on 127.0.0.1:3000, which only accepts connections from the VM itself, so your host gets “connection refused”. With the second server running, ss shows 0.0.0.0:8080, and curl http://<VM IP>:8080/ from the host works while sudo tcpdump -ni any tcp port 8080 on the VM prints the [S], [S.] and [.] packets; stop the server with Ctrl+C afterwards. After editing /etc/hosts with sudo nano, getent hosts orbit.test returns 127.0.0.1 because nsswitch reads the file first. dig +short orbit.test also prints 127.0.0.1, which surprises many people: dig asks the systemd-resolved stub on 127.0.0.53, and resolved serves /etc/hosts entries too. dig @1.1.1.1 +short orbit.test prints nothing, because public DNS has never heard of it and .test is a reserved name.
Common mistakes
- Treating a failed ping as “server down”. ICMP is often blocked; test the actual port with
nc -zvorcurl. - Binding a service to
127.0.0.1and opening the firewall. No firewall rule helps a socket that only listens on loopback. - Editing
/etc/resolv.confon Ubuntu. It is generated; configure DNS in netplan or withresolvectl. - Testing DNS only with the local resolver. Compare
getent hosts(what apps see) withdig @1.1.1.1(what the world sees).
Exercise
From your laptop, curl http://10.0.1.11:3000/ returns “Connection refused”, but on orbit-web-01 itself curl http://127.0.0.1:3000/ works. sudo ss -ltnp shows 127.0.0.1:3000. What is going on?
Show answer
Answer: B. A socket bound to 127.0.0.1 only accepts connections from the same host, and the kernel answers others with a reset (“refused”). A blocking security group would cause a timeout instead. Orbit’s API is meant to be reached through nginx on 80/443.
Exercise
Leo moved Orbit to a new load balancer. dig @1.1.1.1 +short orbit.example.com shows the new IP, but on orbit-reports getent hosts orbit.example.com still returns the old one. Where should he look first?
Show answer
Answer: B. Public DNS already has the new answer, so the record is fine. getent follows nsswitch, which checks /etc/hosts first and then the local resolver cache. Remove the stale hosts entry or flush the cache.
Exercise
Users cannot reach https://orbit.example.com. nginx listens on 0.0.0.0:443, ufw allows 443, and sudo tcpdump -ni any tcp port 443 on the server shows nothing while you retry from your laptop. What is the most likely culprit?
Show answer
Answer: C. tcpdump sees packets before the host firewall acts on them. If not even the SYN arrives, the traffic is stopped earlier: a security group, network ACL, load balancer or a DNS record pointing elsewhere.
Key takeaways
- Use
ip -br addr,ip routeandip route getto see addresses and paths;ifconfigandnetstatare deprecated. - DNS on Ubuntu goes through systemd-resolved; compare
getent hosts(what apps see) withdig @1.1.1.1(what public DNS says). sudo ss -tulpnshows which process listens on which address and port;127.0.0.1is local-only,0.0.0.0is everywhere.- Test the layer you care about:
pingfor reachability,nc -zvfor a port,curl -vfor HTTP and TLS. - Troubleshoot bottom-up and from the server outward: link, IP, route, DNS, listening port, host firewall, cloud security group.
tcpdumpsettles arguments: it shows whether packets arrive and how the server answers.
FAQ
What replaced ifconfig and netstat on Linux?
The iproute2 suite: ip addr and ip link replace ifconfig, ip route replaces route, ip neigh replaces arp, and ss replaces netstat. The old net-tools package is deprecated and no longer installed by default on Ubuntu or RHEL. The new tools show more information and handle modern features such as multiple addresses per interface.
How do I check which ports are open on a Linux server?
On the server, sudo ss -tulpn lists every listening TCP and UDP socket with its process. That shows what is listening, not what is reachable: check the host firewall (sudo ufw status or sudo firewall-cmd --list-all) and test from another machine with nc -zv host port to see what actually gets through.
Why does ping work but my website does not load?
Ping only proves that ICMP packets reach the host. The website needs a process listening on port 80 or 443, a firewall that allows those ports and a working TLS setup. Test the real thing with curl -v https://host/, which shows exactly where the connection stops.
How do I troubleshoot DNS problems on Ubuntu?
Run resolvectl status to see which DNS server each interface uses, getent hosts name to see what applications get, and dig name and dig @1.1.1.1 name to compare your resolver with a public one. Check /etc/hosts for overrides, and clear the local cache with sudo resolvectl flush-caches after a record changes.
What is the difference between 0.0.0.0 and 127.0.0.1?
As a listening address, 127.0.0.1 accepts connections only from the same machine, while 0.0.0.0 accepts them on every IPv4 address the machine has. Bind internal services such as Redis or an API behind a proxy to 127.0.0.1 (or a private IP), and only public-facing services like nginx to 0.0.0.0.
Why is a connection “refused” in one case and “timed out” in another?
“Refused” means a reset came back: the host is reachable but nothing listens on that port, or a firewall actively rejects. “Timed out” means no answer at all: packets are dropped by a firewall or security group, lost to a missing route, or sent to the wrong address. The difference tells you which layer to check next.