Lesson 19 advanced

Performance Monitoring and Troubleshooting

A repeatable method for answering "why is the server slow?": the USE method, a first-60-seconds checklist, reading CPU, memory, disk and network signals, pressure stall information, tracing with strace, perf and eBPF, and a full Orbit incident from symptom to cause.

“Orbit is slow” is the least precise and most common message an operations engineer receives. Behind it can be a saturated disk, a process leaking memory, a noisy neighbour on the cloud host, a network dropping packets or simply more users than the server was sized for. Engineers who guess (restart it, add RAM, tweak a kernel setting they read about) sometimes get lucky. Engineers who follow a method find the cause every time, and can explain it in the postmortem.

In this lesson you learn that method first and the tools second. You go through a checklist that gives you the big picture in about a minute, learn to read each resource properly, use pressure stall information to see how much waiting is happening, trace what a process is actually doing, and then work through a real Orbit incident. You finish by generating load safely on your lab so that every number in this lesson becomes something you have seen yourself.

Warning: Use the Multipass VM for this lesson. Containers and WSL hide or distort many of these metrics, and the load generator below should never run on a shared or production machine.

Install the tools on orbit-lab before you start:

$ sudo apt update
$ sudo apt install -y sysstat stress-ng iotop-c

sysstat provides iostat, mpstat, pidstat and sar; stress-ng generates controlled load; iotop-c shows disk I/O per process. On Rocky and RHEL the packages are sysstat, stress-ng and iotop, installed with dnf.

A method before tools

The USE method, described by performance engineer Brendan Gregg, gives you a checklist that cannot forget a resource. For every resource (CPUs, memory, disks, network interfaces), check three things:

  • Utilization: how busy is it, as a percentage of time or capacity?
  • Saturation: is work queuing because it is too busy?
  • Errors: is anything failing, retried or dropped?
ResourceUtilizationSaturationErrors
CPUmpstat: 100 minus %idlevmstat column r above the CPU count; CPU pressureRare; machine check errors in dmesg
Memoryfree -h: used versus availableSwapping (vmstat si/so); memory pressureOOM killer messages in the kernel log
Diskiostat -x: %utilaqu-sz (queue), high r_await/w_await; I/O pressureI/O errors in dmesg, SMART data
Networksar -n DEV: throughput versus link speedRetransmits, socket backlog dropsip -s link errors and drops

Saturation is the column people skip and the one that matters most: a disk at 60% utilization with an empty queue is fine, a disk at 60% with requests waiting 200 ms is the incident.

The first 60 seconds

When you log in to a slow server, run these in order before you form any theory. Together they cover every row of the USE table in about a minute:

  1. uptime: load averages for the last 1, 5 and 15 minutes. Rising, falling or stable?
  2. sudo dmesg -T | tail: recent kernel complaints such as OOM kills, disk errors or dropped packets.
  3. vmstat 1: run queue, blocked processes, swapping and the CPU split, once per second.
  4. mpstat -P ALL 1: per-CPU usage, to spot one core pinned at 100% by a single-threaded process.
  5. pidstat 1: which processes use the CPU, refreshed every second.
  6. iostat -xz 1: per-disk latency, queue and utilization; -z hides idle devices.
  7. free -h: memory available and swap in use.
  8. sar -n DEV 1: network throughput per interface.
  9. sar -n TCP,ETCP 1: new connections and TCP retransmits per second.
  10. top: a final overview, now that you know what to look for.

Press Ctrl+C after a few samples of each. Ignore the first report of vmstat and iostat: it shows averages since boot, not the current state.

Load average on Linux

$ uptime
 10:02:41 up 41 days,  3:17,  2 users,  load average: 9.12, 7.80, 4.05
$ nproc
2

The load average counts processes that are running or waiting for a CPU, plus, on Linux specifically, processes in uninterruptible sleep (state D), which almost always means waiting for disk or network storage. Compare the number with the CPU count from nproc: a load of 9 on a 2-vCPU server means far more work wants to happen than can happen. The three numbers here (9.12 now, 4.05 a quarter of an hour ago) also say the problem is recent and getting worse.

Because of the D state, a high load does not prove the CPUs are busy. A server can show a load of 20 with idle CPUs when processes queue on a slow disk. vmstat tells the two apart.

CPU

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  6      0  91232  10240 412000    0    0   120 98304 1890 2410  6  9  3 82  0  0
 0  7      0  90816  10240 412104    0    0    96 99120 1912 2388  5  8  4 83  0  0
  • r: runnable processes (running or waiting for a CPU). Consistently above the CPU count means CPU saturation.
  • b: processes blocked in uninterruptible sleep, usually on I/O.
  • us: time in user code (your application). sy: time in the kernel (system calls, networking, file systems).
  • id: idle. wa: idle while waiting for I/O, a hint that storage is the bottleneck.
  • st: steal, time the hypervisor gave your virtual CPU to someone else. gu (newer versions) is time spent running guest VMs.

The output above is a textbook I/O problem: few runnable processes, six or seven blocked, 82% wa. More CPUs would not help at all.

Steal time on cloud VMs

st only exists on virtual machines. A few percent is normal. A sustained 10% or more means your VM is not getting the CPU it is paying for: a busy physical host, or, often, a burstable instance type that has used up its CPU credits. The fix is outside the VM (a different instance type or size), so recognising it saves hours of profiling an application that is not the problem.

Per CPU and per process

mpstat -P ALL 1 shows each core separately. One core at 100% %usr while the others idle points to a single-threaded process: adding cores will not help, but splitting the work or running more instances will. pidstat 1 then names the process, with %usr, %system and %wait (time waiting for a CPU) per PID.

Memory

$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.2Gi       180Mi        12Mi       2.6Gi       2.4Gi
Swap:          1.0Gi          0B       1.0Gi

The column to read is available, not free. Linux uses spare RAM as page cache (buff/cache) to keep recently read files in memory, and gives it back the moment a program needs it. A server with 180 MB “free” and 2.4 GB available is healthy; “free” memory is wasted memory.

Signs of real memory trouble, from mild to severe:

  • available keeps shrinking over hours or days: a leak or a growing cache inside an application.
  • vmstat shows non-zero si and so (swap in and swap out per second) continuously. Some swap used is fine; constant swap activity means the working set does not fit in RAM and everything slows down.
  • The OOM killer fires. Confirm it with journalctl -k -g "killed process", and look in systemctl status <unit> for “Failed with result ‘oom-kill'”. When a service has a memory limit (MemoryMax= in its unit), the kill happens inside its cgroup, and /sys/fs/cgroup/system.slice/orbit-api.service/memory.events counts it in the oom_kill line. Lesson 18 walks through such an investigation.

Disk

$ iostat -xz 1 2
Device     r/s     w/s   rkB/s    wkB/s  r_await  w_await  aqu-sz  %util
sda       2.00  410.00    16.0  98304.0     3.50   180.20   74.10   99.80

The real output has more columns; these are the ones to read:

  • r/s, w/s, rkB/s, wkB/s: reads and writes per second, and the volume.
  • r_await, w_await: average time in milliseconds a request took, including time in the queue. This is the latency your application feels. A few milliseconds is normal for SSDs; hundreds is an incident.
  • aqu-sz: average queue length. Consistently above 1 or 2 on a single disk means saturation.
  • %util: share of time the device had at least one request in flight. Near 100% on a spinning disk means it is maxed out. On SSDs, NVMe and cloud volumes, which serve many requests in parallel, 100% %util can still have headroom, so trust await and aqu-sz more.

To find who is doing the I/O, use sudo iotop-c -oPa (-o only active processes, -P processes rather than threads, -a accumulated totals) or pidstat -d 1, which prints kB read and written per second per process.

Also check space, because a full disk causes slowness and failures of its own: df -h per filesystem, df -i for inodes, and sudo du -xh --max-depth=1 /var | sort -h to find the directory that grew. If df and du disagree, a process is still holding a deleted file open; sudo lsof +L1 lists those files.

Network

$ ss -s
Total: 412
TCP:   389 (estab 302, closed 61, orphaned 0, timewait 58)
$ sar -n TCP,ETCP 1 1
10:05:01     active/s passive/s    iseg/s    oseg/s
10:05:02         2.00     48.00    912.00    955.00

10:05:01     atmptf/s  estres/s retrans/s isegerr/s   orsts/s
10:05:02         0.00      0.00      0.40      0.00      1.00
  • ss -s summarizes sockets. Thousands in timewait mean many short-lived connections; a steadily growing estab count can mean a connection leak.
  • passive/s is new incoming connections per second, active/s outgoing ones: a quick measure of load.
  • retrans/s is TCP retransmits per second. Compared with oseg/s, more than about 1% suggests packet loss somewhere on the path.
  • sar -n DEV 1 shows rxkB/s and txkB/s per interface; compare with the link or instance bandwidth limit.
  • ip -s link shows errors and drops per interface, and ss -tin shows retransmits and round-trip time per connection.

Pressure stall information and cgroups

The metrics above show how busy things are. PSI (pressure stall information) shows directly how much time tasks lost waiting for CPU, memory or I/O, which is the closest thing Linux has to a “how much is this hurting?” number:

$ cat /proc/pressure/io
some avg10=78.42 avg60=65.10 avg300=31.77 total=912384011
full avg10=61.05 avg60=52.33 avg300=24.90 total=701230554

some is the percentage of time at least one task was stalled on I/O over the last 10, 60 and 300 seconds; full is the percentage of time all non-idle tasks were stalled at once, meaning no useful work happened. Here, for 61% of the last ten seconds, the server did nothing but wait for the disk. The same files exist for cpu and memory, and every cgroup has its own copy, so /sys/fs/cgroup/system.slice/orbit-api.service/io.pressure shows how much the API alone is suffering.

Since every systemd service and container runs in its own cgroup (cgroup v2 is the default), systemd-cgtop ranks them like top ranks processes:

$ sudo systemd-cgtop -i
Control Group                          Tasks   %CPU   Memory  Input/s Output/s
/                                        182   38.4     2.9G        -        -
system.slice                              61   36.9     2.1G        -        -
system.slice/orbit-backup.service          3   21.7   612.0M        -    60.0M
system.slice/orbit-api.service             1    9.8   410.2M        -    35.0M

-i sorts by I/O load (-c by CPU, -m by memory). This is often the fastest way to blame a whole service or container rather than one PID among many.

Finding the culprit process

Once you know the resource, name the process:

  • In top, press P to sort by CPU, M by memory, 1 to show each CPU, and c to show full command lines. The S column shows the state: R running, S sleeping, D uninterruptible (usually I/O).
  • pidstat -u 1, pidstat -r 1 and pidstat -d 1 give CPU, memory and disk per process as a scrolling log you can paste into an incident channel.
  • ps -eo pid,stat,wchan:20,cmd | awk '$2 ~ /D/' lists processes stuck in D state and the kernel function they wait in.
  • /proc/<pid>/ holds the details: status (memory, threads, state), io (bytes read and written), limits and cmdline.

What is it doing? strace

strace shows the system calls a process makes: every file it opens, every read, write and network call.

$ sudo strace -f -tt -e trace=openat,write -p 2214
10:06:12.104233 openat(AT_FDCWD, "/srv/orbit/public/notes.json", O_RDONLY|O_CLOEXEC) = 7
10:06:12.104871 write(2, "127.0.0.1 - - [03/Nov/2026 10:06:12] \"GET /notes HTTP/1.0\" 200 -\n", 66) = 66
$ sudo strace -c -f -p 2214

-p attaches to a running PID, -f follows its threads and children, -tt adds timestamps, and -e trace= limits output to the calls you care about. -c prints a summary table of call counts and time when you press Ctrl+C. strace slows the traced process noticeably, so attach briefly in production.

The deep end: perf and eBPF

When you need to know where inside the code the CPU goes, or need to watch the whole system at once with low overhead:

  • sudo perf top shows the hottest functions across all processes live. On Ubuntu install linux-tools-common and linux-tools-$(uname -r) first.
  • eBPF tools run small, verified programs inside the kernel. The bcc collection (sudo apt install bpfcc-tools, commands end in -bpfcc on Ubuntu; bcc-tools in /usr/share/bcc/tools/ on RHEL) answers specific questions: execsnoop-bpfcc lists every new process (great for finding cron jobs or scripts that spawn thousands of short commands), opensnoop-bpfcc every file open, and biolatency-bpfcc draws a histogram of disk latency.
  • bpftrace writes one-liners: sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }' prints every file opened, with the process name.

You will not need these every week, but knowing they exist changes “we could not find it” into “we measured it”.

Incident: Orbit is slow

At 10:05 on a Tuesday, users report that saving a note takes five seconds. orbit-web-01 has 2 vCPUs and 4 GB of RAM. Here is the investigation, step by step.

  1. uptime shows load 9.12, 7.80, 4.05 on 2 CPUs: recent and rising.
  2. vmstat 1 shows r at 1, b at 6 and wa above 80%: processes are waiting for the disk, not the CPU.
  3. iostat -xz 1 shows sda with w_await at 180 ms, aqu-sz at 74 and roughly 98 MB/s of writes. The disk is saturated by writes.
  4. cat /proc/pressure/io confirms it: full avg10=61, so most of the time nothing useful runs.
  5. pidstat -d 1 and systemd-cgtop name two writers: tar from orbit-backup.service at about 60 MB/s and the API’s python3 at about 35 MB/s.
  6. df -h / shows 96% used, and sudo du -xh --max-depth=1 /var/log/orbit shows api.log at 31 GB.
  7. The logs explain both: journalctl -u orbit-backup shows the backup started at 10:00 instead of 03:00 (a recent change set OnCalendar=10:00), and the API has been logging at debug level since Friday’s deploy, so api.log grew by about 2 GB an hour without being rotated.

The immediate fix takes three commands: sudo systemctl stop orbit-backup.service to stop the backup, switch the API back to info-level logging and restart it, then sudo truncate -s 0 /var/log/orbit/api.log to release the space without breaking the open file. Within a minute w_await drops to a few milliseconds, the load falls below 2 and the I/O pressure goes back to near zero. Measure again to prove it; do not assume.

The lasting fixes go into the postmortem: correct the timer back to OnCalendar=*-*-* 03:00, add maxsize 500M to /etc/logrotate.d/orbit, alert when a filesystem passes 80%, and make the backup polite in its unit file with Nice=10, IOSchedulingClass=idle and IOWeight=10 so it can never starve the API again. Note what was not needed: no new instance, no kernel tuning, no restart of the database.

When to stop tuning and scale

Tuning is the right answer when something is wasting resources: a runaway job, a missing index, debug logging, a memory leak. Scaling is the right answer when the USE checklist shows high utilization from legitimate work, saturation persists after the waste is gone, and the trend in sar history or your monitoring is rising with traffic. At that point:

  • Scale up (a bigger instance or faster volume) for a single database or a workload that cannot be split.
  • Scale out (more orbit-web servers behind a load balancer) for stateless services such as the API and the front end.
  • Compare the cost honestly: a day of an engineer’s time often costs more than a year of the next instance size.

Kernel parameters found in blog posts are almost never the first fix. Change them only when a measurement points at the exact limit they control, and one at a time.

Practice lab

You want to recognise each kind of saturation before you meet it in production. Keep a second shell open on orbit-lab with vmstat 1 running, and always pass --timeout to stress-ng so the load ends on its own.

  1. Run stress-ng --cpu 2 --timeout 60s and watch r, us and the load average; name the process with pidstat 1.
  2. Run stress-ng --vm 1 --vm-bytes 60% --timeout 60s and watch free -h, si/so and /proc/pressure/memory.
  3. In /tmp, run stress-ng --hdd 1 --hdd-bytes 512M --timeout 60s and read w_await, aqu-sz and %util in iostat -xz 1, the D state in top and /proc/pressure/io.
  4. Find the busiest cgroup during the disk test with sudo systemd-cgtop, and the writing process with sudo iotop-c -oPa.
  5. Bonus: attach sudo strace -c -f -p <pid> to a stress-ng worker for ten seconds and compare the system calls of the CPU and the disk tests.
Show a solution

During the CPU test vmstat shows r around 2, us near 100 and id near 0, the 1-minute load average climbs toward 2, and pidstat 1 lists stress-ng-cpu workers at about 100% each. During the memory test free -h shows available memory dropping by about 60%, si and so stay zero unless the VM runs short (try a higher percentage to see swapping on a VM with swap), and the some line in /proc/pressure/memory rises if reclaim kicks in. The disk test in /tmp makes w_await and aqu-sz jump, %util approach 100, wa rise in vmstat, stress-ng-hdd appear in D state in top and the some and full values in /proc/pressure/io climb. sudo systemd-cgtop shows your session scope under user.slice at the top, and sudo iotop-c -oPa lists the stress-ng-hdd process with the highest write total. For the bonus, strace -c shows almost no system calls for the CPU worker (it computes in user space) and mostly write and other file calls for the disk worker, which is exactly the user-versus-kernel split you saw in us and sy.

Common mistakes

  • Restarting first. A restart destroys the evidence; capture the 60-second checklist output before you act.
  • Reading the first line of vmstat or iostat. It is the average since boot, not the current state.
  • Treating load average as CPU usage. On Linux it includes processes waiting for I/O.
  • Panicking about low “free” memory. Read “available”; the page cache is doing its job.
  • Trusting %util on SSDs and cloud volumes. Latency and queue length tell you more.
  • Changing several things at once. You will not know which one helped, or which one hurt.

Exercise

orbit-web-02 (4 vCPUs) shows a load average of 12, but vmstat 1 reports us 3, sy 4, id 10, wa 83 and a b column of 11. What should you investigate first?

  1. CPU: the load is three times the core count, so add more vCPUs
  2. Storage: most processes are blocked in uninterruptible I/O wait, so check iostat -xz 1 and find the writer with pidstat -d or iotop
  3. Memory: a high load always means swapping
  4. Network: a load above the CPU count means packet loss
Show answer

Answer: B. Linux counts processes in D state in the load average. With the CPUs mostly idle and 83% iowait, the bottleneck is the disk; more CPUs would not help.

Exercise

On a cloud VM running orbit-api, mpstat 1 shows %steal between 25 and 40 while traffic is normal, and the application has not changed. What is the most likely explanation?

  1. The API has a memory leak
  2. The disk is full
  3. The kernel is spending too much time on system calls
  4. The VM is not getting the CPU time it expects from the hypervisor, for example a busy host or a burstable instance out of credits
Show answer

Answer: D. Steal is time your virtual CPU was ready to run but the hypervisor ran something else. The fix lies outside the VM: a different instance type or size, or moving off a busy host.

Exercise

free -h on orbit-db-01 shows 150Mi free, 5.1Gi buff/cache and 5.4Gi available, and vmstat shows si and so at 0. A colleague wants to add RAM. What do you tell them?

  1. Agree, because free memory below 200 MB means the server is about to crash
  2. Suggest clearing the page cache every hour with a cron job
  3. Memory is healthy: the cache is reclaimable, plenty is available and nothing is swapping, so look elsewhere for the slowness
  4. Disable swap so that more RAM is free
Show answer

Answer: C. Linux fills spare RAM with page cache and releases it on demand. “available” and swap activity are the signals that matter; neither shows memory pressure here.

Key takeaways

  • Use a method: for each resource check utilization, saturation and errors (USE) before forming a theory.
  • The first-60-seconds checklist (uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar, top) gives you the whole picture fast.
  • Linux load average includes I/O wait; compare it with nproc and read wa, st, available, await and retransmits to locate the bottleneck.
  • PSI in /proc/pressure/ and per-cgroup pressure files show how much waiting really hurts, and systemd-cgtop blames a whole service.
  • strace, perf and eBPF tools answer “what exactly is it doing” when the basic tools run out.
  • Fix waste first, measure again, and scale when legitimate demand keeps a resource saturated.

FAQ

What is a good load average for a Linux server?

There is no single number; compare it with the CPU count from nproc. A load roughly equal to the number of CPUs means they are fully used without much queuing; well above it means work is waiting. Because Linux includes processes waiting for disk I/O, check vmstat to see whether the load comes from CPU (r, us, sy) or from I/O (b, wa).

How do I find which process is causing high CPU or disk usage?

For CPU, sort top by CPU with P or run pidstat -u 1. For disk, run pidstat -d 1 or sudo iotop-c -oPa, which show read and write rates per process. For services and containers, sudo systemd-cgtop ranks whole cgroups, which is often quicker than chasing individual PIDs.

Why does Linux show so little free memory?

Linux uses otherwise idle RAM as page cache to speed up file access, and frees it instantly when programs need memory. Look at the available column of free -h instead of free, and at the si and so columns of vmstat for swapping. Clearing the cache by hand only makes the system slower until it fills up again.

What is iowait and is a high value bad?

wa is CPU time spent idle while at least one process waits for I/O. A high value tells you that storage (or network storage) is the bottleneck, not the CPU. Confirm with iostat -xz 1: high await and a long aqu-sz mean the disk is saturated, and pidstat -d names the process causing it.

Do I need sysstat installed on every server?

It is worth it: iostat, mpstat, pidstat and sar are the standard tools for this work, and with its data collection enabled (on Ubuntu, set ENABLED="true" in /etc/default/sysstat and run sudo systemctl enable --now sysstat), sar keeps a history so you can see what happened at 03:12 last night. In fleets with Prometheus or another monitoring system the history lives there, but the sysstat tools remain the fastest way to look live.

When should I use eBPF tools instead of top and iostat?

When the standard tools show that something is slow but not why: which files a process opens, which short-lived processes keep spawning, how disk latency is distributed, which kernel function is hot. bcc tools and bpftrace answer these with low overhead on modern kernels, but they need root and some practice, so start with the 60-second checklist and reach for eBPF afterwards.

↑ ↓ navigate Enter open Esc close