“Orbit is slow” is the least precise and most common message an operations engineer receives. Behind it can be a saturated disk, a process leaking memory, a noisy neighbour on the cloud host, a network dropping packets or simply more users than the server was sized for. Engineers who guess (restart it, add RAM, tweak a kernel setting they read about) sometimes get lucky. Engineers who follow a method find the cause every time, and can explain it in the postmortem.
In this lesson you learn that method first and the tools second. You go through a checklist that gives you the big picture in about a minute, learn to read each resource properly, use pressure stall information to see how much waiting is happening, trace what a process is actually doing, and then work through a real Orbit incident. You finish by generating load safely on your lab so that every number in this lesson becomes something you have seen yourself.
Warning: Use the Multipass VM for this lesson. Containers and WSL hide or distort many of these metrics, and the load generator below should never run on a shared or production machine.
Install the tools on orbit-lab before you start:
$ sudo apt update
$ sudo apt install -y sysstat stress-ng iotop-c
sysstat provides iostat, mpstat, pidstat and sar; stress-ng generates controlled load; iotop-c shows disk I/O per process. On Rocky and RHEL the packages are sysstat, stress-ng and iotop, installed with dnf.
A method before tools
The USE method, described by performance engineer Brendan Gregg, gives you a checklist that cannot forget a resource. For every resource (CPUs, memory, disks, network interfaces), check three things:
- Utilization: how busy is it, as a percentage of time or capacity?
- Saturation: is work queuing because it is too busy?
- Errors: is anything failing, retried or dropped?
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | mpstat: 100 minus %idle | vmstat column r above the CPU count; CPU pressure | Rare; machine check errors in dmesg |
| Memory | free -h: used versus available | Swapping (vmstat si/so); memory pressure | OOM killer messages in the kernel log |
| Disk | iostat -x: %util | aqu-sz (queue), high r_await/w_await; I/O pressure | I/O errors in dmesg, SMART data |
| Network | sar -n DEV: throughput versus link speed | Retransmits, socket backlog drops | ip -s link errors and drops |
Saturation is the column people skip and the one that matters most: a disk at 60% utilization with an empty queue is fine, a disk at 60% with requests waiting 200 ms is the incident.
The first 60 seconds
When you log in to a slow server, run these in order before you form any theory. Together they cover every row of the USE table in about a minute:
uptime: load averages for the last 1, 5 and 15 minutes. Rising, falling or stable?sudo dmesg -T | tail: recent kernel complaints such as OOM kills, disk errors or dropped packets.vmstat 1: run queue, blocked processes, swapping and the CPU split, once per second.mpstat -P ALL 1: per-CPU usage, to spot one core pinned at 100% by a single-threaded process.pidstat 1: which processes use the CPU, refreshed every second.iostat -xz 1: per-disk latency, queue and utilization;-zhides idle devices.free -h: memory available and swap in use.sar -n DEV 1: network throughput per interface.sar -n TCP,ETCP 1: new connections and TCP retransmits per second.top: a final overview, now that you know what to look for.
Press Ctrl+C after a few samples of each. Ignore the first report of vmstat and iostat: it shows averages since boot, not the current state.
Load average on Linux
$ uptime
10:02:41 up 41 days, 3:17, 2 users, load average: 9.12, 7.80, 4.05
$ nproc
2
The load average counts processes that are running or waiting for a CPU, plus, on Linux specifically, processes in uninterruptible sleep (state D), which almost always means waiting for disk or network storage. Compare the number with the CPU count from nproc: a load of 9 on a 2-vCPU server means far more work wants to happen than can happen. The three numbers here (9.12 now, 4.05 a quarter of an hour ago) also say the problem is recent and getting worse.
Because of the D state, a high load does not prove the CPUs are busy. A server can show a load of 20 with idle CPUs when processes queue on a slow disk. vmstat tells the two apart.
CPU
$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
1 6 0 91232 10240 412000 0 0 120 98304 1890 2410 6 9 3 82 0 0
0 7 0 90816 10240 412104 0 0 96 99120 1912 2388 5 8 4 83 0 0
r: runnable processes (running or waiting for a CPU). Consistently above the CPU count means CPU saturation.b: processes blocked in uninterruptible sleep, usually on I/O.us: time in user code (your application).sy: time in the kernel (system calls, networking, file systems).id: idle.wa: idle while waiting for I/O, a hint that storage is the bottleneck.st: steal, time the hypervisor gave your virtual CPU to someone else.gu(newer versions) is time spent running guest VMs.
The output above is a textbook I/O problem: few runnable processes, six or seven blocked, 82% wa. More CPUs would not help at all.
Steal time on cloud VMs
st only exists on virtual machines. A few percent is normal. A sustained 10% or more means your VM is not getting the CPU it is paying for: a busy physical host, or, often, a burstable instance type that has used up its CPU credits. The fix is outside the VM (a different instance type or size), so recognising it saves hours of profiling an application that is not the problem.
Per CPU and per process
mpstat -P ALL 1 shows each core separately. One core at 100% %usr while the others idle points to a single-threaded process: adding cores will not help, but splitting the work or running more instances will. pidstat 1 then names the process, with %usr, %system and %wait (time waiting for a CPU) per PID.
Memory
$ free -h
total used free shared buff/cache available
Mem: 3.8Gi 1.2Gi 180Mi 12Mi 2.6Gi 2.4Gi
Swap: 1.0Gi 0B 1.0Gi
The column to read is available, not free. Linux uses spare RAM as page cache (buff/cache) to keep recently read files in memory, and gives it back the moment a program needs it. A server with 180 MB “free” and 2.4 GB available is healthy; “free” memory is wasted memory.
Signs of real memory trouble, from mild to severe:
- available keeps shrinking over hours or days: a leak or a growing cache inside an application.
vmstatshows non-zero si and so (swap in and swap out per second) continuously. Some swap used is fine; constant swap activity means the working set does not fit in RAM and everything slows down.- The OOM killer fires. Confirm it with
journalctl -k -g "killed process", and look insystemctl status <unit>for “Failed with result ‘oom-kill'”. When a service has a memory limit (MemoryMax=in its unit), the kill happens inside its cgroup, and/sys/fs/cgroup/system.slice/orbit-api.service/memory.eventscounts it in theoom_killline. Lesson 18 walks through such an investigation.
Disk
$ iostat -xz 1 2
Device r/s w/s rkB/s wkB/s r_await w_await aqu-sz %util
sda 2.00 410.00 16.0 98304.0 3.50 180.20 74.10 99.80
The real output has more columns; these are the ones to read:
r/s,w/s,rkB/s,wkB/s: reads and writes per second, and the volume.r_await,w_await: average time in milliseconds a request took, including time in the queue. This is the latency your application feels. A few milliseconds is normal for SSDs; hundreds is an incident.aqu-sz: average queue length. Consistently above 1 or 2 on a single disk means saturation.%util: share of time the device had at least one request in flight. Near 100% on a spinning disk means it is maxed out. On SSDs, NVMe and cloud volumes, which serve many requests in parallel, 100%%utilcan still have headroom, so trustawaitandaqu-szmore.
To find who is doing the I/O, use sudo iotop-c -oPa (-o only active processes, -P processes rather than threads, -a accumulated totals) or pidstat -d 1, which prints kB read and written per second per process.
Also check space, because a full disk causes slowness and failures of its own: df -h per filesystem, df -i for inodes, and sudo du -xh --max-depth=1 /var | sort -h to find the directory that grew. If df and du disagree, a process is still holding a deleted file open; sudo lsof +L1 lists those files.
Network
$ ss -s
Total: 412
TCP: 389 (estab 302, closed 61, orphaned 0, timewait 58)
$ sar -n TCP,ETCP 1 1
10:05:01 active/s passive/s iseg/s oseg/s
10:05:02 2.00 48.00 912.00 955.00
10:05:01 atmptf/s estres/s retrans/s isegerr/s orsts/s
10:05:02 0.00 0.00 0.40 0.00 1.00
ss -ssummarizes sockets. Thousands intimewaitmean many short-lived connections; a steadily growingestabcount can mean a connection leak.passive/sis new incoming connections per second,active/soutgoing ones: a quick measure of load.retrans/sis TCP retransmits per second. Compared withoseg/s, more than about 1% suggests packet loss somewhere on the path.sar -n DEV 1showsrxkB/sandtxkB/sper interface; compare with the link or instance bandwidth limit.ip -s linkshows errors and drops per interface, andss -tinshows retransmits and round-trip time per connection.
Pressure stall information and cgroups
The metrics above show how busy things are. PSI (pressure stall information) shows directly how much time tasks lost waiting for CPU, memory or I/O, which is the closest thing Linux has to a “how much is this hurting?” number:
$ cat /proc/pressure/io
some avg10=78.42 avg60=65.10 avg300=31.77 total=912384011
full avg10=61.05 avg60=52.33 avg300=24.90 total=701230554
some is the percentage of time at least one task was stalled on I/O over the last 10, 60 and 300 seconds; full is the percentage of time all non-idle tasks were stalled at once, meaning no useful work happened. Here, for 61% of the last ten seconds, the server did nothing but wait for the disk. The same files exist for cpu and memory, and every cgroup has its own copy, so /sys/fs/cgroup/system.slice/orbit-api.service/io.pressure shows how much the API alone is suffering.
Since every systemd service and container runs in its own cgroup (cgroup v2 is the default), systemd-cgtop ranks them like top ranks processes:
$ sudo systemd-cgtop -i
Control Group Tasks %CPU Memory Input/s Output/s
/ 182 38.4 2.9G - -
system.slice 61 36.9 2.1G - -
system.slice/orbit-backup.service 3 21.7 612.0M - 60.0M
system.slice/orbit-api.service 1 9.8 410.2M - 35.0M
-i sorts by I/O load (-c by CPU, -m by memory). This is often the fastest way to blame a whole service or container rather than one PID among many.
Finding the culprit process
Once you know the resource, name the process:
- In
top, pressPto sort by CPU,Mby memory,1to show each CPU, andcto show full command lines. TheScolumn shows the state:Rrunning,Ssleeping,Duninterruptible (usually I/O). pidstat -u 1,pidstat -r 1andpidstat -d 1give CPU, memory and disk per process as a scrolling log you can paste into an incident channel.ps -eo pid,stat,wchan:20,cmd | awk '$2 ~ /D/'lists processes stuck inDstate and the kernel function they wait in./proc/<pid>/holds the details:status(memory, threads, state),io(bytes read and written),limitsandcmdline.
What is it doing? strace
strace shows the system calls a process makes: every file it opens, every read, write and network call.
$ sudo strace -f -tt -e trace=openat,write -p 2214
10:06:12.104233 openat(AT_FDCWD, "/srv/orbit/public/notes.json", O_RDONLY|O_CLOEXEC) = 7
10:06:12.104871 write(2, "127.0.0.1 - - [03/Nov/2026 10:06:12] \"GET /notes HTTP/1.0\" 200 -\n", 66) = 66
$ sudo strace -c -f -p 2214
-p attaches to a running PID, -f follows its threads and children, -tt adds timestamps, and -e trace= limits output to the calls you care about. -c prints a summary table of call counts and time when you press Ctrl+C. strace slows the traced process noticeably, so attach briefly in production.
The deep end: perf and eBPF
When you need to know where inside the code the CPU goes, or need to watch the whole system at once with low overhead:
sudo perf topshows the hottest functions across all processes live. On Ubuntu installlinux-tools-commonandlinux-tools-$(uname -r)first.- eBPF tools run small, verified programs inside the kernel. The bcc collection (
sudo apt install bpfcc-tools, commands end in-bpfccon Ubuntu;bcc-toolsin/usr/share/bcc/tools/on RHEL) answers specific questions:execsnoop-bpfcclists every new process (great for finding cron jobs or scripts that spawn thousands of short commands),opensnoop-bpfccevery file open, andbiolatency-bpfccdraws a histogram of disk latency. bpftracewrites one-liners:sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'prints every file opened, with the process name.
You will not need these every week, but knowing they exist changes “we could not find it” into “we measured it”.
Incident: Orbit is slow
At 10:05 on a Tuesday, users report that saving a note takes five seconds. orbit-web-01 has 2 vCPUs and 4 GB of RAM. Here is the investigation, step by step.
uptimeshows load 9.12, 7.80, 4.05 on 2 CPUs: recent and rising.vmstat 1showsrat 1,bat 6 andwaabove 80%: processes are waiting for the disk, not the CPU.iostat -xz 1showssdawithw_awaitat 180 ms,aqu-szat 74 and roughly 98 MB/s of writes. The disk is saturated by writes.cat /proc/pressure/ioconfirms it:full avg10=61, so most of the time nothing useful runs.pidstat -d 1andsystemd-cgtopname two writers:tarfromorbit-backup.serviceat about 60 MB/s and the API’spython3at about 35 MB/s.df -h /shows 96% used, andsudo du -xh --max-depth=1 /var/log/orbitshowsapi.logat 31 GB.- The logs explain both:
journalctl -u orbit-backupshows the backup started at 10:00 instead of 03:00 (a recent change setOnCalendar=10:00), and the API has been logging at debug level since Friday’s deploy, soapi.loggrew by about 2 GB an hour without being rotated.
The immediate fix takes three commands: sudo systemctl stop orbit-backup.service to stop the backup, switch the API back to info-level logging and restart it, then sudo truncate -s 0 /var/log/orbit/api.log to release the space without breaking the open file. Within a minute w_await drops to a few milliseconds, the load falls below 2 and the I/O pressure goes back to near zero. Measure again to prove it; do not assume.
The lasting fixes go into the postmortem: correct the timer back to OnCalendar=*-*-* 03:00, add maxsize 500M to /etc/logrotate.d/orbit, alert when a filesystem passes 80%, and make the backup polite in its unit file with Nice=10, IOSchedulingClass=idle and IOWeight=10 so it can never starve the API again. Note what was not needed: no new instance, no kernel tuning, no restart of the database.
When to stop tuning and scale
Tuning is the right answer when something is wasting resources: a runaway job, a missing index, debug logging, a memory leak. Scaling is the right answer when the USE checklist shows high utilization from legitimate work, saturation persists after the waste is gone, and the trend in sar history or your monitoring is rising with traffic. At that point:
- Scale up (a bigger instance or faster volume) for a single database or a workload that cannot be split.
- Scale out (more
orbit-webservers behind a load balancer) for stateless services such as the API and the front end. - Compare the cost honestly: a day of an engineer’s time often costs more than a year of the next instance size.
Kernel parameters found in blog posts are almost never the first fix. Change them only when a measurement points at the exact limit they control, and one at a time.
Practice lab
You want to recognise each kind of saturation before you meet it in production. Keep a second shell open on orbit-lab with vmstat 1 running, and always pass --timeout to stress-ng so the load ends on its own.
- Run
stress-ng --cpu 2 --timeout 60sand watchr,usand the load average; name the process withpidstat 1. - Run
stress-ng --vm 1 --vm-bytes 60% --timeout 60sand watchfree -h,si/soand/proc/pressure/memory. - In
/tmp, runstress-ng --hdd 1 --hdd-bytes 512M --timeout 60sand readw_await,aqu-szand%utiliniostat -xz 1, theDstate intopand/proc/pressure/io. - Find the busiest cgroup during the disk test with
sudo systemd-cgtop, and the writing process withsudo iotop-c -oPa. - Bonus: attach
sudo strace -c -f -p <pid>to a stress-ng worker for ten seconds and compare the system calls of the CPU and the disk tests.
Show a solution
During the CPU test vmstat shows r around 2, us near 100 and id near 0, the 1-minute load average climbs toward 2, and pidstat 1 lists stress-ng-cpu workers at about 100% each. During the memory test free -h shows available memory dropping by about 60%, si and so stay zero unless the VM runs short (try a higher percentage to see swapping on a VM with swap), and the some line in /proc/pressure/memory rises if reclaim kicks in. The disk test in /tmp makes w_await and aqu-sz jump, %util approach 100, wa rise in vmstat, stress-ng-hdd appear in D state in top and the some and full values in /proc/pressure/io climb. sudo systemd-cgtop shows your session scope under user.slice at the top, and sudo iotop-c -oPa lists the stress-ng-hdd process with the highest write total. For the bonus, strace -c shows almost no system calls for the CPU worker (it computes in user space) and mostly write and other file calls for the disk worker, which is exactly the user-versus-kernel split you saw in us and sy.
Common mistakes
- Restarting first. A restart destroys the evidence; capture the 60-second checklist output before you act.
- Reading the first line of vmstat or iostat. It is the average since boot, not the current state.
- Treating load average as CPU usage. On Linux it includes processes waiting for I/O.
- Panicking about low “free” memory. Read “available”; the page cache is doing its job.
- Trusting
%utilon SSDs and cloud volumes. Latency and queue length tell you more. - Changing several things at once. You will not know which one helped, or which one hurt.
Exercise
orbit-web-02 (4 vCPUs) shows a load average of 12, but vmstat 1 reports us 3, sy 4, id 10, wa 83 and a b column of 11. What should you investigate first?
Show answer
Answer: B. Linux counts processes in D state in the load average. With the CPUs mostly idle and 83% iowait, the bottleneck is the disk; more CPUs would not help.
Exercise
On a cloud VM running orbit-api, mpstat 1 shows %steal between 25 and 40 while traffic is normal, and the application has not changed. What is the most likely explanation?
Show answer
Answer: D. Steal is time your virtual CPU was ready to run but the hypervisor ran something else. The fix lies outside the VM: a different instance type or size, or moving off a busy host.
Exercise
free -h on orbit-db-01 shows 150Mi free, 5.1Gi buff/cache and 5.4Gi available, and vmstat shows si and so at 0. A colleague wants to add RAM. What do you tell them?
Show answer
Answer: C. Linux fills spare RAM with page cache and releases it on demand. “available” and swap activity are the signals that matter; neither shows memory pressure here.
Key takeaways
- Use a method: for each resource check utilization, saturation and errors (USE) before forming a theory.
- The first-60-seconds checklist (uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar, top) gives you the whole picture fast.
- Linux load average includes I/O wait; compare it with
nprocand readwa,st,available,awaitand retransmits to locate the bottleneck. - PSI in
/proc/pressure/and per-cgroup pressure files show how much waiting really hurts, andsystemd-cgtopblames a whole service. - strace, perf and eBPF tools answer “what exactly is it doing” when the basic tools run out.
- Fix waste first, measure again, and scale when legitimate demand keeps a resource saturated.
FAQ
What is a good load average for a Linux server?
There is no single number; compare it with the CPU count from nproc. A load roughly equal to the number of CPUs means they are fully used without much queuing; well above it means work is waiting. Because Linux includes processes waiting for disk I/O, check vmstat to see whether the load comes from CPU (r, us, sy) or from I/O (b, wa).
How do I find which process is causing high CPU or disk usage?
For CPU, sort top by CPU with P or run pidstat -u 1. For disk, run pidstat -d 1 or sudo iotop-c -oPa, which show read and write rates per process. For services and containers, sudo systemd-cgtop ranks whole cgroups, which is often quicker than chasing individual PIDs.
Why does Linux show so little free memory?
Linux uses otherwise idle RAM as page cache to speed up file access, and frees it instantly when programs need memory. Look at the available column of free -h instead of free, and at the si and so columns of vmstat for swapping. Clearing the cache by hand only makes the system slower until it fills up again.
What is iowait and is a high value bad?
wa is CPU time spent idle while at least one process waits for I/O. A high value tells you that storage (or network storage) is the bottleneck, not the CPU. Confirm with iostat -xz 1: high await and a long aqu-sz mean the disk is saturated, and pidstat -d names the process causing it.
Do I need sysstat installed on every server?
It is worth it: iostat, mpstat, pidstat and sar are the standard tools for this work, and with its data collection enabled (on Ubuntu, set ENABLED="true" in /etc/default/sysstat and run sudo systemctl enable --now sysstat), sar keeps a history so you can see what happened at 03:12 last night. In fleets with Prometheus or another monitoring system the history lives there, but the sysstat tools remain the fastest way to look live.
When should I use eBPF tools instead of top and iostat?
When the standard tools show that something is slow but not why: which files a process opens, which short-lived processes keep spawning, how disk latency is distributed, which kernel function is hot. bcc tools and bpftrace answer these with low overhead on modern kernels, but they need root and some practice, so start with the 60-second checklist and reach for eBPF afterwards.