Every incident starts with the same questions: what is running, who started it, what is it using, and how do I stop it without breaking something? “The API is slow”, “the disk is full but du disagrees”, “port 3000 is already in use”: each of these is a process problem in disguise. Knowing how to read the process table is what separates guessing from diagnosing.
In this lesson you learn how the kernel tracks running programs, read ps and top output with confidence, control jobs in your shell, send the right signal at the right time and look inside a live process through /proc. Everything works on the orbit-lab VM; WSL works too, but a container shows only its own few processes, so pstree and PID 1 look very different there.
Programs, processes and threads
A program is a file on disk, such as /usr/bin/python3. A process is that program loaded into memory and running, with its own memory, open files, user identity and current directory. Run the same program three times and you get three processes. A thread is a separate line of execution inside one process that shares its memory; a busy web server is often one process with dozens of threads.
The kernel gives every process a few labels you will use constantly:
| Label | Meaning | Why you care |
|---|---|---|
| PID | Process ID, a unique number | Every tool that acts on a process takes a PID |
| PPID | PID of the parent that started it | Tells you who launched it and who will clean up after it |
| UID | The user it runs as | Decides which files and ports it may touch |
| State | Running, sleeping, stopped, zombie… | Explains why it is slow or stuck |
| Nice | Scheduling priority hint | Decides who gets the CPU when it is contended |
The process tree and PID 1
Processes are created by other processes, so they form a tree. At the root sits PID 1, started by the kernel at boot. On every mainstream server distribution PID 1 is systemd (lesson 9). PID 1 has two special jobs: start everything else, and adopt orphaned processes whose parent died, so that someone is always there to collect their exit status.
$ pstree -p | head -8
systemd(1)-+-agetty(645)
|-cron(588)
|-sshd(702)---sshd-session(1310)---sshd-session(1369)---bash(1370)---pstree(1402)
|-systemd-journal(312)
|-systemd-logind(590)
|-systemd-resolve(511)
`-systemd-udevd(355)
Read the long branch from left to right: systemd started the SSH daemon, which started a session for your login, which started your bash shell, which started pstree itself. pstree comes from the psmisc package; install it with sudo apt install psmisc if it is missing.
Tip: Inside a container your application is PID 1. The kernel does not apply default signal actions to PID 1, so an app that never installed a SIGTERM handler ignores docker stop and gets killed after the timeout. That is why docker run --init and tiny init programs such as tini exist.
Listing processes with ps
ps takes a snapshot. Two classic forms do the same job with different column sets:
$ ps aux | head -4
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.6 22432 13256 ? Ss 09:12 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S 09:12 0:00 [kthreadd]
orbit 1544 0.3 1.1 33012 23040 ? Ss 09:40 0:02 /usr/bin/python3 -m http.server 3000 ...
$ ps -ef | grep [h]ttp.server
orbit 1544 1 0 09:40 ? 00:00:02 /usr/bin/python3 -m http.server 3000 ...
ps auxis BSD style:aall users,uuser-oriented columns,xinclude processes without a terminal (daemons). It shows CPU, memory and state.ps -efis POSIX style:-eevery process,-ffull format. It shows the PPID, whichauxdoes not.- RSS is real memory in use (KiB); VSZ is virtual address space, which is usually much larger and rarely what you want.
- Names in
[brackets]are kernel threads, not programs. - The
[h]trick stopsgrepfrom matching its own command line.pgrep(below) is cleaner.
For scripts and investigations, pick exactly the columns you need and sort:
$ ps -eo pid,ppid,user,stat,%cpu,%mem,rss,etime,cmd --sort=-%mem | head -4
PID PPID USER STAT %CPU %MEM RSS ELAPSED CMD
1544 1 orbit Ss 0.3 1.1 23040 12:31 /usr/bin/python3 -m http.server 3000 ...
702 1 root Ss 0.0 0.4 8448 40:02 sshd: /usr/sbin/sshd -D [listener] ...
312 1 root Ss 0.0 0.4 8320 40:05 /usr/lib/systemd/systemd-journald
-o chooses columns, --sort=-%mem sorts by memory descending (drop the minus for ascending), and etime shows how long each process has been running. ps -o pid,cmd -p 1544 describes a single PID, and ps -u orbit lists one user’s processes.
Live views: top and htop
top refreshes every few seconds. The header answers “is the machine busy?” and the table answers “who is busy?”:
$ top
top - 10:02:11 up 50 min, 1 user, load average: 0.31, 0.12, 0.04
Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.3 us, 0.7 sy, 0.0 ni, 96.8 id, 0.0 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 1967.4 total, 1121.6 free, 364.0 used, 632.5 buff/cache
The load averages cover 1, 5 and 15 minutes: the number of processes running or waiting (including those in state D). Compare them with nproc; a load of 2.0 on 2 cores is busy, on 16 cores it is idle. wa is time spent waiting for I/O and st is CPU stolen by the hypervisor on cloud VMs. Inside top, press P to sort by CPU, M by memory, 1 for per-core lines, c for full command lines and q to quit.
htop (sudo apt install htop) shows the same data with colours, a tree view (F5), search (F3) and mouse support. Use it interactively; use ps in scripts and runbooks because its output is stable. Lesson 19 goes deeper into load, memory and I/O.
Process states and zombies
The STAT column is a short diagnosis. The first letter is the state; extra letters add detail (s session leader, + foreground job, l multithreaded, < high priority, N low priority).
| State | Name | What it usually means |
|---|---|---|
R | Running or runnable | On a CPU or waiting for one. Many R processes means CPU pressure |
S | Interruptible sleep | Waiting for something (a request, a timer). Normal: most processes live here |
D | Uninterruptible sleep | Stuck inside the kernel, usually on disk or network storage I/O. Cannot be killed until the I/O returns |
T | Stopped | Paused by Ctrl+Z, SIGSTOP or a debugger |
Z | Zombie | Finished, but the parent has not read its exit status yet |
I | Idle | Idle kernel thread; ignore it |
A zombie uses no CPU or memory, only a slot in the process table, and you cannot kill it because it is already dead. The fix is the parent: it should call wait(). If the parent is buggy, restarting or killing it makes PID 1 adopt the zombies and reap them instantly. A handful of zombies is harmless; thousands point to a bug that can exhaust PIDs. Many processes in D usually means a slow or hung disk or NFS mount, not an application bug.
Foreground, background and jobs
Your shell can juggle several jobs. A command started normally runs in the foreground and holds the prompt; add & to run it in the background.
$ python3 -m http.server 8080 --bind 127.0.0.1 > /tmp/web.log 2>&1 &
[1] 2210
$ sleep 600
^Z
[2]+ Stopped sleep 600
$ jobs -l
[1]- 2210 Running python3 -m http.server 8080 --bind 127.0.0.1 > /tmp/web.log 2>&1 &
[2]+ 2214 Stopped sleep 600
$ bg %2
[2]+ sleep 600 &
$ fg %1
python3 -m http.server 8080 --bind 127.0.0.1 > /tmp/web.log 2>&1
^C
- Ctrl+Z sends SIGTSTP and pauses the foreground job (state
T). bg %2resumes job 2 in the background;fg %1brings job 1 back to the foreground; Ctrl+C sends SIGINT to it.jobs -llists the shell’s jobs with PIDs. Job numbers (%1) only exist inside this shell.
Background jobs belong to your shell. When your SSH session drops, the shell sends them SIGHUP and they usually die. nohup command & ignores SIGHUP and writes output to nohup.out; disown detaches a job you already started; tmux keeps a whole session alive so you can reattach later. All three are fine for a one-off migration. None of them is how you run a service: nothing restarts it after a crash or a reboot, its logs end up in a random file, and nobody else knows it exists. Services belong in systemd, which you set up in lesson 9. For a quick one-off with systemd’s benefits, sudo systemd-run --unit=reindex /usr/local/bin/some-task runs a command as a temporary unit with logs in the journal.
Signals
A signal is a small numbered message the kernel delivers to a process. Most of them a process can catch and handle; two it cannot.
| Signal | Number | Sent by | Default effect | Typical use |
|---|---|---|---|---|
| SIGTERM | 15 | kill, systemd, Docker, Kubernetes | Terminate | “Please shut down cleanly.” Always try this first |
| SIGINT | 2 | Ctrl+C | Terminate | Interrupt a foreground command |
| SIGHUP | 1 | Closed terminal, or you | Terminate | Many daemons (nginx, sshd) treat it as “reload your config” |
| SIGKILL | 9 | kill -9 | Kill, cannot be caught | Last resort when a process ignores SIGTERM |
| SIGSTOP / SIGCONT | 19 / 18 | kill, debuggers | Pause / resume | Freeze a runaway process while you investigate |
| SIGUSR1 | 10 | You | Terminate unless handled | App-specific: reopen logs, dump state |
kill, pkill, killall and pgrep
Despite the name, kill sends a signal; SIGTERM is the default.
$ pgrep -a http.server
2210 python3 -m http.server 8080 --bind 127.0.0.1
$ kill 2210
$ kill -HUP $(pidof nginx)
$ pkill -u maya -f 'report.py'
$ killall -TERM sleep
$ kill -0 2210 && echo alive || echo gone
bash: kill: (2210) - No such process
gone
pgrep -a patternprints matching PIDs with their command line;-u userlimits to a user,-fmatches the full command line instead of just the name.pkilltakes the same options aspgrepand signals every match. Run thepgrepfirst to see what you are about to hit.killall namesignals every process with that exact name (frompsmisc).kill -0sends nothing; it only checks whether the PID exists and you may signal it. Handy in scripts.
Graceful shutdown
When a well-written service receives SIGTERM it stops accepting new work, finishes requests in flight, flushes buffers and closes database connections. SIGKILL skips all of that: half-written files, lost queued jobs, stale lock files. Every orchestrator therefore uses the same pattern: SIGTERM, wait, then SIGKILL. systemd waits 90 seconds by default (TimeoutStopSec=), docker stop waits 10 and Kubernetes 30 (terminationGracePeriodSeconds). Do the same by hand: kill PID, give it some seconds, check with kill -0, and only then reach for kill -9.
Warning: kill -9 on a process in state D does nothing until its I/O completes, and on a zombie it does nothing at all. If SIGKILL “does not work”, check the state before trying harder.
Priorities with nice and renice
When processes compete for CPU, the scheduler favours those with a lower nice value. The range is -20 (most favoured) to 19 (most polite); the default is 0. Any user may make their own process nicer; only root may lower the value.
$ nice -n 15 tar -czf /var/backups/orbit/data.tar.gz /var/lib/orbit &
[1] 2301
$ ps -o pid,ni,cmd -p 2301
PID NI CMD
2301 15 tar -czf /var/backups/orbit/data.tar.gz /var/lib/orbit
$ sudo renice -n 5 -p 2301
2301 (process ID) old priority 15, new priority 5
Nice only matters when the CPU is contended, and it does nothing for disk I/O; for that, ionice -c3 command runs a job in the idle I/O class. For services, set Nice= or a hard CPUQuota= in the unit file instead (lesson 9).
Looking inside a process with /proc
Every process has a directory /proc/<pid>/ generated live by the kernel. It is the ground truth that ps, top and lsof read. Here it is for the Orbit API on orbit-web-01, where it runs as a service:
$ PID=$(pgrep -f 'http.server 3000')
$ tr '\0' ' ' < /proc/$PID/cmdline; echo
/usr/bin/python3 -m http.server 3000 --bind 127.0.0.1 --directory /srv/orbit/public
$ grep -E 'State|PPid|Threads|VmRSS' /proc/$PID/status
State: S (sleeping)
PPid: 1
Threads: 1
VmRSS: 23040 kB
$ sudo tr '\0' '\n' < /proc/$PID/environ | grep ORBIT
ORBIT_ENV=lab
$ sudo ls -l /proc/$PID/fd
lr-x------ 1 orbit orbit 64 Nov 3 10:20 0 -> /dev/null
lrwx------ 1 orbit orbit 64 Nov 3 10:20 1 -> socket:[21877]
lrwx------ 1 orbit orbit 64 Nov 3 10:20 3 -> socket:[22014]
$ sudo readlink /proc/$PID/cwd
/srv/orbit
cmdlineandenvironseparate entries with NUL bytes, hencetr.environshows the environment the process started with, and only its owner or root can read it, because it often holds secrets.statusis a readable summary: state, parent, threads, memory.fd/lists every open file descriptor: files, sockets, pipes.cwdandexepoint to the working directory and the binary.limitsshows the resource limits in force for that process.
Who holds this file or port?
lsof (list open files) answers two classic questions. Install it with sudo apt install lsof if needed.
$ sudo lsof -i :3000
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3 1544 orbit 3u IPv4 22014 0t0 TCP localhost:3000 (LISTEN)
$ sudo ss -ltnp 'sport = :3000'
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 5 127.0.0.1:3000 0.0.0.0:* users:(("python3",pid=1544,fd=3))
$ sudo lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1544 orbit 4w REG 252,1 2147483648 0 51234 /var/log/orbit/api.log (deleted)
The last command explains a famous mystery: someone deleted a 2 GB log file, df still shows the disk full, and du cannot find the space. The file is gone from the directory but a process still has it open, so the kernel cannot free the blocks. Restart (or signal to reopen logs) the process holding it and the space comes back. fuser -v 3000/tcp is a shorter alternative for ports.
Resource limits with ulimit
The kernel caps what each process may use: open files, processes, core dump size. ulimit -n shows your shell’s open-file limit (often 1024 soft) and ulimit -a shows them all. A busy proxy or database that logs “Too many open files” has hit this limit. For login sessions the limits live in /etc/security/limits.conf; for services they live in the unit file (LimitNOFILE=65536), because systemd services do not read limits.conf. Check what a running process really got with cat /proc/<pid>/limits.
Red Hat and Alpine: Rocky and RHEL have the same procps tools, so ps, top, pgrep and /proc behave identically. Alpine’s BusyBox ps ignores most options (ps aux prints a short fixed format); install procps with apk add procps to get the full version, and psmisc for pstree and killall.
Practice lab
Before Orbit had a proper service, a developer started the API by hand on orbit-lab and went home. Your job is to find it, inspect it and stop it cleanly. Set the scene first with sudo mkdir -p /srv/orbit/public and nohup python3 -m http.server 3000 --bind 127.0.0.1 --directory /srv/orbit/public > /tmp/api.log 2>&1 &, then log out and back in.
- Find the process, its PID, its parent PID and its user. Why is the parent what it is?
- Show its full command line and working directory from
/proc, and confirm which port it listens on. - Lower its priority to nice 10 and verify the change.
- Stop it gracefully, then prove that it is gone and the port is free.
Show a solution
pgrep -af http.server gives the PID, and ps -o pid,ppid,user,stat,cmd -p <PID> shows a PPID of 1 (or a systemd --user process): the shell that launched it exited when you logged out, so the process was orphaned and adopted. tr '\0' ' ' < /proc/<PID>/cmdline prints the full command, readlink /proc/<PID>/cwd the directory, and sudo ss -ltnp 'sport = :3000' or sudo lsof -i :3000 confirm the listening socket on 127.0.0.1:3000. renice -n 10 -p <PID> lowers the priority (no sudo needed, it is your process), checked with ps -o pid,ni -p <PID>. Finally kill <PID> sends SIGTERM; kill -0 <PID> then reports “No such process” and ss -ltn 'sport = :3000' prints only the header line.
Common mistakes
- Reaching for
kill -9first. It skips cleanup and can corrupt data. SIGTERM, wait, then SIGKILL. - Running
pkill -fwithout a dry run.-f apialso matchesrapid-backupand your own editor. Runpgrep -afwith the same pattern first. - Running services with
nohupor intmux. They will not survive a reboot or crash. Write a systemd unit. - Trying to kill zombies. Fix or restart the parent instead.
Exercise
The Orbit API on orbit-web-01 still runs after kill 1544. ps -o stat -p 1544 shows S. What is the sensible next step?
Show answer
Answer: B. SIGTERM asks the process to finish its work and exit, which can take a moment. Escalate to SIGKILL only after a grace period, and remember that a process in D state cannot die until its I/O returns.
Exercise
df -h says /var on orbit-web-02 is 100% full, but du -sh /var/* adds up to far less. What is the most likely cause and the tool that proves it?
Show answer
Answer: C. When a file is deleted while open, its directory entry disappears (so du cannot see it) but the blocks stay allocated until the process closes it. lsof +L1 lists open files with zero links; restart or signal that process.
Exercise
Leo starts a two-hour data migration over SSH with ./migrate.sh & and his Wi-Fi drops. The migration stops too. What would have kept it running?
Show answer
Answer: C. Background jobs belong to the login shell, which sends them SIGHUP when the session ends. tmux keeps the session alive, nohup ignores SIGHUP, and systemd-run hands the process to systemd entirely.
Key takeaways
- A process is a running program with a PID, a parent, a user and a state; PID 1 (systemd) starts everything and adopts orphans.
- Use
ps -eo ... --sortfor precise snapshots,top/htopfor live views, and read the state letter before acting. - Signals are messages: SIGTERM for a clean stop, SIGHUP often for reload, SIGKILL only as a last resort.
- Job control,
nohupandtmuxare for interactive work; services belong in systemd. /proc/<pid>shows the truth about a process, andlsoforss -ptell you who holds a file or a port.
FAQ
What is the difference between kill and kill -9?
kill PID sends SIGTERM (15), which the program can catch to finish requests, save state and exit cleanly. kill -9 PID sends SIGKILL, which the kernel enforces immediately without giving the program any chance to clean up. Use SIGTERM first and SIGKILL only when a process ignores it after a reasonable wait.
How do I find which process is using a port?
Run sudo ss -ltnp 'sport = :3000' (replace the port), which shows the process name and PID in the last column. sudo lsof -i :3000 and sudo fuser -v 3000/tcp give the same answer. Without sudo you only see process details for your own processes.
How do I get rid of zombie processes?
You cannot kill a zombie because it has already exited; it is waiting for its parent to read its exit status. Find the parent with ps -o ppid= -p <zombie PID>, then fix or restart that parent. Once the parent exits, PID 1 adopts the zombies and reaps them immediately.
What does the load average actually mean?
It is the average number of processes that were running, waiting for a CPU, or in uninterruptible sleep (usually I/O) over the last 1, 5 and 15 minutes. Divide it by the number of cores from nproc: below 1 per core is comfortable, sustained values well above it mean work is queuing. A high load with idle CPUs usually means processes stuck on disk or network storage.
Should I use nohup, disown, screen or tmux?
For an interactive task you want to watch later, use tmux (or screen): you can detach and reattach. nohup is fine for a quick fire-and-forget command, and disown rescues a job you already started without it. For anything that should run permanently or restart after failure, write a systemd service instead.
Why can’t I kill a process even as root?
Usually it is in state D, waiting for I/O inside the kernel, often on a hung NFS mount or failing disk; it dies once the I/O completes or times out. Otherwise it is a zombie that is already dead, or it is restarted instantly by systemd (check systemctl status <PID>), so stop the service rather than the process.