In the last lesson you started the Orbit API by hand and watched it die with your SSH session. Real services need someone to start them at boot, restart them when they crash, collect their logs, run them as the right user and stop them politely. On every mainstream server distribution in 2026 that someone is systemd. Docker’s daemon, nginx, PostgreSQL, the SSH server and the Kubernetes kubelet are all systemd services, so this is a skill you use every single day.
In this lesson you learn the vocabulary of units, drive services with systemctl, write a unit file for Orbit, add dependencies, overrides, hardening and resource limits, and watch systemd bring the API back after a crash. You need the orbit-lab VM (or WSL with systemd): containers do not run systemd.
What an init system does
When the kernel finishes booting it starts exactly one program, PID 1, and from then on PID 1 is responsible for everything else. On Ubuntu, Debian, RHEL and their relatives that program is systemd. Its jobs:
- Boot the system in the right order: mount filesystems, bring up the network, start services.
- Supervise services: track every process they spawn, restart them on failure, stop them cleanly.
- Collect logs from every service into the journal (read with
journalctl). - Enforce limits and isolation through cgroups v2 and kernel sandboxing features.
It replaced SysV init, where shell scripts in /etc/init.d ran one after another, with short declarative files, parallel startup and exact knowledge of which processes belong to which service.
Units and unit types
Everything systemd manages is a unit, described by a small text file. The suffix tells you the type:
| Type | Suffix | What it represents | Example |
|---|---|---|---|
| Service | .service | A program to run and supervise | nginx.service, orbit-api.service |
| Socket | .socket | A listening socket that starts a service on the first connection | ssh.socket on Ubuntu |
| Timer | .timer | A schedule that starts a service | logrotate.timer, see lesson 15 |
| Target | .target | A named group of units, a milestone in boot | multi-user.target, network-online.target |
| Mount | .mount | A mounted filesystem, often generated from /etc/fstab | var-lib-orbit.mount |
Unit files live in three places, and the first one wins: /etc/systemd/system/ (yours, the administrator’s), /run/systemd/system/ (generated at runtime) and /usr/lib/systemd/system/ (installed by packages). Never edit the package copies; an upgrade overwrites them. Targets play the role of the old runlevels: a server normally boots into multi-user.target, and a desktop into graphical.target.
Driving services with systemctl
systemctl is the remote control for systemd. Most actions need sudo; read-only queries do not.
$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
Active: active (running) since Tue 2026-11-03 09:12:40 UTC; 1h 3min ago
TriggeredBy: ● ssh.socket
Main PID: 701 (sshd)
CGroup: /system.slice/ssh.service
└─701 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
Read it top down: Loaded shows the unit file path and whether it starts at boot; Active shows the current state and since when; CGroup lists every process that belongs to the service. The last few log lines follow at the bottom. Here ssh.service is disabled but running anyway, because ssh.socket starts it on the first connection.
| Command | What it does |
|---|---|
sudo systemctl start orbit-api | Start now (does not survive a reboot) |
sudo systemctl stop orbit-api | Stop now: SIGTERM, wait, then SIGKILL |
sudo systemctl restart orbit-api | Stop and start; brief downtime |
sudo systemctl reload nginx | Ask the service to reread its config without stopping (only if the unit supports it) |
sudo systemctl enable --now orbit-api | Start at every boot and start right now |
sudo systemctl disable --now orbit-api | Do not start at boot, and stop now |
systemctl is-active orbit-api | Prints active, inactive or failed; exit code 0 only if active |
systemctl is-enabled orbit-api | Prints enabled, disabled, static or masked |
systemctl list-units --failed | Every unit that is currently broken |
systemctl cat orbit-api | The unit file plus all overrides, as systemd sees it |
Start and enable are independent. start affects now, enable affects the next boot. Forgetting enable is the classic “it worked until the maintenance reboot” incident; enable --now does both. is-active and is-enabled are made for scripts and health checks: systemctl is-active --quiet orbit-api || echo "Orbit API is down".
Writing the Orbit API service
Time to give Orbit a proper home. In the lab the API is simulated with Python’s built-in web server, so you do not need Node.js.
If you skipped ahead, create the service account and folders first. If lesson 7 already created the orbit user, skip the first line; the rest is safe to run again:
$ sudo useradd --system --user-group --home-dir /srv/orbit --shell /usr/sbin/nologin orbit
$ sudo mkdir -p /srv/orbit/public /var/lib/orbit /var/log/orbit /etc/orbit
$ echo '<h1>Orbit API (lab)</h1>' | sudo tee /srv/orbit/public/index.html
$ printf 'ORBIT_ENV=lab\nORBIT_PORT=3000\n' | sudo tee /etc/orbit/orbit.env
$ sudo chown root:orbit /etc/orbit/orbit.env && sudo chmod 640 /etc/orbit/orbit.env && sudo chown orbit:orbit /var/lib/orbit /var/log/orbit
Now create /etc/systemd/system/orbit-api.service with sudo nano (or your editor):
[Unit]
Description=Orbit API (lab simulation)
After=network-online.target
Wants=network-online.target
[Service]
Type=exec
User=orbit
Group=orbit
WorkingDirectory=/srv/orbit
EnvironmentFile=/etc/orbit/orbit.env
ExecStart=/usr/bin/python3 -m http.server 3000 --bind 127.0.0.1 --directory /srv/orbit/public
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
The three sections have clear roles:
- [Unit] describes the unit and its relationships: a human-readable
Descriptionand ordering against other units. - [Service] says how to run it.
Type=execmeans “the service is up once the program has been executed”, and unlike the defaultsimpleit reports a missing binary as a failed start.User=andGroup=drop root privileges;WorkingDirectory=sets the current directory;EnvironmentFile=loadsKEY=valuelines as environment variables, read by systemd as root so the file can stay mode 640.ExecStart=needs an absolute path.Restart=on-failurerestarts after a non-zero exit, a crash signal or a timeout, waitingRestartSec=5seconds between attempts. - [Install] is only used by
enable:WantedBy=multi-user.targetmeans “start me whenever the normal multi-user system starts”.
Tell systemd that a unit file changed, then start and enable it:
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now orbit-api
Created symlink '/etc/systemd/system/multi-user.target.wants/orbit-api.service' → '/etc/systemd/system/orbit-api.service'.
$ systemctl is-active orbit-api
active
$ curl -s http://127.0.0.1:3000/
<h1>Orbit API (lab)</h1>
daemon-reload makes systemd reread unit files from disk. Skip it and systemd keeps using the old version, and systemctl status warns that the unit file changed on disk. enable simply creates that symlink in multi-user.target.wants/: there is no hidden magic.
Dependencies and ordering
systemd keeps two questions apart: does A need B? and does A start after B?
| Directive | Meaning |
|---|---|
Wants=B | Start B too when A starts; A keeps running if B fails. The usual choice |
Requires=B | Start B too; if B fails to start or is stopped, A is stopped as well |
After=B | Ordering only: if both are starting, start A once B is up. Pulls nothing in |
Before=B | The reverse ordering |
That is why the Orbit unit has both lines for the network: Wants=network-online.target pulls in “the network is actually configured”, and After= makes the API wait for it. network.target only means the network stack is up, which is too early for a service that connects to a database on another host. On orbit-web-01, where the API needs PostgreSQL on the same machine, you would add Wants=postgresql.service and After=postgresql.service. Prefer Wants= over Requires=; a strict dependency turns one restart into a cascade.
Overrides with systemctl edit
When you want to change a unit, especially one installed by a package, do not edit the file itself. Create a drop-in:
$ sudo systemctl edit orbit-api
An editor opens an empty override.conf with the original file shown as comments. Add only what you change, save, and systemd writes /etc/systemd/system/orbit-api.service.d/override.conf and reloads for you:
[Service]
Environment=ORBIT_LOG_LEVEL=debug
RestartSec=2
systemctl cat orbit-api shows the original followed by the drop-in, and sudo systemctl revert orbit-api deletes all drop-ins. Lists such as ExecStart= add up rather than replace, so to replace a command first clear it with an empty assignment: a line ExecStart= followed by the new ExecStart=.... Restart the service afterwards so the running process picks up the change.
Logs with journalctl
Whatever the service writes to standard output or standard error lands in the journal, tagged with the unit name:
$ journalctl -u orbit-api -n 5 --no-pager
Nov 03 10:42:17 orbit-lab systemd[1]: Starting orbit-api.service - Orbit API (lab simulation)...
Nov 03 10:42:17 orbit-lab systemd[1]: Started orbit-api.service - Orbit API (lab simulation).
Nov 03 10:42:31 orbit-lab python3[2875]: 127.0.0.1 - - [03/Nov/2026 10:42:31] "GET / HTTP/1.1" 200 -
$ journalctl -u orbit-api --since "15 min ago" -p warning
-u filters by unit, -n 5 shows the last five lines, -f follows live like tail -f, --since takes times like "2026-11-03 10:00" or "1 hour ago", and -p warning shows only warnings and worse. Lesson 18 covers the journal in depth.
When a service will not start
systemctl status usually tells you why in the Main PID or Process: line, as a code and a name:
| Status | Meaning | Usual fix |
|---|---|---|
203/EXEC | The program could not be executed | Wrong ExecStart= path, missing binary, no execute permission |
217/USER | The User= does not exist | Create the user or fix the typo |
200/CHDIR | WorkingDirectory= missing or not accessible | Create it, check permissions |
226/NAMESPACE | Sandboxing could not be set up | A path in ReadWritePaths= does not exist |
1/FAILURE | The program itself exited with an error | Read the journal for the program’s own message |
systemd-analyze verify /etc/systemd/system/orbit-api.service checks a unit for typos before you start it.
Watch it heal after a crash
Restart=on-failure is worth seeing in action. Kill the API the hard way and watch:
$ sudo systemctl kill --signal=SIGKILL orbit-api
$ systemctl status orbit-api | grep Active
Active: activating (auto-restart) (Result: signal) since Tue 2026-11-03 10:51:02 UTC; 2s ago
$ journalctl -u orbit-api -n 3 --no-pager
Nov 03 10:51:02 orbit-lab systemd[1]: orbit-api.service: Main process exited, code=killed, status=9/KILL
Nov 03 10:51:02 orbit-lab systemd[1]: orbit-api.service: Failed with result 'signal'.
Nov 03 10:51:07 orbit-lab systemd[1]: orbit-api.service: Scheduled restart job, restart counter is at 1.
Five seconds later the API is back with a new PID. Two details matter. First, a plain SIGTERM counts as a clean exit, so sudo kill <PID> stops the API and on-failure does not restart it; use Restart=always if you want that too. Second, systemd gives up after five starts within ten seconds by default and marks the unit failed, so a crash loop does not spin forever. Tune that with StartLimitBurst= and StartLimitIntervalSec= in [Unit], and clear the counter with sudo systemctl reset-failed orbit-api.
Hardening the service
A service running as orbit already cannot touch root’s files, but it can still read most of the system and write to /tmp. systemd can fence it in further with a few lines in [Service]:
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/orbit /var/log/orbit
NoNewPrivileges=yes: the process and its children can never gain privileges, even through setuid programs such assudo.ProtectSystem=strict: the whole filesystem becomes read-only for the service, apart from the kernel’s own trees/dev,/procand/sys, which have separate switches.ReadWritePaths=: the exceptions the API really needs to write. They must exist, or the start fails with226/NAMESPACE. (StateDirectory=orbitandLogsDirectory=orbitcan create and own them for you.)ProtectHome=yeshides/home,/rootand/run/user;PrivateTmp=yesgives the service its own empty/tmp, so it cannot read or plant files in the shared one.
Measure the effect with systemd-analyze security, which scores a service’s exposure from 0 (locked down) to 10:
$ systemd-analyze security orbit-api.service | tail -1
→ Overall exposure level for orbit-api.service: 9.2 UNSAFE
Your exact number may differ. Add the block above with sudo systemctl edit orbit-api, restart, and run the check again: the score drops noticeably. The tool lists dozens of further options (ProtectKernelModules=, RestrictAddressFamilies=, CapabilityBoundingSet=, SystemCallFilter=); add them one at a time and test after each, because the wrong one can break a real application in subtle ways.
Resource limits with cgroups
Every service runs in its own cgroup (/system.slice/orbit-api.service in the status output), so systemd can cap it without touching the program:
[Service]
MemoryMax=256M
CPUQuota=50%
TasksMax=64
MemoryMax= is a hard ceiling: if the API exceeds it, the kernel’s OOM killer kills it inside its cgroup, and Restart= brings it back, instead of the whole server running out of memory. CPUQuota=50% allows half of one core, and TasksMax= caps processes and threads, a cheap defence against fork bombs. Check the result with systemctl show orbit-api -p MemoryMax -p CPUQuotaPerSecUSec, and watch live usage per service with systemd-cgtop.
How long did boot take?
$ systemd-analyze blame | head -3
4.812s cloud-init.service
2.130s snapd.seeded.service
1.004s systemd-networkd-wait-online.service
Plain systemd-analyze prints the total boot time; blame sorts units by how long they took to start; systemd-analyze critical-chain orbit-api.service shows which chain of dependencies actually delayed a specific unit. Units that start in parallel do not add up, so the critical chain is usually the more honest view.
Red Hat and Alpine: Rocky and RHEL use the same systemd and every command above, with package units in /usr/lib/systemd/system. SELinux adds a layer: a service that fails with unexplained “permission denied” may be blocked by policy, so check sudo ausearch -m avc -ts recent and restore file labels with sudo restorecon -Rv /srv/orbit. Alpine does not use systemd at all; it uses OpenRC, with rc-service nginx start and rc-update add nginx default and init scripts in /etc/init.d.
Practice lab
Orbit’s API must survive crashes and reboots on orbit-lab, and the security team wants it locked down. Use the unit file from this lesson.
- Install the unit, start it, enable it at boot and prove that
curl http://127.0.0.1:3000/answers. - Kill the main process with SIGKILL and confirm it comes back with a new PID. Then stop it with plain
killand explain why it stays down. - Using
systemctl edit, add the hardening options and a 128 MB memory limit, and comparesystemd-analyze securityscores before and after. - Show the last ten log lines of the service and list any failed units on the system.
Show a solution
After creating the file, sudo systemctl daemon-reload and sudo systemctl enable --now orbit-api start and enable it, and curl returns the index page. systemctl show -p MainPID --value orbit-api gives the PID; sudo kill -9 on it (or sudo systemctl kill --signal=SIGKILL orbit-api) leads to “Scheduled restart job” in the journal and a new MainPID after five seconds. sudo kill <PID> sends SIGTERM, which systemd treats as a clean exit, so Restart=on-failure leaves it inactive until you run sudo systemctl start orbit-api. sudo systemctl edit orbit-api with the five hardening lines plus MemoryMax=128M, followed by sudo systemctl restart orbit-api, lowers the score shown by systemd-analyze security orbit-api.service, and systemctl show orbit-api -p MemoryMax prints 134217728. Finally journalctl -u orbit-api -n 10 and systemctl list-units --failed answer the last task.
Common mistakes
- Editing a unit and forgetting
daemon-reload. systemd keeps running the old definition. - Starting without enabling.
startruns the service now, but it is gone after the next reboot. Useenable --now. - Relative paths or shell syntax in
ExecStart=. There is no shell: no~, no pipes, no&&. Wrap complex logic in a script or use/bin/sh -c '...'.
Exercise
Maya adds MemoryMax=256M to /etc/systemd/system/orbit-api.service and runs sudo systemctl restart orbit-api, but systemctl show orbit-api -p MemoryMax still prints infinity. What did she miss?
Show answer
Answer: B. systemd caches unit files. After editing a file directly, run daemon-reload and then restart. systemctl edit does the reload for you, which is one more reason to prefer it.
Exercise
After the monthly reboot of orbit-web-02, nginx is up but the Orbit API is not, although Leo started it last week and it ran fine. systemctl is-enabled orbit-api prints disabled. What fixes it for good?
Show answer
Answer: C. start only affects the current boot. enable creates the symlink in multi-user.target.wants so the unit starts at every boot, and --now starts it immediately as well.
Exercise
The Orbit API connects to PostgreSQL on orbit-db-01 at startup and sometimes fails right after boot with “network unreachable”. Its unit has After=network.target. What is the best change?
Show answer
Answer: C. network.target only means the network stack exists. network-online.target means interfaces are configured; Wants pulls it in and After orders the API behind it. A sleep only hides the race.
Key takeaways
- systemd is PID 1: it boots the system, supervises services, collects logs and enforces limits through cgroups.
- Your own units go in
/etc/systemd/system; change package units withsystemctl editdrop-ins, and rundaemon-reloadafter editing files by hand. startis for now,enableis for every boot;enable --nowdoes both.Restart=on-failure,User=,EnvironmentFile=and correctWants=/After=turn a script into a reliable service.- A few hardening lines and
MemoryMax=/CPUQuota=make a service safer; measure withsystemd-analyze security. - When something fails, read
systemctl statusandjournalctl -ubefore changing anything.
FAQ
What is the difference between systemctl start and systemctl enable?
start runs the service now but has no effect on the next boot. enable creates a symlink so the service starts automatically at boot, but does not start it now. Use sudo systemctl enable --now name to do both at once, and disable --now to undo both.
Why does my change to a unit file not take effect?
systemd caches unit definitions. After editing a file in /etc/systemd/system, run sudo systemctl daemon-reload and then restart the service. Also check systemctl cat name: a drop-in in a .service.d directory may be overriding the value you changed.
How do I see the logs of a systemd service?
Run journalctl -u name for everything, journalctl -u name -n 50 for the last fifty lines, journalctl -u name -f to follow live and journalctl -u name -b for the current boot only. systemctl status name also shows the last few lines.
Which Type= should I use for my service?
Use exec (or the default simple) for a program that stays in the foreground, which is how modern apps behave. Use notify if the program supports sd_notify and should only count as started when it is ready. Use oneshot for scripts that run and exit, and avoid forking unless an old daemon forces it.
Is it better to run a service with Restart=always or Restart=on-failure?
on-failure restarts after crashes, error exits and timeouts, but respects a deliberate clean stop, which suits most apps. always also restarts after a clean exit, which is useful for workers that exit on purpose to reload. In both cases the start limit stops endless crash loops, so check systemctl status if a service ends up failed.