Lesson 09 intermediate

systemd Services

How systemd starts and supervises everything on a modern Linux server, how to manage units with systemctl, and how to write, harden, limit and troubleshoot your own service unit for the Orbit API.

In the last lesson you started the Orbit API by hand and watched it die with your SSH session. Real services need someone to start them at boot, restart them when they crash, collect their logs, run them as the right user and stop them politely. On every mainstream server distribution in 2026 that someone is systemd. Docker’s daemon, nginx, PostgreSQL, the SSH server and the Kubernetes kubelet are all systemd services, so this is a skill you use every single day.

In this lesson you learn the vocabulary of units, drive services with systemctl, write a unit file for Orbit, add dependencies, overrides, hardening and resource limits, and watch systemd bring the API back after a crash. You need the orbit-lab VM (or WSL with systemd): containers do not run systemd.

What an init system does

When the kernel finishes booting it starts exactly one program, PID 1, and from then on PID 1 is responsible for everything else. On Ubuntu, Debian, RHEL and their relatives that program is systemd. Its jobs:

  • Boot the system in the right order: mount filesystems, bring up the network, start services.
  • Supervise services: track every process they spawn, restart them on failure, stop them cleanly.
  • Collect logs from every service into the journal (read with journalctl).
  • Enforce limits and isolation through cgroups v2 and kernel sandboxing features.

It replaced SysV init, where shell scripts in /etc/init.d ran one after another, with short declarative files, parallel startup and exact knowledge of which processes belong to which service.

Units and unit types

Everything systemd manages is a unit, described by a small text file. The suffix tells you the type:

TypeSuffixWhat it representsExample
Service.serviceA program to run and supervisenginx.service, orbit-api.service
Socket.socketA listening socket that starts a service on the first connectionssh.socket on Ubuntu
Timer.timerA schedule that starts a servicelogrotate.timer, see lesson 15
Target.targetA named group of units, a milestone in bootmulti-user.target, network-online.target
Mount.mountA mounted filesystem, often generated from /etc/fstabvar-lib-orbit.mount

Unit files live in three places, and the first one wins: /etc/systemd/system/ (yours, the administrator’s), /run/systemd/system/ (generated at runtime) and /usr/lib/systemd/system/ (installed by packages). Never edit the package copies; an upgrade overwrites them. Targets play the role of the old runlevels: a server normally boots into multi-user.target, and a desktop into graphical.target.

Driving services with systemctl

systemctl is the remote control for systemd. Most actions need sudo; read-only queries do not.

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/usr/lib/systemd/system/ssh.service; disabled; preset: enabled)
     Active: active (running) since Tue 2026-11-03 09:12:40 UTC; 1h 3min ago
TriggeredBy: ● ssh.socket
   Main PID: 701 (sshd)
     CGroup: /system.slice/ssh.service
             └─701 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

Read it top down: Loaded shows the unit file path and whether it starts at boot; Active shows the current state and since when; CGroup lists every process that belongs to the service. The last few log lines follow at the bottom. Here ssh.service is disabled but running anyway, because ssh.socket starts it on the first connection.

CommandWhat it does
sudo systemctl start orbit-apiStart now (does not survive a reboot)
sudo systemctl stop orbit-apiStop now: SIGTERM, wait, then SIGKILL
sudo systemctl restart orbit-apiStop and start; brief downtime
sudo systemctl reload nginxAsk the service to reread its config without stopping (only if the unit supports it)
sudo systemctl enable --now orbit-apiStart at every boot and start right now
sudo systemctl disable --now orbit-apiDo not start at boot, and stop now
systemctl is-active orbit-apiPrints active, inactive or failed; exit code 0 only if active
systemctl is-enabled orbit-apiPrints enabled, disabled, static or masked
systemctl list-units --failedEvery unit that is currently broken
systemctl cat orbit-apiThe unit file plus all overrides, as systemd sees it

Start and enable are independent. start affects now, enable affects the next boot. Forgetting enable is the classic “it worked until the maintenance reboot” incident; enable --now does both. is-active and is-enabled are made for scripts and health checks: systemctl is-active --quiet orbit-api || echo "Orbit API is down".

Writing the Orbit API service

Time to give Orbit a proper home. In the lab the API is simulated with Python’s built-in web server, so you do not need Node.js.

If you skipped ahead, create the service account and folders first. If lesson 7 already created the orbit user, skip the first line; the rest is safe to run again:

$ sudo useradd --system --user-group --home-dir /srv/orbit --shell /usr/sbin/nologin orbit
$ sudo mkdir -p /srv/orbit/public /var/lib/orbit /var/log/orbit /etc/orbit
$ echo '<h1>Orbit API (lab)</h1>' | sudo tee /srv/orbit/public/index.html
$ printf 'ORBIT_ENV=lab\nORBIT_PORT=3000\n' | sudo tee /etc/orbit/orbit.env
$ sudo chown root:orbit /etc/orbit/orbit.env && sudo chmod 640 /etc/orbit/orbit.env && sudo chown orbit:orbit /var/lib/orbit /var/log/orbit

Now create /etc/systemd/system/orbit-api.service with sudo nano (or your editor):

[Unit]
Description=Orbit API (lab simulation)
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=orbit
Group=orbit
WorkingDirectory=/srv/orbit
EnvironmentFile=/etc/orbit/orbit.env
ExecStart=/usr/bin/python3 -m http.server 3000 --bind 127.0.0.1 --directory /srv/orbit/public
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

The three sections have clear roles:

  • [Unit] describes the unit and its relationships: a human-readable Description and ordering against other units.
  • [Service] says how to run it. Type=exec means “the service is up once the program has been executed”, and unlike the default simple it reports a missing binary as a failed start. User= and Group= drop root privileges; WorkingDirectory= sets the current directory; EnvironmentFile= loads KEY=value lines as environment variables, read by systemd as root so the file can stay mode 640. ExecStart= needs an absolute path. Restart=on-failure restarts after a non-zero exit, a crash signal or a timeout, waiting RestartSec=5 seconds between attempts.
  • [Install] is only used by enable: WantedBy=multi-user.target means “start me whenever the normal multi-user system starts”.

Tell systemd that a unit file changed, then start and enable it:

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now orbit-api
Created symlink '/etc/systemd/system/multi-user.target.wants/orbit-api.service' → '/etc/systemd/system/orbit-api.service'.
$ systemctl is-active orbit-api
active
$ curl -s http://127.0.0.1:3000/
<h1>Orbit API (lab)</h1>

daemon-reload makes systemd reread unit files from disk. Skip it and systemd keeps using the old version, and systemctl status warns that the unit file changed on disk. enable simply creates that symlink in multi-user.target.wants/: there is no hidden magic.

Dependencies and ordering

systemd keeps two questions apart: does A need B? and does A start after B?

DirectiveMeaning
Wants=BStart B too when A starts; A keeps running if B fails. The usual choice
Requires=BStart B too; if B fails to start or is stopped, A is stopped as well
After=BOrdering only: if both are starting, start A once B is up. Pulls nothing in
Before=BThe reverse ordering

That is why the Orbit unit has both lines for the network: Wants=network-online.target pulls in “the network is actually configured”, and After= makes the API wait for it. network.target only means the network stack is up, which is too early for a service that connects to a database on another host. On orbit-web-01, where the API needs PostgreSQL on the same machine, you would add Wants=postgresql.service and After=postgresql.service. Prefer Wants= over Requires=; a strict dependency turns one restart into a cascade.

Overrides with systemctl edit

When you want to change a unit, especially one installed by a package, do not edit the file itself. Create a drop-in:

$ sudo systemctl edit orbit-api

An editor opens an empty override.conf with the original file shown as comments. Add only what you change, save, and systemd writes /etc/systemd/system/orbit-api.service.d/override.conf and reloads for you:

[Service]
Environment=ORBIT_LOG_LEVEL=debug
RestartSec=2

systemctl cat orbit-api shows the original followed by the drop-in, and sudo systemctl revert orbit-api deletes all drop-ins. Lists such as ExecStart= add up rather than replace, so to replace a command first clear it with an empty assignment: a line ExecStart= followed by the new ExecStart=.... Restart the service afterwards so the running process picks up the change.

Logs with journalctl

Whatever the service writes to standard output or standard error lands in the journal, tagged with the unit name:

$ journalctl -u orbit-api -n 5 --no-pager
Nov 03 10:42:17 orbit-lab systemd[1]: Starting orbit-api.service - Orbit API (lab simulation)...
Nov 03 10:42:17 orbit-lab systemd[1]: Started orbit-api.service - Orbit API (lab simulation).
Nov 03 10:42:31 orbit-lab python3[2875]: 127.0.0.1 - - [03/Nov/2026 10:42:31] "GET / HTTP/1.1" 200 -
$ journalctl -u orbit-api --since "15 min ago" -p warning

-u filters by unit, -n 5 shows the last five lines, -f follows live like tail -f, --since takes times like "2026-11-03 10:00" or "1 hour ago", and -p warning shows only warnings and worse. Lesson 18 covers the journal in depth.

When a service will not start

systemctl status usually tells you why in the Main PID or Process: line, as a code and a name:

StatusMeaningUsual fix
203/EXECThe program could not be executedWrong ExecStart= path, missing binary, no execute permission
217/USERThe User= does not existCreate the user or fix the typo
200/CHDIRWorkingDirectory= missing or not accessibleCreate it, check permissions
226/NAMESPACESandboxing could not be set upA path in ReadWritePaths= does not exist
1/FAILUREThe program itself exited with an errorRead the journal for the program’s own message

systemd-analyze verify /etc/systemd/system/orbit-api.service checks a unit for typos before you start it.

Watch it heal after a crash

Restart=on-failure is worth seeing in action. Kill the API the hard way and watch:

$ sudo systemctl kill --signal=SIGKILL orbit-api
$ systemctl status orbit-api | grep Active
     Active: activating (auto-restart) (Result: signal) since Tue 2026-11-03 10:51:02 UTC; 2s ago
$ journalctl -u orbit-api -n 3 --no-pager
Nov 03 10:51:02 orbit-lab systemd[1]: orbit-api.service: Main process exited, code=killed, status=9/KILL
Nov 03 10:51:02 orbit-lab systemd[1]: orbit-api.service: Failed with result 'signal'.
Nov 03 10:51:07 orbit-lab systemd[1]: orbit-api.service: Scheduled restart job, restart counter is at 1.

Five seconds later the API is back with a new PID. Two details matter. First, a plain SIGTERM counts as a clean exit, so sudo kill <PID> stops the API and on-failure does not restart it; use Restart=always if you want that too. Second, systemd gives up after five starts within ten seconds by default and marks the unit failed, so a crash loop does not spin forever. Tune that with StartLimitBurst= and StartLimitIntervalSec= in [Unit], and clear the counter with sudo systemctl reset-failed orbit-api.

Hardening the service

A service running as orbit already cannot touch root’s files, but it can still read most of the system and write to /tmp. systemd can fence it in further with a few lines in [Service]:

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/orbit /var/log/orbit
  • NoNewPrivileges=yes: the process and its children can never gain privileges, even through setuid programs such as sudo.
  • ProtectSystem=strict: the whole filesystem becomes read-only for the service, apart from the kernel’s own trees /dev, /proc and /sys, which have separate switches.
  • ReadWritePaths=: the exceptions the API really needs to write. They must exist, or the start fails with 226/NAMESPACE. (StateDirectory=orbit and LogsDirectory=orbit can create and own them for you.)
  • ProtectHome=yes hides /home, /root and /run/user; PrivateTmp=yes gives the service its own empty /tmp, so it cannot read or plant files in the shared one.

Measure the effect with systemd-analyze security, which scores a service’s exposure from 0 (locked down) to 10:

$ systemd-analyze security orbit-api.service | tail -1
→ Overall exposure level for orbit-api.service: 9.2 UNSAFE

Your exact number may differ. Add the block above with sudo systemctl edit orbit-api, restart, and run the check again: the score drops noticeably. The tool lists dozens of further options (ProtectKernelModules=, RestrictAddressFamilies=, CapabilityBoundingSet=, SystemCallFilter=); add them one at a time and test after each, because the wrong one can break a real application in subtle ways.

Resource limits with cgroups

Every service runs in its own cgroup (/system.slice/orbit-api.service in the status output), so systemd can cap it without touching the program:

[Service]
MemoryMax=256M
CPUQuota=50%
TasksMax=64

MemoryMax= is a hard ceiling: if the API exceeds it, the kernel’s OOM killer kills it inside its cgroup, and Restart= brings it back, instead of the whole server running out of memory. CPUQuota=50% allows half of one core, and TasksMax= caps processes and threads, a cheap defence against fork bombs. Check the result with systemctl show orbit-api -p MemoryMax -p CPUQuotaPerSecUSec, and watch live usage per service with systemd-cgtop.

How long did boot take?

$ systemd-analyze blame | head -3
4.812s cloud-init.service
2.130s snapd.seeded.service
1.004s systemd-networkd-wait-online.service

Plain systemd-analyze prints the total boot time; blame sorts units by how long they took to start; systemd-analyze critical-chain orbit-api.service shows which chain of dependencies actually delayed a specific unit. Units that start in parallel do not add up, so the critical chain is usually the more honest view.

Red Hat and Alpine: Rocky and RHEL use the same systemd and every command above, with package units in /usr/lib/systemd/system. SELinux adds a layer: a service that fails with unexplained “permission denied” may be blocked by policy, so check sudo ausearch -m avc -ts recent and restore file labels with sudo restorecon -Rv /srv/orbit. Alpine does not use systemd at all; it uses OpenRC, with rc-service nginx start and rc-update add nginx default and init scripts in /etc/init.d.

Practice lab

Orbit’s API must survive crashes and reboots on orbit-lab, and the security team wants it locked down. Use the unit file from this lesson.

  1. Install the unit, start it, enable it at boot and prove that curl http://127.0.0.1:3000/ answers.
  2. Kill the main process with SIGKILL and confirm it comes back with a new PID. Then stop it with plain kill and explain why it stays down.
  3. Using systemctl edit, add the hardening options and a 128 MB memory limit, and compare systemd-analyze security scores before and after.
  4. Show the last ten log lines of the service and list any failed units on the system.
Show a solution

After creating the file, sudo systemctl daemon-reload and sudo systemctl enable --now orbit-api start and enable it, and curl returns the index page. systemctl show -p MainPID --value orbit-api gives the PID; sudo kill -9 on it (or sudo systemctl kill --signal=SIGKILL orbit-api) leads to “Scheduled restart job” in the journal and a new MainPID after five seconds. sudo kill <PID> sends SIGTERM, which systemd treats as a clean exit, so Restart=on-failure leaves it inactive until you run sudo systemctl start orbit-api. sudo systemctl edit orbit-api with the five hardening lines plus MemoryMax=128M, followed by sudo systemctl restart orbit-api, lowers the score shown by systemd-analyze security orbit-api.service, and systemctl show orbit-api -p MemoryMax prints 134217728. Finally journalctl -u orbit-api -n 10 and systemctl list-units --failed answer the last task.

Common mistakes

  • Editing a unit and forgetting daemon-reload. systemd keeps running the old definition.
  • Starting without enabling. start runs the service now, but it is gone after the next reboot. Use enable --now.
  • Relative paths or shell syntax in ExecStart=. There is no shell: no ~, no pipes, no &&. Wrap complex logic in a script or use /bin/sh -c '...'.

Exercise

Maya adds MemoryMax=256M to /etc/systemd/system/orbit-api.service and runs sudo systemctl restart orbit-api, but systemctl show orbit-api -p MemoryMax still prints infinity. What did she miss?

  1. She must reboot for cgroup limits to apply
  2. She did not run sudo systemctl daemon-reload, so systemd still uses the old unit definition
  3. MemoryMax only works on RHEL
  4. She must disable and re-enable the unit
Show answer

Answer: B. systemd caches unit files. After editing a file directly, run daemon-reload and then restart. systemctl edit does the reload for you, which is one more reason to prefer it.

Exercise

After the monthly reboot of orbit-web-02, nginx is up but the Orbit API is not, although Leo started it last week and it ran fine. systemctl is-enabled orbit-api prints disabled. What fixes it for good?

  1. sudo systemctl restart orbit-api
  2. Adding Restart=always to the unit
  3. sudo systemctl enable --now orbit-api
  4. sudo systemctl daemon-reload
Show answer

Answer: C. start only affects the current boot. enable creates the symlink in multi-user.target.wants so the unit starts at every boot, and --now starts it immediately as well.

Exercise

The Orbit API connects to PostgreSQL on orbit-db-01 at startup and sometimes fails right after boot with “network unreachable”. Its unit has After=network.target. What is the best change?

  1. Add Requires=network.target
  2. Add ExecStartPre=/bin/sleep 30
  3. Use Wants=network-online.target together with After=network-online.target
  4. Change Type=exec to Type=forking
Show answer

Answer: C. network.target only means the network stack exists. network-online.target means interfaces are configured; Wants pulls it in and After orders the API behind it. A sleep only hides the race.

Key takeaways

  • systemd is PID 1: it boots the system, supervises services, collects logs and enforces limits through cgroups.
  • Your own units go in /etc/systemd/system; change package units with systemctl edit drop-ins, and run daemon-reload after editing files by hand.
  • start is for now, enable is for every boot; enable --now does both.
  • Restart=on-failure, User=, EnvironmentFile= and correct Wants=/After= turn a script into a reliable service.
  • A few hardening lines and MemoryMax=/CPUQuota= make a service safer; measure with systemd-analyze security.
  • When something fails, read systemctl status and journalctl -u before changing anything.

FAQ

What is the difference between systemctl start and systemctl enable?

start runs the service now but has no effect on the next boot. enable creates a symlink so the service starts automatically at boot, but does not start it now. Use sudo systemctl enable --now name to do both at once, and disable --now to undo both.

Why does my change to a unit file not take effect?

systemd caches unit definitions. After editing a file in /etc/systemd/system, run sudo systemctl daemon-reload and then restart the service. Also check systemctl cat name: a drop-in in a .service.d directory may be overriding the value you changed.

How do I see the logs of a systemd service?

Run journalctl -u name for everything, journalctl -u name -n 50 for the last fifty lines, journalctl -u name -f to follow live and journalctl -u name -b for the current boot only. systemctl status name also shows the last few lines.

Which Type= should I use for my service?

Use exec (or the default simple) for a program that stays in the foreground, which is how modern apps behave. Use notify if the program supports sd_notify and should only count as started when it is ready. Use oneshot for scripts that run and exit, and avoid forking unless an old daemon forces it.

Is it better to run a service with Restart=always or Restart=on-failure?

on-failure restarts after crashes, error exits and timeouts, but respects a deliberate clean stop, which suits most apps. always also restarts after a clean exit, which is useful for workers that exit on purpose to reload. In both cases the start limit stops endless crash loops, so check systemctl status if a service ends up failed.

↑ ↓ navigate Enter open Esc close