Lesson 18 intermediate

Logs with journald and logrotate

Where Linux keeps its logs, how to query the systemd journal by service, time, priority and boot, how to keep log files from filling the disk with logrotate, and how to trace an unexplained Orbit restart back to its cause.

When something breaks on a server, the logs are where the story is written down. The API that restarted in the night, the SSH bot hammering port 22, the kernel killing a process because memory ran out: all of it lands in a log somewhere, with a timestamp. The engineers who fix incidents fast are rarely the ones who know the most commands; they are the ones who know where to look and how to cut a million lines down to the ten that matter.

In this lesson you learn where logs live on a modern Linux server, how to query the systemd journal precisely, how to write your own log entries from scripts, how to keep log files from eating the disk with logrotate, and how logs leave the machine in a real setup. You finish by investigating why Orbit’s API restarted at 03:12.

Where logs live

A 2026 server has two logging systems side by side, plus whatever the applications do on their own:

SourceWhere it goesHow you read it
systemd services (stdout and stderr)The journal, /var/log/journal/journalctl -u orbit-api
Kernel messagesThe kernel ring buffer and the journaljournalctl -k, sudo dmesg -T
Classic syslog files/var/log/syslog, /var/log/auth.log (Ubuntu); /var/log/messages, /var/log/secure (RHEL)less, grep, tail -f
nginx/var/log/nginx/access.log and error.logtail, grep, awk
Orbit’s own files/var/log/orbit/*.logtail, jq

journald (the systemd-journald service) collects everything that systemd-managed programs print, plus kernel and syslog messages, and stores it in an indexed binary format with metadata such as the unit name, PID and priority. The classic text files in /var/log come from rsyslog, which receives a copy of each message from journald and writes it out in the old format. Most Ubuntu server images still ship rsyslog; minimal and cloud images may not, in which case the journal is the only place to look.

Querying the journal with journalctl

Run without arguments, journalctl dumps everything since the oldest entry into a pager. That is rarely what you want. The power is in the filters, and they combine freely. Reading system logs needs membership in the adm or systemd-journal group, or sudo; the default ubuntu user is in adm already.

By unit, live and latest

$ journalctl -u orbit-api -n 3
Nov 03 09:13:02 orbit-web-01 python3[2214]: 127.0.0.1 - - [03/Nov/2026 09:13:02] "GET /health HTTP/1.0" 200 -
Nov 03 09:13:05 orbit-web-01 python3[2214]: 127.0.0.1 - - [03/Nov/2026 09:13:05] "GET /notes HTTP/1.0" 200 -
Nov 03 09:13:09 orbit-web-01 python3[2214]: 127.0.0.1 - - [03/Nov/2026 09:13:09] "GET /missing HTTP/1.0" 404 -
$ journalctl -u orbit-api -f
  • -u orbit-api filters by systemd unit (the .service suffix is optional). Repeat it to merge several units: -u orbit-api -u nginx.
  • -n 3 shows the last three entries; -n alone means the last ten.
  • -f follows new entries as they arrive, like tail -f (Ctrl+C stops it). -e jumps to the end of the pager, and --no-pager prints straight to the terminal, which is what you want in scripts and pipes.

By time

$ journalctl -u orbit-api --since "2026-11-03 03:00" --until "2026-11-03 03:20"
$ journalctl -u nginx --since "1 hour ago"

--since and --until accept full timestamps, "10 min ago", today, yesterday and times alone (--since 03:00 means today at 03:00). Narrowing the window first is the single biggest time saver during an incident.

By priority

Every message carries a syslog priority, from most to least severe: 0 emerg (system unusable), 1 alert (act now), 2 crit (critical condition such as failing hardware), 3 err (an operation failed), 4 warning (may become a problem), 5 notice (normal but significant), 6 info (routine) and 7 debug (developer detail). journalctl -p err shows priority 3 and everything more severe (0 to 3). A range works too: -p warning..err. Combine with a boot or a time window, for example journalctl -p err -b, for a quick “what went wrong since the last boot” overview.

By boot, kernel and text

$ journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
 -1 5c1e0f8e3b0a4d7c9a6e2f1b8d4c3a21 Mon 2026-11-02 08:01:12 UTC Mon 2026-11-02 22:14:40 UTC
  0 a93f7c2d1e6b4f08b5d9c3e7a1f2d640 Tue 2026-11-03 07:58:03 UTC Tue 2026-11-03 09:41:07 UTC
$ journalctl -b -1 -p warning
$ journalctl -u orbit-api -g "Traceback|ERROR"
  • -b limits output to the current boot, -b -1 to the previous one. Indispensable after an unexpected reboot: the reason is in the last lines of -b -1.
  • -k shows kernel messages only (the same as dmesg, but with the journal’s filters and history).
  • -g (--grep) filters messages with a regular expression, case-insensitive when the pattern is all lowercase.
  • -t orbit-backup filters by syslog identifier, the tag a script used when it logged (see logger below).

Output formats

The default short format is made for humans. Other formats help with correlation and tooling:

$ journalctl -u orbit-api -n 1 -o short-iso
2026-11-03T09:13:09+00:00 orbit-web-01 python3[2214]: 127.0.0.1 - - [03/Nov/2026 09:13:09] "GET /missing HTTP/1.0" 404 -
$ journalctl -u orbit-api -n 1 -o json | jq -c '{unit: ._SYSTEMD_UNIT, pid: ._PID, prio: .PRIORITY}'
{"unit":"orbit-api.service","pid":"2214","prio":"6"}

-o short-iso prints sortable ISO 8601 timestamps with the time zone, which is what you want when you compare logs from several servers. -o json (or json-pretty) exposes every field journald stored, and -o cat prints only the message text, handy for piping into grep or awk.

Keeping the journal under control

$ journalctl --disk-usage
Archived and active journals take up 1.8G in the file system.
$ sudo journalctl --vacuum-size=500M
Vacuuming done, freed 1.3G of archived journals from /var/log/journal/7e2b9c41d0a84f3e9b61c5d2f8a07e13.
$ sudo journalctl --vacuum-time=2weeks

Vacuuming only removes archived journal files, never the active one, so it is safe to run on a busy server. To make limits permanent, configure journald itself.

Persistent storage and size limits

With the default Storage=auto, journald keeps logs on disk only if /var/log/journal/ exists; otherwise they live in memory under /run/log/journal/ and vanish at reboot, and -b -1 has nothing to show. Ubuntu creates the directory, but some other distributions and container images do not. Rather than editing the main /etc/systemd/journald.conf, put your settings in a drop-in file:

# /etc/systemd/journald.conf.d/50-orbit.conf
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=2G
MaxRetentionSec=1month
  • Storage=persistent creates /var/log/journal/ if needed and always writes to disk.
  • SystemMaxUse=1G caps the journal’s total size. Without it, the default is 10% of the filesystem, capped at 4 GB.
  • SystemKeepFree=2G makes journald back off when the disk has less than 2 GB free.
  • MaxRetentionSec=1month drops entries older than a month.

Apply it with sudo systemctl restart systemd-journald. Drop-in files survive package upgrades and are easy to ship with configuration management.

Writing logs from scripts with logger

Scripts such as /usr/local/bin/orbit-backup.sh should log to the same place as everything else, not to a random file in /tmp. The logger command sends a message to the journal (and rsyslog) with a tag and a priority:

$ logger -t orbit-backup -p user.info "backup started"
$ logger -t orbit-backup -p user.err "backup failed: /var/backups/orbit is full"
$ journalctl -t orbit-backup --since today
Nov 03 09:30:01 orbit-web-01 orbit-backup[3307]: backup started
Nov 03 09:30:14 orbit-web-01 orbit-backup[3311]: backup failed: /var/backups/orbit is full

-t sets the identifier you filter on later, and -p takes a facility.priority pair. If the script runs as a systemd service or timer (see lesson 15), plain echo lines already land in the journal under the unit’s name; logger is still useful for setting a priority so that -p err finds the failures.

Reading SSH and authentication logs

Any server with port 22 open to the internet receives a steady stream of login attempts from bots. Knowing how to read them tells you whether you are looking at noise or at a real problem.

$ journalctl -u ssh --since today -g "invalid user|accepted|failed"
Nov 03 04:12:55 orbit-web-01 sshd-session[1520]: Invalid user admin from 203.0.113.50 port 51234
Nov 03 08:41:07 orbit-web-01 sshd-session[2001]: Accepted publickey for leo from 198.51.100.24 port 60122 ssh2: ED25519 SHA256:k3J...
$ sudo grep "Invalid user" /var/log/auth.log | awk '{print $(NF-2)}' | sort | uniq -c | sort -rn | head -3
    412 203.0.113.50
    127 192.0.2.77
     38 198.51.100.200

On Ubuntu the unit is ssh; on RHEL-family systems it is sshd, and the file is /var/log/secure instead of /var/log/auth.log. Recent OpenSSH versions run each connection in a separate sshd-session process, so that name appears in the log lines. “Invalid user” and “[preauth]” lines from strangers are normal background noise once password login is disabled (lesson 16). An Accepted line for a user or address you do not recognise is not noise: that is the one to escalate.

The kernel ring buffer and the OOM killer

The kernel keeps its own messages in a fixed-size memory area called the ring buffer. When it fills up, the oldest messages are overwritten. Read it with dmesg:

$ sudo dmesg -T --level=err,warn | tail -3
[Tue Nov  3 03:12:07 2026] tar invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
[Tue Nov  3 03:12:07 2026] Out of memory: Killed process 1874 (python3) total-vm:1402312kB, anon-rss:1198340kB, file-rss:2304kB, shmem-rss:0kB, UID:998 pgtables:2412kB oom_score_adj:0

-T converts the raw seconds-since-boot timestamps into wall-clock time (approximately: it drifts after suspend), and --level filters by priority. On Ubuntu, dmesg needs sudo because unprivileged access to the kernel log is restricted. Because the ring buffer is small and lost at reboot, journalctl -k is the better tool for anything older than a few hours. The OOM killer (out of memory) is the kernel’s last resort when RAM and swap are exhausted: it picks the process with the highest “badness” score, usually the biggest one, and kills it with SIGKILL. The process gets no chance to log anything itself, so the kernel log is the only witness. Lesson 19 shows how to find out why memory ran out.

Rotating log files with logrotate

The journal manages its own size, but plain files such as /var/log/orbit/api.log and nginx’s access log grow forever unless something rotates them. logrotate renames the current file, starts a fresh one, compresses old copies and deletes the oldest. On Ubuntu and RHEL it runs once a day from logrotate.timer, reading /etc/logrotate.conf and every file in /etc/logrotate.d/. Here is a policy for Orbit’s application logs:

# /etc/logrotate.d/orbit
/var/log/orbit/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    su orbit orbit
    create 0640 orbit orbit
    sharedscripts
    postrotate
        systemctl reload orbit-api.service > /dev/null 2>&1 || true
    endscript
}
DirectiveWhat it does
dailyRotate once a day (weekly, monthly, or maxsize 100M to also rotate early when a file gets big)
rotate 14Keep 14 old files, then delete the oldest
compress, delaycompressGzip rotated files, but leave the most recent one uncompressed, so a process still writing to it briefly does not corrupt a gzip file, and today’s and yesterday’s logs stay greppable
missingok, notifemptyDo not complain if no file matches; skip rotation when the file is empty
su orbit orbitRun the rotation as the orbit user and group, required when the log directory is not owned by root
create 0640 orbit orbitCreate the new empty file with this mode, owner and group
sharedscriptsRun postrotate once for all matching files, not once per file
postrotate … endscriptCommands to run after rotation, typically telling the app to reopen its log file

postrotate or copytruncate

After a rename, a program that still holds the old file open keeps writing into api.log.1, because it writes to the open file, not to the name. There are two fixes, and the choice is a real trade-off:

  • postrotate with a reload or signal: logrotate renames the file and then asks the program to reopen its logs. nginx does this on USR1, many daemons on HUP or a systemctl reload. No lines are lost. Use it whenever the application supports it.
  • copytruncate: logrotate copies the file and then truncates the original to zero in place, so the program keeps writing to the same open file. It works with any program, but lines written between the copy and the truncate are lost, and the program must open its file in append mode or the file fills with zeros up to the old size. In the lab, where the simulated API cannot reopen files, it is the pragmatic choice.

Test before you trust

$ sudo logrotate -d /etc/logrotate.d/orbit
rotating pattern: /var/log/orbit/*.log  after 1 days (14 rotations)
considering log /var/log/orbit/api.log
  log does not need rotating (log has already been rotated)
$ sudo logrotate -f -v /etc/logrotate.d/orbit

-d (debug) shows what logrotate would do without changing anything, and catches syntax and permission errors. -f forces a rotation now regardless of schedule (afterwards ls /var/log/orbit shows api.log and api.log.1), and -v explains each step. logrotate remembers when it last rotated each file in /var/lib/logrotate/status.

Structured logs

A log line such as user 42 saved note in 87ms is easy to read and painful to query. The same event as JSON is easy for both people and machines:

{"ts":"2026-11-03T09:13:05Z","level":"info","service":"orbit-api","event":"note_saved","user_id":42,"duration_ms":87,"request_id":"c0a8f3"}

Structured logging pays off quickly: jq 'select(.duration_ms > 500)' /var/log/orbit/api.log finds slow requests without fragile regular expressions, a log backend can index each field, and a shared request_id lets you follow one request across nginx, the API and the database. Keep the field names consistent across services, log in UTC, and never log secrets, tokens or full request bodies.

Shipping logs off the box

Logs that only exist on the server disappear with it, and nobody wants to SSH into twenty machines to grep. Production setups run a small agent on each server that forwards logs to a central store:

PieceExamplesRole
Built-in forwardingrsyslog (TCP or RELP with TLS), systemd-journal-uploadSimple, already installed, good for syslog-style text
Agent or collectorVector, Fluent Bit, OpenTelemetry CollectorRead the journal and files, parse, enrich and send to one or more backends
BackendGrafana Loki, Elasticsearch or OpenSearch, a cloud log serviceStore, index, search, alert

The OpenTelemetry Collector is increasingly the default choice in 2026 because the same agent can handle logs, metrics and traces. Whatever you pick, keep a short local retention (the journal settings above) as a fallback for when the network or backend is down.

Investigation: why did orbit-api restart at 03:12?

The morning dashboard shows that orbit-api on orbit-web-01 restarted at 03:12. Nobody deployed. Start with the unit itself, in a narrow window:

$ journalctl -u orbit-api --since "03:05" --until "03:15" -o short-iso
2026-11-03T03:12:07+00:00 orbit-web-01 systemd[1]: orbit-api.service: A process of this unit has been killed by the OOM killer.
2026-11-03T03:12:07+00:00 orbit-web-01 systemd[1]: orbit-api.service: Main process exited, code=killed, status=9/KILL
2026-11-03T03:12:07+00:00 orbit-web-01 systemd[1]: orbit-api.service: Failed with result 'oom-kill'.
2026-11-03T03:12:12+00:00 orbit-web-01 systemd[1]: orbit-api.service: Scheduled restart job, restart counter is at 1.
2026-11-03T03:12:12+00:00 orbit-web-01 systemd[1]: Started orbit-api.service - Orbit API.

So it was not a crash in the code: the kernel killed it. Confirm with the kernel log (journalctl -k --since 03:05 --until 03:15), which shows the OOM lines from the previous section and, importantly, who was allocating memory at the time: tar invoked oom-killer. Then widen the view to everything that happened around it:

$ journalctl --since "03:00" --until "03:13" -o short-iso --no-pager | grep -E "backup|invoked oom|Killed process"
2026-11-03T03:00:01+00:00 orbit-web-01 orbit-backup[3307]: backup started
2026-11-03T03:12:07+00:00 orbit-web-01 kernel: tar invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
2026-11-03T03:12:07+00:00 orbit-web-01 kernel: Out of memory: Killed process 1874 (python3) total-vm:1402312kB, anon-rss:1198340kB, file-rss:2304kB, shmem-rss:0kB, UID:998 pgtables:2412kB oom_score_adj:0
2026-11-03T03:12:30+00:00 orbit-web-01 orbit-backup[3307]: backup finished in 12m29s

The nightly backup started at 03:00, and the memory it needed pushed the server over the edge; the kernel picked the biggest process, which was the API. systemctl show orbit-api -p NRestarts confirms the restart count. The fixes belong to the next lesson (cap the backup with MemoryMax= in its unit, give the API a lower OOMScoreAdjust=, look at swap and instance size), but the logs gave you the cause in four commands.

Practice lab

You are preparing orbit-lab for production logging. If you skipped ahead, create the service first: sudo useradd --system --home-dir /srv/orbit --shell /usr/sbin/nologin orbit, sudo mkdir -p /srv/orbit/public /var/log/orbit, sudo chown orbit:orbit /var/log/orbit, then write the orbit-api.service unit from lesson 9 and start it.

  1. Show the last 20 lines of orbit-api with ISO timestamps, then follow it live while you run curl -s localhost:3000/ > /dev/null in a second shell.
  2. Log a success and a failure message tagged orbit-backup with logger, and show only the failure using a priority filter.
  3. Create a journald drop-in that keeps the journal persistent and under 500 MB, apply it, and check the disk usage.
  4. Fill /var/log/orbit/api.log with a few thousand lines, write /etc/logrotate.d/orbit with copytruncate, test it with -d, then force a rotation. Bonus: reboot and show the previous boot’s warnings.
Show a solution

journalctl -u orbit-api -n 20 -o short-iso shows the recent lines and journalctl -u orbit-api -f follows them while the curl requests appear. logger -t orbit-backup -p user.info "backup ok" and logger -t orbit-backup -p user.err "backup failed" write the messages, and journalctl -t orbit-backup -p err shows only the failure. Create /etc/systemd/journald.conf.d/50-orbit.conf with [Journal], Storage=persistent and SystemMaxUse=500M, run sudo systemctl restart systemd-journald and check with journalctl --disk-usage. Generate the file with seq 1 5000 | sudo -u orbit tee /var/log/orbit/api.log > /dev/null, write the logrotate file with daily, rotate 14, compress, delaycompress, missingok, notifempty, su orbit orbit and copytruncate, run sudo logrotate -d /etc/logrotate.d/orbit to check it and sudo logrotate -f -v /etc/logrotate.d/orbit to rotate; ls -l /var/log/orbit now shows an empty api.log and a full api.log.1. For the bonus, journalctl --list-boots shows two entries after a reboot and journalctl -b -1 -p warning shows the previous boot’s warnings.

Exercise

orbit-web-02 rebooted unexpectedly overnight. Which command most directly shows what was logged just before it went down?

  1. journalctl -f
  2. sudo dmesg -T, because the ring buffer survives reboots
  3. journalctl -b -1 -n 50, assuming the journal is persistent
  4. tail /var/log/nginx/error.log
Show answer

Answer: C. -b -1 selects the previous boot, and -n 50 its last 50 entries. It only works when the journal is stored on disk; the kernel ring buffer read by dmesg is cleared at every boot.

Exercise

After you add /etc/logrotate.d/orbit with create but no postrotate or copytruncate, api.log stays empty every day and api.log.1 keeps growing. Why?

  1. logrotate is not running because the timer is disabled
  2. compress is corrupting the new file
  3. The API still holds the old file open and writes to it under its new name; it must reopen its log (postrotate) or the file must be truncated in place (copytruncate)
  4. The create mode 0640 prevents the API from writing
Show answer

Answer: C. Renaming a file does not change what an open file descriptor points to. Signal or reload the app in postrotate so it reopens the path, or use copytruncate and accept that a few lines may be lost.

Exercise

The journal on orbit-db-01 uses 3.9 GB and the disk is nearly full. What is the right immediate action and the right lasting fix?

  1. Delete /var/log/journal with rm -rf and restart the server
  2. Run sudo journalctl --vacuum-size=500M now, and set SystemMaxUse= in a drop-in under /etc/systemd/journald.conf.d/
  3. Switch the journal to volatile storage so it never uses disk again
  4. Disable systemd-journald
Show answer

Answer: B. Vacuuming safely removes archived journal files. A drop-in with SystemMaxUse= (and optionally SystemKeepFree=) makes the limit permanent and survives upgrades. Volatile storage would lose logs at every reboot.

Key takeaways

  • systemd services log to the journal; query it with journalctl -u, narrowed by --since, -p, -b and -g.
  • Make the journal persistent and size-limited with a drop-in in /etc/systemd/journald.conf.d/.
  • Use logger -t in scripts and journalctl -k or dmesg -T for the kernel, including OOM killer messages.
  • logrotate keeps files in check; prefer postrotate with a reload, use copytruncate only for apps that cannot reopen logs, and test with -d.
  • Structured JSON logs and an agent that ships them to a central backend turn logs into something you can search across servers.

FAQ

Where are logs stored in Linux?

On a systemd-based server, service and kernel messages go to the journal in /var/log/journal/, which you read with journalctl. Many distributions also run rsyslog, which writes text files such as /var/log/syslog and /var/log/auth.log on Ubuntu or /var/log/messages and /var/log/secure on RHEL. Applications like nginx write their own files under /var/log/<app>/.

What is the difference between journalctl and the files in /var/log?

The journal is a structured, indexed store with metadata for every entry (unit, PID, priority, boot), so filters such as -u or -p err are fast and precise. The files in /var/log are plain text written by rsyslog or by applications themselves, which suits grep, tail and old tools. On most servers the same system messages exist in both places.

How do I see logs in real time on Linux?

Use journalctl -u <unit> -f for a systemd service and tail -F /var/log/<file> for a text file. Capital -F keeps following the file name across log rotation, while lowercase -f keeps reading the old file after it is renamed. Pipe either one through grep --line-buffered to filter while you watch.

How do I clear or shrink journal logs safely?

Run sudo journalctl --vacuum-size=500M or --vacuum-time=2weeks to delete archived journal files, and set SystemMaxUse= in a drop-in under /etc/systemd/journald.conf.d/ so it does not grow back. Do not delete files in /var/log/journal by hand while journald is running.

How can I tell whether a process was killed by the OOM killer?

Search the kernel log with journalctl -k -g "killed process" or sudo dmesg -T | grep -i oom. systemd also records it: systemctl status <unit> or journalctl -u <unit> shows “killed by the OOM killer” and “Failed with result ‘oom-kill'”. For services with a memory limit, the cgroup file memory.events counts oom_kill events.

↑ ↓ navigate Enter open Esc close