Lesson 15 intermediate

Scheduling with cron and systemd Timers

How to run jobs on a schedule with cron and with systemd timers, read and test schedule expressions, give jobs the right paths and logs, stop runs from overlapping and notice when a nightly job silently stopped running.

Servers are full of work that nobody should do by hand: nightly backups, certificate renewals, log clean-ups, reports, cache warm-ups. Linux has two schedulers for it. cron is the classic: one line per job, available on every Unix since the 1970s. systemd timers are the modern alternative, with logs in the journal, catch-up after downtime and resource limits for free.

In this lesson you schedule Orbit’s backup script both ways, learn why a job that works in your terminal fails at 2 a.m., stop two runs from overlapping and set up a check that tells you when a job did not run. The timer part needs the orbit-lab VM from lesson 1, because containers do not run systemd.

If you skipped ahead

This lesson schedules /usr/local/bin/orbit-backup.sh from lesson 12. If you do not have it, create a minimal stand-in with sudo nano /usr/local/bin/orbit-backup.sh:

#!/usr/bin/env bash
# Minimal stand-in for lesson 12's orbit-backup.sh: archive data and config, keep 7
set -euo pipefail
umask 077
dest=/var/backups/orbit
file="$dest/orbit-$(date +%Y%m%d-%H%M%S).tar.gz"
mkdir -p "$dest"
tar -czf "$file" -C / var/lib/orbit etc/orbit
ls -1t "$dest"/orbit-*.tar.gz | tail -n +8 | xargs -r rm -f --
printf '%s INFO  Wrote %s
' "$(date '+%F %T')" "$file"
$ sudo chmod 755 /usr/local/bin/orbit-backup.sh
$ sudo mkdir -p /var/lib/orbit /etc/orbit /var/log/orbit
$ echo '{"note":"hello"}' | sudo tee /var/lib/orbit/notes.json >/dev/null
$ sudo /usr/local/bin/orbit-backup.sh
2026-11-03 10:12:44 INFO  Wrote /var/backups/orbit/orbit-20261103-101244.tar.gz

Always run a script by hand once before you schedule it. A scheduler cannot fix a script that does not work.

Reading a cron schedule

A cron line has five time fields followed by the command:

┌──────── minute        0-59
│ ┌────── hour          0-23
│ │ ┌──── day of month  1-31
│ │ │ ┌── month         1-12 or jan-dec
│ │ │ │ ┌ day of week   0-7 or sun-sat (0 and 7 are both Sunday)
│ │ │ │ │
30 2 * * *  /usr/local/bin/orbit-backup.sh

Each field accepts * (every value), a list 1,15, a range 1-5 and a step */15 (every 15th value, starting at the lowest). Some examples:

ExpressionRuns
30 2 * * *Every day at 02:30
*/15 * * * *Every 15 minutes (:00, :15, :30, :45)
0 9 * * 1-5At 09:00, Monday to Friday
0 */6 * * *At 00:00, 06:00, 12:00 and 18:00
0 3 1,15 * *At 03:00 on the 1st and the 15th of each month
45 23 * * sunSundays at 23:45
0 0 1 1 *Once a year, at midnight on 1 January

Instead of the five fields you can use a shortcut: @hourly, @daily (same as 0 0 * * *), @weekly, @monthly, @yearly and @reboot, which runs once when cron starts after a boot.

Warning: If you restrict both day of month and day of week, cron runs the job when either matches. 0 4 1 * mon means “on the 1st and on every Monday”, not “on the 1st if it is a Monday”. Do that kind of check inside the script instead.

Your crontab

Each user has a personal crontab, edited with the crontab command (it opens $EDITOR, see lesson 14):

CommandWhat it does
crontab -eEdit your crontab; the syntax is checked when you save
crontab -lList your crontab
crontab -rRemove your whole crontab, without asking (back up first with crontab -l > ~/crontab.bak)
sudo crontab -u leo -lList another user’s crontab
sudo crontab -eEdit root’s crontab

The files live in /var/spool/cron/crontabs/ (Ubuntu) or /var/spool/cron/ (RHEL), but never edit them directly: crontab -e validates and tells the daemon to reload.

System-wide cron files

Jobs that belong to the server rather than to a person go in system files, which have an extra user column between the schedule and the command:

LocationFormatUse it for
crontab -e (per user)five fields + commandA person’s own jobs
/etc/crontabfive fields + user + commandOwned by the distribution; avoid editing
/etc/cron.d/<name>five fields + user + commandJobs installed by packages or your config management
/etc/cron.hourly/, cron.daily/, cron.weekly/, cron.monthly/executable scripts, no schedule“Roughly once a day” jobs; Ubuntu runs the daily ones around 06:25

/etc/cron.d/ is the best home for Orbit’s job, because a tool like Ansible can manage one file per job:

# /etc/cron.d/orbit-backup
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""
30 2 * * * root /usr/bin/flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh >> /var/log/orbit/backup.log 2>&1

Files in /etc/cron.d/ and the cron.daily-style folders must be owned by root, must not be group or world writable, and on Debian and Ubuntu their names may contain only letters, digits, - and _: a file called orbit-backup.sh in cron.daily is silently skipped. run-parts --test /etc/cron.daily lists what would actually run. End every cron file with a newline, or the last line may be ignored.

Why jobs fail under cron

The number one cron bug: the script works when you run it and fails when cron does. cron starts jobs with a tiny environment, usually SHELL=/bin/sh, PATH=/usr/bin:/bin, HOME and LOGNAME, and no startup files are read. So:

  • Use absolute paths for commands and files, or set PATH= at the top of the crontab as above.
  • Do not rely on the current directory; cd /srv/orbit || exit 1 inside the script if needed.
  • Do not rely on aliases, ~/.bashrc or variables you exported in your terminal.
  • Escape % as \%: cron turns a bare % into a newline, so date +%F must be written date +\%F on a cron line.

Reproduce cron’s world before blaming cron: env -i HOME=/root PATH=/usr/bin:/bin /bin/sh -c '/usr/local/bin/orbit-backup.sh'.

Capture output and logs

Cron mails a job’s output to the user (or to MAILTO), but most servers have no mail transfer agent, so the output is thrown away and you see only a hint in the log. Redirect it yourself:

  • >> /var/log/orbit/backup.log 2>&1 appends standard output and errors to a file (rotate it, see lesson 18).
  • 2>&1 | logger -t orbit-backup sends everything to the journal under a tag, readable with journalctl -t orbit-backup.
  • MAILTO=ops@example.com only helps if the server can actually send mail; MAILTO="" disables mail.

The cron daemon logs each start to the journal. A CMD line proves cron started the job, not that it succeeded; that is what your own log is for:

$ journalctl -u cron --since today
Nov 03 02:30:01 orbit-web-01 CRON[48211]: (root) CMD (/usr/bin/flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh >> /var/log/orbit/backup.log 2>&1)

Time zones

cron uses the system time zone (timedatectl shows it), and most servers run in UTC, so “02:30” means 02:30 UTC. Setting TZ= in a crontab only changes the job’s environment, not when it fires. cronie (the cron on RHEL) understands CRON_TZ=Europe/Lisbon to schedule in another zone; on Debian and Ubuntu check man 5 crontab before relying on it. Jobs between 01:00 and 03:00 local time can be skipped or doubled on daylight-saving days, one more reason servers stay on UTC.

Prevent overlapping runs

If a backup takes 70 minutes and runs hourly, the second copy starts while the first is still writing. flock from util-linux takes a lock on a file and refuses (with -n, non-blocking) to start another copy while the lock is held:

# Hold the lock for a minute with a stand-in for a slow backup
$ sudo flock -n /run/lock/orbit-backup.lock sleep 60 &
[1] 5231
$ sudo flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh; echo "exit: $?"
exit: 1

The lock disappears automatically when the process ends, even after a crash, which makes it safer than home-made PID files. Use -w 300 instead of -n to wait up to five minutes rather than give up.

Red Hat and Alpine

RHEL, Rocky and Alma use cronie: the service is crond (systemctl status crond), the log is /var/log/cron and journalctl -u crond, and the package is cronie if it is missing. Alpine uses BusyBox crond under OpenRC (rc-service crond start), root’s crontab is /etc/crontabs/root, and periodic scripts go in /etc/periodic/15min, daily and friends.

systemd timers

A timer is a pair of units: a service that says what to run and a timer that says when. Create the service first:

# /etc/systemd/system/orbit-backup.service
[Unit]
Description=Orbit nightly backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/orbit-backup.sh
Nice=10
IOSchedulingClass=idle

Type=oneshot tells systemd the process runs to completion and exits, instead of staying up like orbit-api. Nice=10 and IOSchedulingClass=idle keep the backup from stealing CPU and disk time from the API. There is no [Install] section: the timer starts the service.

# /etc/systemd/system/orbit-backup.timer
[Unit]
Description=Run the Orbit backup every night

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10m

[Install]
WantedBy=timers.target
  • OnCalendar=*-*-* 02:30:00 means any year, month and day at 02:30. daily, weekly, Mon..Fri 09:00 and *:0/15 (every 15 minutes) also work, and you can append a time zone: *-*-* 02:30 Europe/Lisbon.
  • Persistent=true stores the last run on disk; if the machine was off at 02:30, the job runs soon after the next boot. cron simply misses it.
  • RandomizedDelaySec=10m adds a random delay of up to ten minutes, so a fleet of servers does not hit the backup storage at the same second.

Check your expression before trusting it, then enable the timer (not the service):

$ systemd-analyze calendar '*-*-* 02:30'
  Original form: *-*-* 02:30
Normalized form: *-*-* 02:30:00
    Next elapse: Wed 2026-11-04 02:30:00 UTC
       From now: 16h left
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now orbit-backup.timer
Created symlink '/etc/systemd/system/timers.target.wants/orbit-backup.timer' → '/etc/systemd/system/orbit-backup.timer'.
$ systemctl list-timers orbit-backup.timer
NEXT                         LEFT LAST PASSED UNIT               ACTIVATES
Wed 2026-11-04 02:36:51 UTC   16h -         - orbit-backup.timer orbit-backup.service

NEXT includes the random delay. Run the job right now to test it, and read its output in the journal (-n 5 shows the last five lines): everything the script prints is captured without any redirection.

$ sudo systemctl start orbit-backup.service
$ journalctl -u orbit-backup.service -n 5 --no-pager
Nov 03 10:31:02 orbit-lab systemd[1]: Starting orbit-backup.service - Orbit nightly backup...
Nov 03 10:31:02 orbit-lab orbit-backup.sh[5120]: 2026-11-03 10:31:02 INFO  Creating /var/backups/orbit/orbit-20261103-103102.tar.gz
Nov 03 10:31:02 orbit-lab orbit-backup.sh[5120]: 2026-11-03 10:31:02 INFO  Wrote /var/backups/orbit/orbit-20261103-103102.tar.gz (4.0K)
Nov 03 10:31:02 orbit-lab systemd[1]: orbit-backup.service: Deactivated successfully.
Nov 03 10:31:02 orbit-lab systemd[1]: Finished orbit-backup.service - Orbit nightly backup.

A service cannot run twice at once, so overlapping runs are impossible without flock. systemctl list-timers without arguments also shows the timers your distribution already uses, such as logrotate.timer and apt-daily.timer: many classic cron jobs moved here. For a one-off timer without writing files, sudo systemd-run --on-calendar='*-*-* 02:30' /usr/local/bin/orbit-backup.sh creates a transient one.

cron or timer?

cronsystemd timer
SetupOne lineTwo unit files
LogsOnly what you redirectEvery line in the journal, plus exit status
Missed runs while powered offLostCaught up with Persistent=true
Overlap protectionAdd flockBuilt in
EnvironmentMinimal, set by handDefined in the unit (Environment=, User=)
Resource limits, sandboxingNoneNice=, MemoryMax=, ProtectSystem= and more
Failure hooksNoneOnFailure= starts another unit
PortabilityEvery Unix, BusyBox, many containerssystemd hosts only

Rule of thumb: on a systemd server, new system jobs are timers; personal one-liners and minimal or BusyBox systems stay with cron. In Kubernetes use a CronJob instead of either (see the Kubernetes Tutorial).

Know that the job actually ran

The scariest scheduled job is the one that stopped running in March and nobody noticed until a restore was needed. Logs do not help, because a job that never starts writes nothing. Use a heartbeat (also called a dead man’s switch): the job reports success to a monitoring endpoint at the end of every run, and the monitor alerts when a report is missing. Add it as the last line of the script:

curl -fsS -m 10 --retry 3 https://hc.example.com/ping/orbit-backup > /dev/null

-f turns HTTP errors into a failure, -sS hides the progress bar but keeps error messages, -m 10 gives up after ten seconds and --retry 3 survives a network blip; the URL is a placeholder for your monitor’s check. Hosted services and most monitoring stacks offer such push checks. Two cheaper complements: alert when the newest file in /var/backups/orbit is older than 26 hours, and use OnFailure= in the service to notify on errors. Finally, test a restore now and then; a backup you never restored is a hope.

One-off jobs with at

For “run this once, tonight”, at is simpler than a schedule (install it with sudo apt install at). at now + 30 minutes and at 02:00 tomorrow also work, and on systemd hosts sudo systemd-run --on-active=30m /usr/local/bin/orbit-backup.sh does the same with its output in the journal:

$ echo '/usr/local/bin/orbit-backup.sh' | sudo at 23:00
warning: commands will be executed using /bin/sh
job 3 at Tue Nov  3 23:00:00 2026
$ sudo atq
3	Tue Nov  3 23:00:00 2026 a root
$ sudo atrm 3

Practice lab

Leo wants Orbit’s backup automated on orbit-lab before the weekend, and wants proof it works.

  1. Add a cron job in /etc/cron.d/orbit-backup that runs the backup every 5 minutes as root, with flock and output appended to /var/log/orbit/backup.log. Wait for one run and find it in the log and in journalctl -u cron.
  2. Remove the cron file and create orbit-backup.service and orbit-backup.timer for 02:30 daily with Persistent=true.
  3. Check the expression with systemd-analyze calendar, enable the timer and confirm its next run with systemctl list-timers.
  4. Trigger the service by hand and read its output in the journal.
Show a solution

Write the /etc/cron.d/orbit-backup file from this lesson but with */5 * * * * as the schedule, check it with ls -l (root-owned, mode 644, no dot in the name), wait five minutes and run tail /var/log/orbit/backup.log and journalctl -u cron --since "10 min ago". Then sudo rm /etc/cron.d/orbit-backup, create both unit files under /etc/systemd/system/, run systemd-analyze calendar '*-*-* 02:30', sudo systemctl daemon-reload and sudo systemctl enable --now orbit-backup.timer, and look at the NEXT column of systemctl list-timers orbit-backup.timer. Finally sudo systemctl start orbit-backup.service and journalctl -u orbit-backup.service -n 5 show the “Wrote” line and a new archive in /var/backups/orbit.

Exercise

The backup must run at 02:30 on weekdays only, in the server’s time zone. Which crontab line does that?

  1. 2 30 * * 1-5 /usr/local/bin/orbit-backup.sh
  2. 30 2 * * 1-5 /usr/local/bin/orbit-backup.sh
  3. 30 2 1-5 * * /usr/local/bin/orbit-backup.sh
  4. */30 2 * * 1-5 /usr/local/bin/orbit-backup.sh
Show answer

Answer: B. The order is minute, hour, day of month, month, day of week. 1-5 in the last field is Monday to Friday; in the third field it would mean the 1st to the 5th of the month, and */30 in the minute field runs at 02:00 as well.

Exercise

orbit-backup.sh runs fine when Leo types it, but under cron the log says “pg_dump: command not found”. pg_dump lives in /usr/local/bin on this server. What is the most likely cause?

  1. cron does not allow PostgreSQL tools
  2. The script needs to be in /etc/cron.daily to see pg_dump
  3. cron’s PATH is only /usr/bin:/bin, so the script must use /usr/local/bin/pg_dump or set PATH
  4. Leo’s ~/.bashrc must be sourced in /etc/crontab
Show answer

Answer: C. cron starts jobs with a minimal environment and reads no startup files. Use absolute paths in scripts or set PATH= at the top of the crontab or in the script.

Exercise

The orbit-lab VM is shut down every evening at 18:00 and started at 08:00. Which setup still produces one backup per day?

  1. A crontab line 30 2 * * * with MAILTO set
  2. A timer with OnCalendar=*-*-* 02:30 and RandomizedDelaySec=1h
  3. A timer with OnCalendar=*-*-* 02:30 and Persistent=true
  4. A crontab line with flock -n
Show answer

Answer: C. With Persistent=true, systemd remembers the last run and starts a missed run shortly after boot. cron skips any run that falls while the machine is off, and flock only prevents overlap.

Key takeaways

  • A cron line is minute, hour, day of month, month, day of week, then the command; system files add a user column.
  • cron’s environment is nearly empty: use absolute paths, escape % and redirect output to a log or the journal.
  • Wrap jobs that might run long in flock -n so two copies never overlap.
  • systemd timers pair a Type=oneshot service with a timer; Persistent=true, journal logs and systemd-analyze calendar make them the better default on servers.
  • Monitor that jobs ran, not just that they failed: a heartbeat alerts when the ping goes missing.

FAQ

Why does my cron job not run at all?

Check in order: is the daemon running (systemctl status cron, or crond on RHEL), does journalctl -u cron show a CMD line at the expected time, is the schedule right (remember UTC), and, for /etc/cron.d/ or cron.daily, is the file root-owned, not writable by others and named without a dot. If cron did start it, the problem is inside the job: run it with env -i and a minimal PATH to reproduce.

How do I see the output of a cron job?

Cron only mails output, and without a mail agent it is discarded. Append >> /var/log/orbit/job.log 2>&1 to the line, or pipe into logger -t name and read it with journalctl -t name. A systemd timer captures all output in the journal automatically with journalctl -u name.service.

Should I use cron or systemd timers in 2026?

On a server with systemd, prefer timers for anything important: logging, catch-up after downtime, overlap protection, resource limits and failure hooks come with them. cron is still fine for quick personal jobs and the only option on BusyBox and many minimal systems. Both are fine; consistency within a team matters more.

How do I run a cron job every 30 seconds?

cron’s smallest unit is one minute. The usual workaround is two lines, one plain and one with sleep 30; in front, which is clumsy. A systemd timer handles it directly with OnCalendar=*:*:0/30 or OnUnitActiveSec=30s, but if something must run that often consider a long-running service instead.

Where are systemd timer logs?

The timer itself only logs when it triggers; the useful output is the service’s. Use journalctl -u orbit-backup.service for the job’s output and exit status, systemctl status orbit-backup.service for the last result, and systemctl list-timers --all for last and next run times.

↑ ↓ navigate Enter open Esc close