Servers are full of work that nobody should do by hand: nightly backups, certificate renewals, log clean-ups, reports, cache warm-ups. Linux has two schedulers for it. cron is the classic: one line per job, available on every Unix since the 1970s. systemd timers are the modern alternative, with logs in the journal, catch-up after downtime and resource limits for free.
In this lesson you schedule Orbit’s backup script both ways, learn why a job that works in your terminal fails at 2 a.m., stop two runs from overlapping and set up a check that tells you when a job did not run. The timer part needs the orbit-lab VM from lesson 1, because containers do not run systemd.
If you skipped ahead
This lesson schedules /usr/local/bin/orbit-backup.sh from lesson 12. If you do not have it, create a minimal stand-in with sudo nano /usr/local/bin/orbit-backup.sh:
#!/usr/bin/env bash
# Minimal stand-in for lesson 12's orbit-backup.sh: archive data and config, keep 7
set -euo pipefail
umask 077
dest=/var/backups/orbit
file="$dest/orbit-$(date +%Y%m%d-%H%M%S).tar.gz"
mkdir -p "$dest"
tar -czf "$file" -C / var/lib/orbit etc/orbit
ls -1t "$dest"/orbit-*.tar.gz | tail -n +8 | xargs -r rm -f --
printf '%s INFO Wrote %s
' "$(date '+%F %T')" "$file"
$ sudo chmod 755 /usr/local/bin/orbit-backup.sh
$ sudo mkdir -p /var/lib/orbit /etc/orbit /var/log/orbit
$ echo '{"note":"hello"}' | sudo tee /var/lib/orbit/notes.json >/dev/null
$ sudo /usr/local/bin/orbit-backup.sh
2026-11-03 10:12:44 INFO Wrote /var/backups/orbit/orbit-20261103-101244.tar.gz
Always run a script by hand once before you schedule it. A scheduler cannot fix a script that does not work.
Reading a cron schedule
A cron line has five time fields followed by the command:
┌──────── minute 0-59
│ ┌────── hour 0-23
│ │ ┌──── day of month 1-31
│ │ │ ┌── month 1-12 or jan-dec
│ │ │ │ ┌ day of week 0-7 or sun-sat (0 and 7 are both Sunday)
│ │ │ │ │
30 2 * * * /usr/local/bin/orbit-backup.sh
Each field accepts * (every value), a list 1,15, a range 1-5 and a step */15 (every 15th value, starting at the lowest). Some examples:
| Expression | Runs |
|---|---|
30 2 * * * | Every day at 02:30 |
*/15 * * * * | Every 15 minutes (:00, :15, :30, :45) |
0 9 * * 1-5 | At 09:00, Monday to Friday |
0 */6 * * * | At 00:00, 06:00, 12:00 and 18:00 |
0 3 1,15 * * | At 03:00 on the 1st and the 15th of each month |
45 23 * * sun | Sundays at 23:45 |
0 0 1 1 * | Once a year, at midnight on 1 January |
Instead of the five fields you can use a shortcut: @hourly, @daily (same as 0 0 * * *), @weekly, @monthly, @yearly and @reboot, which runs once when cron starts after a boot.
Warning: If you restrict both day of month and day of week, cron runs the job when either matches. 0 4 1 * mon means “on the 1st and on every Monday”, not “on the 1st if it is a Monday”. Do that kind of check inside the script instead.
Your crontab
Each user has a personal crontab, edited with the crontab command (it opens $EDITOR, see lesson 14):
| Command | What it does |
|---|---|
crontab -e | Edit your crontab; the syntax is checked when you save |
crontab -l | List your crontab |
crontab -r | Remove your whole crontab, without asking (back up first with crontab -l > ~/crontab.bak) |
sudo crontab -u leo -l | List another user’s crontab |
sudo crontab -e | Edit root’s crontab |
The files live in /var/spool/cron/crontabs/ (Ubuntu) or /var/spool/cron/ (RHEL), but never edit them directly: crontab -e validates and tells the daemon to reload.
System-wide cron files
Jobs that belong to the server rather than to a person go in system files, which have an extra user column between the schedule and the command:
| Location | Format | Use it for |
|---|---|---|
crontab -e (per user) | five fields + command | A person’s own jobs |
/etc/crontab | five fields + user + command | Owned by the distribution; avoid editing |
/etc/cron.d/<name> | five fields + user + command | Jobs installed by packages or your config management |
/etc/cron.hourly/, cron.daily/, cron.weekly/, cron.monthly/ | executable scripts, no schedule | “Roughly once a day” jobs; Ubuntu runs the daily ones around 06:25 |
/etc/cron.d/ is the best home for Orbit’s job, because a tool like Ansible can manage one file per job:
# /etc/cron.d/orbit-backup
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAILTO=""
30 2 * * * root /usr/bin/flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh >> /var/log/orbit/backup.log 2>&1
Files in /etc/cron.d/ and the cron.daily-style folders must be owned by root, must not be group or world writable, and on Debian and Ubuntu their names may contain only letters, digits, - and _: a file called orbit-backup.sh in cron.daily is silently skipped. run-parts --test /etc/cron.daily lists what would actually run. End every cron file with a newline, or the last line may be ignored.
Why jobs fail under cron
The number one cron bug: the script works when you run it and fails when cron does. cron starts jobs with a tiny environment, usually SHELL=/bin/sh, PATH=/usr/bin:/bin, HOME and LOGNAME, and no startup files are read. So:
- Use absolute paths for commands and files, or set
PATH=at the top of the crontab as above. - Do not rely on the current directory;
cd /srv/orbit || exit 1inside the script if needed. - Do not rely on aliases,
~/.bashrcor variables you exported in your terminal. - Escape
%as\%: cron turns a bare%into a newline, sodate +%Fmust be writtendate +\%Fon a cron line.
Reproduce cron’s world before blaming cron: env -i HOME=/root PATH=/usr/bin:/bin /bin/sh -c '/usr/local/bin/orbit-backup.sh'.
Capture output and logs
Cron mails a job’s output to the user (or to MAILTO), but most servers have no mail transfer agent, so the output is thrown away and you see only a hint in the log. Redirect it yourself:
>> /var/log/orbit/backup.log 2>&1appends standard output and errors to a file (rotate it, see lesson 18).2>&1 | logger -t orbit-backupsends everything to the journal under a tag, readable withjournalctl -t orbit-backup.MAILTO=ops@example.comonly helps if the server can actually send mail;MAILTO=""disables mail.
The cron daemon logs each start to the journal. A CMD line proves cron started the job, not that it succeeded; that is what your own log is for:
$ journalctl -u cron --since today
Nov 03 02:30:01 orbit-web-01 CRON[48211]: (root) CMD (/usr/bin/flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh >> /var/log/orbit/backup.log 2>&1)
Time zones
cron uses the system time zone (timedatectl shows it), and most servers run in UTC, so “02:30” means 02:30 UTC. Setting TZ= in a crontab only changes the job’s environment, not when it fires. cronie (the cron on RHEL) understands CRON_TZ=Europe/Lisbon to schedule in another zone; on Debian and Ubuntu check man 5 crontab before relying on it. Jobs between 01:00 and 03:00 local time can be skipped or doubled on daylight-saving days, one more reason servers stay on UTC.
Prevent overlapping runs
If a backup takes 70 minutes and runs hourly, the second copy starts while the first is still writing. flock from util-linux takes a lock on a file and refuses (with -n, non-blocking) to start another copy while the lock is held:
# Hold the lock for a minute with a stand-in for a slow backup
$ sudo flock -n /run/lock/orbit-backup.lock sleep 60 &
[1] 5231
$ sudo flock -n /run/lock/orbit-backup.lock /usr/local/bin/orbit-backup.sh; echo "exit: $?"
exit: 1
The lock disappears automatically when the process ends, even after a crash, which makes it safer than home-made PID files. Use -w 300 instead of -n to wait up to five minutes rather than give up.
Red Hat and Alpine
RHEL, Rocky and Alma use cronie: the service is crond (systemctl status crond), the log is /var/log/cron and journalctl -u crond, and the package is cronie if it is missing. Alpine uses BusyBox crond under OpenRC (rc-service crond start), root’s crontab is /etc/crontabs/root, and periodic scripts go in /etc/periodic/15min, daily and friends.
systemd timers
A timer is a pair of units: a service that says what to run and a timer that says when. Create the service first:
# /etc/systemd/system/orbit-backup.service
[Unit]
Description=Orbit nightly backup
[Service]
Type=oneshot
ExecStart=/usr/local/bin/orbit-backup.sh
Nice=10
IOSchedulingClass=idle
Type=oneshot tells systemd the process runs to completion and exits, instead of staying up like orbit-api. Nice=10 and IOSchedulingClass=idle keep the backup from stealing CPU and disk time from the API. There is no [Install] section: the timer starts the service.
# /etc/systemd/system/orbit-backup.timer
[Unit]
Description=Run the Orbit backup every night
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
RandomizedDelaySec=10m
[Install]
WantedBy=timers.target
OnCalendar=*-*-* 02:30:00means any year, month and day at 02:30.daily,weekly,Mon..Fri 09:00and*:0/15(every 15 minutes) also work, and you can append a time zone:*-*-* 02:30 Europe/Lisbon.Persistent=truestores the last run on disk; if the machine was off at 02:30, the job runs soon after the next boot. cron simply misses it.RandomizedDelaySec=10madds a random delay of up to ten minutes, so a fleet of servers does not hit the backup storage at the same second.
Check your expression before trusting it, then enable the timer (not the service):
$ systemd-analyze calendar '*-*-* 02:30'
Original form: *-*-* 02:30
Normalized form: *-*-* 02:30:00
Next elapse: Wed 2026-11-04 02:30:00 UTC
From now: 16h left
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now orbit-backup.timer
Created symlink '/etc/systemd/system/timers.target.wants/orbit-backup.timer' → '/etc/systemd/system/orbit-backup.timer'.
$ systemctl list-timers orbit-backup.timer
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2026-11-04 02:36:51 UTC 16h - - orbit-backup.timer orbit-backup.service
NEXT includes the random delay. Run the job right now to test it, and read its output in the journal (-n 5 shows the last five lines): everything the script prints is captured without any redirection.
$ sudo systemctl start orbit-backup.service
$ journalctl -u orbit-backup.service -n 5 --no-pager
Nov 03 10:31:02 orbit-lab systemd[1]: Starting orbit-backup.service - Orbit nightly backup...
Nov 03 10:31:02 orbit-lab orbit-backup.sh[5120]: 2026-11-03 10:31:02 INFO Creating /var/backups/orbit/orbit-20261103-103102.tar.gz
Nov 03 10:31:02 orbit-lab orbit-backup.sh[5120]: 2026-11-03 10:31:02 INFO Wrote /var/backups/orbit/orbit-20261103-103102.tar.gz (4.0K)
Nov 03 10:31:02 orbit-lab systemd[1]: orbit-backup.service: Deactivated successfully.
Nov 03 10:31:02 orbit-lab systemd[1]: Finished orbit-backup.service - Orbit nightly backup.
A service cannot run twice at once, so overlapping runs are impossible without flock. systemctl list-timers without arguments also shows the timers your distribution already uses, such as logrotate.timer and apt-daily.timer: many classic cron jobs moved here. For a one-off timer without writing files, sudo systemd-run --on-calendar='*-*-* 02:30' /usr/local/bin/orbit-backup.sh creates a transient one.
cron or timer?
| cron | systemd timer | |
|---|---|---|
| Setup | One line | Two unit files |
| Logs | Only what you redirect | Every line in the journal, plus exit status |
| Missed runs while powered off | Lost | Caught up with Persistent=true |
| Overlap protection | Add flock | Built in |
| Environment | Minimal, set by hand | Defined in the unit (Environment=, User=) |
| Resource limits, sandboxing | None | Nice=, MemoryMax=, ProtectSystem= and more |
| Failure hooks | None | OnFailure= starts another unit |
| Portability | Every Unix, BusyBox, many containers | systemd hosts only |
Rule of thumb: on a systemd server, new system jobs are timers; personal one-liners and minimal or BusyBox systems stay with cron. In Kubernetes use a CronJob instead of either (see the Kubernetes Tutorial).
Know that the job actually ran
The scariest scheduled job is the one that stopped running in March and nobody noticed until a restore was needed. Logs do not help, because a job that never starts writes nothing. Use a heartbeat (also called a dead man’s switch): the job reports success to a monitoring endpoint at the end of every run, and the monitor alerts when a report is missing. Add it as the last line of the script:
curl -fsS -m 10 --retry 3 https://hc.example.com/ping/orbit-backup > /dev/null
-f turns HTTP errors into a failure, -sS hides the progress bar but keeps error messages, -m 10 gives up after ten seconds and --retry 3 survives a network blip; the URL is a placeholder for your monitor’s check. Hosted services and most monitoring stacks offer such push checks. Two cheaper complements: alert when the newest file in /var/backups/orbit is older than 26 hours, and use OnFailure= in the service to notify on errors. Finally, test a restore now and then; a backup you never restored is a hope.
One-off jobs with at
For “run this once, tonight”, at is simpler than a schedule (install it with sudo apt install at). at now + 30 minutes and at 02:00 tomorrow also work, and on systemd hosts sudo systemd-run --on-active=30m /usr/local/bin/orbit-backup.sh does the same with its output in the journal:
$ echo '/usr/local/bin/orbit-backup.sh' | sudo at 23:00
warning: commands will be executed using /bin/sh
job 3 at Tue Nov 3 23:00:00 2026
$ sudo atq
3 Tue Nov 3 23:00:00 2026 a root
$ sudo atrm 3
Practice lab
Leo wants Orbit’s backup automated on orbit-lab before the weekend, and wants proof it works.
- Add a cron job in
/etc/cron.d/orbit-backupthat runs the backup every 5 minutes as root, withflockand output appended to/var/log/orbit/backup.log. Wait for one run and find it in the log and injournalctl -u cron. - Remove the cron file and create
orbit-backup.serviceandorbit-backup.timerfor 02:30 daily withPersistent=true. - Check the expression with
systemd-analyze calendar, enable the timer and confirm its next run withsystemctl list-timers. - Trigger the service by hand and read its output in the journal.
Show a solution
Write the /etc/cron.d/orbit-backup file from this lesson but with */5 * * * * as the schedule, check it with ls -l (root-owned, mode 644, no dot in the name), wait five minutes and run tail /var/log/orbit/backup.log and journalctl -u cron --since "10 min ago". Then sudo rm /etc/cron.d/orbit-backup, create both unit files under /etc/systemd/system/, run systemd-analyze calendar '*-*-* 02:30', sudo systemctl daemon-reload and sudo systemctl enable --now orbit-backup.timer, and look at the NEXT column of systemctl list-timers orbit-backup.timer. Finally sudo systemctl start orbit-backup.service and journalctl -u orbit-backup.service -n 5 show the “Wrote” line and a new archive in /var/backups/orbit.
Exercise
The backup must run at 02:30 on weekdays only, in the server’s time zone. Which crontab line does that?
Show answer
Answer: B. The order is minute, hour, day of month, month, day of week. 1-5 in the last field is Monday to Friday; in the third field it would mean the 1st to the 5th of the month, and */30 in the minute field runs at 02:00 as well.
Exercise
orbit-backup.sh runs fine when Leo types it, but under cron the log says “pg_dump: command not found”. pg_dump lives in /usr/local/bin on this server. What is the most likely cause?
Show answer
Answer: C. cron starts jobs with a minimal environment and reads no startup files. Use absolute paths in scripts or set PATH= at the top of the crontab or in the script.
Exercise
The orbit-lab VM is shut down every evening at 18:00 and started at 08:00. Which setup still produces one backup per day?
Show answer
Answer: C. With Persistent=true, systemd remembers the last run and starts a missed run shortly after boot. cron skips any run that falls while the machine is off, and flock only prevents overlap.
Key takeaways
- A cron line is minute, hour, day of month, month, day of week, then the command; system files add a user column.
- cron’s environment is nearly empty: use absolute paths, escape
%and redirect output to a log or the journal. - Wrap jobs that might run long in
flock -nso two copies never overlap. - systemd timers pair a
Type=oneshotservice with a timer;Persistent=true, journal logs andsystemd-analyze calendarmake them the better default on servers. - Monitor that jobs ran, not just that they failed: a heartbeat alerts when the ping goes missing.
FAQ
Why does my cron job not run at all?
Check in order: is the daemon running (systemctl status cron, or crond on RHEL), does journalctl -u cron show a CMD line at the expected time, is the schedule right (remember UTC), and, for /etc/cron.d/ or cron.daily, is the file root-owned, not writable by others and named without a dot. If cron did start it, the problem is inside the job: run it with env -i and a minimal PATH to reproduce.
How do I see the output of a cron job?
Cron only mails output, and without a mail agent it is discarded. Append >> /var/log/orbit/job.log 2>&1 to the line, or pipe into logger -t name and read it with journalctl -t name. A systemd timer captures all output in the journal automatically with journalctl -u name.service.
Should I use cron or systemd timers in 2026?
On a server with systemd, prefer timers for anything important: logging, catch-up after downtime, overlap protection, resource limits and failure hooks come with them. cron is still fine for quick personal jobs and the only option on BusyBox and many minimal systems. Both are fine; consistency within a team matters more.
How do I run a cron job every 30 seconds?
cron’s smallest unit is one minute. The usual workaround is two lines, one plain and one with sleep 30; in front, which is clumsy. A systemd timer handles it directly with OnCalendar=*:*:0/30 or OnUnitActiveSec=30s, but if something must run that often consider a long-running service instead.
Where are systemd timer logs?
The timer itself only logs when it triggers; the useful output is the service’s. Use journalctl -u orbit-backup.service for the job’s output and exit status, systemctl status orbit-backup.service for the last result, and systemctl list-timers --all for last and next run times.