Lesson 02 beginner

The Linux Filesystem

How Linux arranges every file, disk and device in one tree under /, what each standard directory is for on a server, why Orbit's files live where they do, and how /proc and /sys let you read the kernel's mind.

On a server you rarely ask “where did I save that?”. You ask “where should this be?”, and Linux answers the same way on almost every machine. Configuration is in /etc, logs are in /var/log, programs are in /usr/bin. Once the map is in your head, you can walk up to a server you have never seen, find the nginx config in seconds and know which directory is filling the disk before anyone opens a ticket.

In this lesson you learn the layout of the Linux tree, what “everything is a file” means in practice, how extra disks join the tree, and where Orbit keeps its code, configuration, data and logs. At the end you create Orbit’s directories on your lab machine. A container from lesson 1 is fine for most of it; the mount examples look more realistic on the Multipass VM.

One tree, starting at /

Windows gives every disk a letter: C:\, D:\. Linux has no letters. There is a single tree, and its top is the root directory, written as a lone slash: /. Every file on the machine has a path that starts there, such as /etc/hostname or /var/log/syslog.

Take a first look:

$ ls /
bin   dev  home  lib64       media  opt   root  sbin  srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   snap  sys  usr

Do not confuse three things that share the word “root”: the root directory /, the root user (the administrator), and /root, which is simply the root user’s home directory.

Everything is a file

Linux presents almost everything through the same interface: a path you can open, read and write. Regular documents are files, but so are disks (/dev/sda), the bottomless bin /dev/null, a running process’s details (/proc/1234/status) and many kernel settings (/proc/sys/...). The payoff is huge: the same small tools (cat, grep, ls) work on all of them, and scripts can inspect the system without special APIs.

The idea has limits. Network interfaces, for example, are not device files in /dev; you manage them with ip and can only peek at them through /sys/class/net. Think “almost everything”.

The directories that matter on a server

The layout follows the Filesystem Hierarchy Standard (FHS), a shared convention that distributions follow closely. You do not need to memorise all of it. These are the directories you will touch in real work:

DirectoryWhat lives thereYou go there when
/etcSystem-wide configuration, as text filesChanging how a service behaves (/etc/nginx/, /etc/ssh/sshd_config)
/varData that changes while the system runsAnything grows over time
/var/logLog filesSomething broke
/var/libPersistent state of services (databases, Docker, package lists)Backing up or moving data
/var/cacheData that can be rebuilt (downloaded packages)Freeing disk space safely
/homePersonal directories of human users (/home/maya)Working as yourself
/rootThe root user’s homeRarely; prefer sudo from your own account
/usrInstalled programs, libraries and docs from the distributionFinding a binary or its documentation
/usr/localSoftware and scripts you add by hand, outside the package managerInstalling a small tool or a cron script
/optSelf-contained third-party bundles (/opt/vendor-agent/)A vendor tells you to
/srvData this server serves: websites, app codeDeploying an application
/tmpShort-lived scratch files, writable by everyoneNever for anything you need tomorrow
/bootThe kernel, initramfs and boot loader filesKernel updates, a full /boot partition
/devDevice files for disks, terminals and pseudo-devicesWorking with disks
/procA live view of processes and the kernel, generated on the flyInspecting the running system
/sysA live view of devices, drivers and cgroupsInspecting hardware and resource limits
/runRuntime state since the last boot: PID files, socketsDebugging a service that will not start
/mnt, /mediaPlaces to attach disks temporarilyMounting something by hand

A useful way to remember the split: /usr is what the distribution installed, /etc is how you configured it, and /var is what it produced since.

Tip: When a disk fills up on a server, the culprit is almost always under /var: logs in /var/log, a database in /var/lib, Docker images in /var/lib/docker or old packages in /var/cache. Check there first.

/tmp and /var/tmp

Both hold temporary files, but with different lifetimes. /tmp is for files nobody needs after a reboot; on Debian 13, for example, it is a RAM-backed tmpfs by default, and other distributions clean it at boot or on a timer. /var/tmp survives reboots and is cleaned less often. Check what your machine does with findmnt /tmp. Either way, never keep anything important in them.

The merged /usr

On old Unix systems, /bin held essential commands needed during boot and /usr/bin held everything else, because /usr might live on a separate disk. That split no longer makes sense, so modern distributions merged them. /bin, /sbin and /lib still exist, but only as symbolic links:

$ ls -ld /bin /sbin /lib
lrwxrwxrwx 1 root root 7 Apr 21 10:12 /bin -> usr/bin
lrwxrwxrwx 1 root root 7 Apr 21 10:12 /lib -> usr/lib
lrwxrwxrwx 1 root root 8 Apr 21 10:12 /sbin -> usr/sbin

The arrow means “this name points to that directory”. So /bin/bash and /usr/bin/bash are the same file, and old scripts that start with #!/bin/bash keep working.

Mount points: how disks join the tree

If there is only one tree, where does a second disk go? You mount it: you attach its filesystem to an existing directory, called a mount point. From then on, everything under that directory lives on the other disk, and nobody using the path can tell the difference.

df -h shows mounted filesystems and how full they are (-h prints human-readable sizes):

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           197M  1.1M  196M   1% /run
/dev/sda1       9.6G  2.4G  7.2G  25% /
tmpfs           983M     0  983M   0% /dev/shm
/dev/sda16      881M   62M  758M   8% /boot
/dev/sda15      105M  6.1M   99M   6% /boot/efi
tmpfs           197M   12K  197M   1% /run/user/1000

On Orbit’s database server the picture shows a second disk. The cloud attaches it as nvme1n1 and it is mounted on PostgreSQL’s data directory, so the database can grow without filling the system disk:

$ lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS
NAME          SIZE TYPE MOUNTPOINTS
nvme0n1        30G disk
├─nvme0n1p1    29G part /
├─nvme0n1p15  106M part /boot/efi
└─nvme0n1p16  913M part /boot
nvme1n1       200G disk /var/lib/postgresql
$ findmnt /var/lib/postgresql
TARGET              SOURCE       FSTYPE OPTIONS
/var/lib/postgresql /dev/nvme1n1 xfs    rw,relatime,attr2,inode64,noquota

lsblk lists block devices as a tree; findmnt tells you which filesystem a path lives on. Disk names vary: sda on many VMs, vda on some hypervisors, nvme0n1 on most clouds. Lesson 11 (disks and storage) shows how to format, mount and grow disks safely.

Warning: If you mount a disk on a directory that already contains files, those files do not disappear, they are just hidden until you unmount. That explains many “the disk is full but I cannot find the files” mysteries.

Where Orbit lives, and why

Orbit follows the FHS so that any engineer can find its pieces without a guide:

PathContentsWhy there
/srv/orbitApplication code: api/, public/, releases//srv is for what the server serves; it is the home of the orbit service account
/etc/orbit/orbit.envSettings and secrets, readable by root and the orbit groupConfiguration belongs in /etc, where backups and config management expect it
/var/lib/orbitUploaded files and other state the app writesPersistent service data belongs in /var/lib
/var/log/orbitThe API’s own log filesLogs belong in /var/log, where rotation and monitoring tools look
/var/backups/orbitNightly backup archivesNext to the system’s own backups
/usr/local/bin/orbit-backup.shThe backup scriptHand-written tools go in /usr/local, never in /usr/bin
/etc/systemd/system/orbit-api.serviceThe service definitionYour own unit files go here (lesson 9)

This separation pays off. Deploying new code only touches /srv/orbit. Rebuilding the server means restoring /etc/orbit and /var/lib/orbit. A package update never overwrites your script, because the package manager owns /usr/bin and leaves /usr/local alone.

File types you will see in ls -l

The first character of each ls -l line tells you what kind of object you are looking at:

$ ls -ld /bin /dev/null /dev/sda /etc /etc/hostname /run/systemd/journal/socket
lrwxrwxrwx  1 root root    7 Apr 21 10:12 /bin -> usr/bin
crw-rw-rw-  1 root root 1, 3 Nov  3 09:30 /dev/null
brw-rw----  1 root disk 8, 0 Nov  3 09:30 /dev/sda
drwxr-xr-x 98 root root 4096 Nov  3 09:30 /etc
-rw-r--r--  1 root root   10 Oct 28 09:05 /etc/hostname
srw-rw-rw-  1 root root    0 Nov  3 09:30 /run/systemd/journal/socket
CharacterTypeExample
-Regular file: text, binaries, images/etc/hostname
dDirectory/etc
lSymbolic link, a pointer to another path/bin
cCharacter device, read byte by byte/dev/null, terminals
bBlock device, read in blocksDisks such as /dev/sda
sSocket, for communication between processesThe journal socket
pNamed pipe (FIFO)Rare on servers

For devices, the two numbers in place of the size identify the driver and the device. The -d flag lists directories themselves instead of their contents. Lesson 6 (permissions) explains the rwx letters that follow the type.

/proc and /sys: the kernel as files

/proc and /sys take no space on disk. They are virtual filesystems: every time you read a file, the kernel generates its content on the spot. That makes them the most honest source of truth about a running machine.

$ cat /proc/loadavg
0.08 0.03 0.01 1/187 2214
$ grep -E 'MemTotal|MemAvailable' /proc/meminfo
MemTotal:        2013784 kB
MemAvailable:    1689412 kB
$ cat /proc/sys/net/ipv4/ip_forward
0
$ ls /sys/class/net
ens3  lo
$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc
  • /proc/loadavg and /proc/meminfo are where uptime and free get their numbers.
  • Every running process has a directory /proc/<PID>/ with its command line, open files and memory use. /proc/self always points to the process reading it.
  • /proc/sys/ holds tunable kernel settings; ip_forward set to 0 means this machine does not route packets. The sysctl command reads and writes the same values.
  • /sys/class/net lists network interfaces (your names may differ), and /sys/fs/cgroup is where cgroup v2 limits CPU and memory for services and containers.

You will come back here in lesson 8 and lesson 19. Inside a container, /proc shows only the container’s own processes: namespaces at work.

Case sensitivity and hidden files

Two small rules cause a surprising number of tickets:

  • Names are case-sensitive. orbit.env, Orbit.env and ORBIT.ENV are three different files. Files copied from Windows or macOS (which usually ignore case) can break a deploy for this reason alone.
  • A leading dot hides a file. Names such as .bashrc, .profile and the .ssh/ directory do not appear in a plain ls. They are not secret or protected, just tucked away. Add -a to see them.
$ ls ~
notes.txt
$ ls -a ~
.  ..  .bash_history  .bash_logout  .bashrc  .cache  .profile  .ssh  notes.txt

. is the current directory and .. its parent; every directory contains both.

Red Hat and Alpine

Rocky Linux and RHEL use the same layout and also have a merged /usr. Two differences you will notice: nginx’s default web root is /usr/share/nginx/html, and SELinux labels files by location. Content in a non-standard path may be refused until it gets the right label (ls -Z shows labels), one more reason to follow the FHS. Alpine follows the same tree, but many of its commands are links to a single BusyBox binary; ls -l /bin/ls shows where yours points.

Practice lab

Orbit is about to move onto your lab machine. Before installing anything, the team wants a quick survey and the empty directory layout in place.

  1. Find the CPU model and the number of running processes using only files under /proc.
  2. Check which device holds / and how full it is, and which directory under /var uses the most space.
  3. Create Orbit’s directories: /srv/orbit with api, public and releases inside, plus /etc/orbit, /var/lib/orbit, /var/log/orbit and /var/backups/orbit.
  4. In your home directory, create report.txt, Report.txt and .report.txt, then show that they are three separate files and that one is hidden.
Show a solution

grep 'model name' /proc/cpuinfo prints the CPU, and the fourth field of /proc/loadavg (for example 1/187) shows running and total processes; ls -d /proc/[0-9]* lists one directory per process. df -h / shows the device and how full it is, and sudo du -sh /var/* gives the size of each directory under /var (usually lib, cache or log wins). Create the layout in one go with sudo mkdir -p /srv/orbit/{api,public,releases} /etc/orbit /var/lib/orbit /var/log/orbit /var/backups/orbit; the braces are a shell shortcut you meet in lesson 4. For the last task, touch ~/report.txt ~/Report.txt ~/.report.txt, then compare ls ~ (two files) with ls -a ~ (three).

Common mistakes

  • Putting your own scripts in /usr/bin. A package update may overwrite them. Use /usr/local/bin.
  • Keeping data in /tmp. It may live in RAM or be cleaned at boot.
  • Editing files under /proc/sys and expecting the change to survive a reboot. Live changes are temporary; persistent settings go in /etc/sysctl.d/.
  • Deleting big files in /var/lib to free space. That is usually a database or Docker’s storage. Clean through the owning tool instead.

Exercise

Leo installs a monitoring agent from a vendor tarball (not a package) on orbit-web-01. Where does it belong according to the FHS?

  1. /usr/bin, next to the other programs
  2. /opt (as a self-contained bundle) or /usr/local, with its configuration in /etc
  3. /tmp, so it can be removed easily
  4. /root, because only root runs it
Show answer

Answer: B. /usr is owned by the package manager and may be overwritten. Software you add by hand goes in /opt or /usr/local, and its settings still belong in /etc.

Exercise

orbit-db-01 has a 200 GB disk mounted on /var/lib/postgresql. Someone unmounts it by mistake and PostgreSQL starts writing. Where do the new files go?

  1. Nowhere; writing fails because the disk is gone
  2. Into the /var/lib/postgresql directory on the root disk, which may fill up the system disk
  3. Into /mnt, the default location for unmounted disks
  4. Onto the 200 GB disk anyway, because the path is the same
Show answer

Answer: B. A mount point is an ordinary directory. Without the disk mounted, writes land on whatever filesystem holds that directory, here the root disk. Lesson 11 shows how to make a service refuse to start when its disk is missing.

Exercise

A developer asks why /proc/meminfo is 0 bytes in ls -l yet cat prints dozens of lines. What is the correct explanation?

  1. The file is compressed on disk
  2. cat reads a cached copy from /var/cache
  3. /proc is a virtual filesystem; the kernel generates the content each time the file is read
  4. The file is a symbolic link to /sys/meminfo
Show answer

Answer: C. Files in /proc and /sys do not exist on disk. Their size is reported as 0 and the content is produced live, which is why they always show the current state.

Key takeaways

  • Linux has one tree starting at /; extra disks are mounted onto directories inside it.
  • /etc holds configuration, /var changing data and logs, /usr installed software, /usr/local and /opt what you add yourself, /srv what the server serves.
  • Orbit follows the FHS: code in /srv/orbit, config in /etc/orbit, data in /var/lib/orbit, logs in /var/log/orbit.
  • The first letter of ls -l shows the file type: -, d, l, c, b, s, p.
  • /proc and /sys are live views of the kernel; read them to learn the truth about a running system.
  • Names are case-sensitive and dotfiles are hidden from plain ls.

FAQ

What is the difference between /etc, /var and /usr?

/usr contains the programs and libraries installed by the package manager and should be treated as read-only. /etc contains the configuration that tells those programs how to behave, and it is what you edit. /var contains whatever the system produces while running: logs, databases, caches, queues. Backing up a server means mainly /etc and the relevant parts of /var.

Should I install my own software in /opt or /usr/local?

Use /usr/local (its bin, lib and share subdirectories) for small tools and scripts that fit the normal layout, like /usr/local/bin/orbit-backup.sh. Use /opt/<name> for self-contained bundles that keep everything in one folder, which is how many vendors ship agents. In both cases keep configuration in /etc and data in /var/lib.

Why does /bin point to /usr/bin?

Modern distributions merged the old split between /bin and /usr/bin, because /usr is no longer on a separate disk that might be missing at boot. The old paths stay as symbolic links so that scripts and programs that use them keep working. Ubuntu, Debian, Fedora and RHEL all use the merged layout.

Which directories should I check first when a disk is full?

Run df -h to see which filesystem is full, then sudo du -xh --max-depth=1 /var | sort -h to rank directories on it (-x stays on one filesystem). The usual suspects are /var/log, /var/lib/docker, database directories in /var/lib and /var/cache. Lesson 11 and lesson 18 cover cleaning up safely.

Is it safe to delete files in /tmp?

Usually, if they are old and not in use, but there is rarely a reason to do it by hand: the system cleans /tmp at boot or on a schedule through systemd-tmpfiles. Deleting a socket or lock file that a running program still uses can break that program, so let the system do it.

Why can I not see .bashrc or .ssh with ls?

Names that start with a dot are hidden by convention, which keeps configuration files out of everyday listings. Use ls -a to show everything, or ls -A to show hidden files without the . and .. entries. Hidden is not the same as protected: permissions decide who can read them.

↑ ↓ navigate Enter open Esc close