Lesson 11 intermediate

Disks, Filesystems and Storage

How Linux names disks, how to see what is using space, create and mount filesystems safely, write fstab entries that survive reboots, grow LVM and cloud volumes, add swap and work through a real "disk full" incident.

“No space left on device” is one of the most common pages a DevOps engineer receives, and it is rarely about buying more disk. A log that nobody rotates, a deleted file a process still holds open, a volume that was attached but never mounted: storage problems are usually small mistakes that took months to fill up. Knowing where the bytes live, and how disks become directories, turns a stressful outage into a ten-minute fix.

In this lesson you learn how Linux names block devices, how to measure space and inodes, how partitions, filesystems and mounts fit together, and how to make a mount permanent without risking the next boot. You then build a real filesystem on a loop device (a file pretending to be a disk), grow volumes with LVM and in the cloud, add swap, and walk through a disk-full incident on Orbit.

Note: this lesson needs the Multipass VM from lesson 1. Containers cannot create loop devices or mount filesystems without extra privileges.

From disk to directory

Four layers sit between a physical disk and the path /var/lib/orbit:

LayerWhat it isTools
Block deviceA disk the kernel can read in fixed-size blockslsblk, blkid
Partition or logical volumeA slice of a disk (or of several disks, with LVM)fdisk, parted, lvcreate
FilesystemThe structure that turns blocks into files, directories and permissionsmkfs.ext4, mkfs.xfs
Mount pointThe directory where that filesystem appears in the single treemount, findmnt, /etc/fstab

Linux has one tree starting at / (lesson 2). A second disk does not get a drive letter; you mount its filesystem on an empty directory and it becomes part of the tree.

How block devices are named

Every block device appears under /dev. The name tells you the driver, not the size or importance of the disk:

NameWhere you see itPartitions
sda, sdbSATA, SAS, USB, many hypervisors (Hyper-V, VMware)sda1, sda2
nvme0n1NVMe drives; AWS Nitro and most modern cloud instancesnvme0n1p1 (note the p)
vda, vdbvirtio disks on KVM, QEMU and OpenStackvda1
xvdaXen guests, older AWS instance typesxvda1
loop0A file used as a disk (snaps use these too)loop0p1

On nvme0n1, 0 is the controller and n1 the namespace. Names are handed out in the order devices are discovered, so a disk that was nvme1n1 today can be nvme2n1 after a reboot. On AWS, a volume you attach as /dev/sdf shows up as an NVMe name. That is why configuration should never rely on these names, as you will see in the fstab section.

Inspect disks and filesystems

lsblk draws the tree of devices; -f adds filesystem type, label, UUID and usage:

$ lsblk -f
NAME    FSTYPE FSVER LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1  ext4   1.0   cloudimg-rootfs 3f1c2a9e-7d4b-4c55-9a51-0b8e6c2d1f44    6.1G    33% /
├─sda14
├─sda15 vfat   FAT32 UEFI            7A1B-3C2D                              98.2M     6% /boot/efi
└─sda16 ext4   1.0   BOOT            b5e0c7d2-1a3f-4e8b-9c6d-2f4a8e1b7c90  731.4M    10% /boot

Your device names may be vda instead of sda, depending on how Multipass runs on your host, and you will also see loop devices used by snaps. sudo blkid /dev/sda1 prints the label, UUID and type of a single device in KEY="value" form, which is handy in scripts.

Space and inodes with df

df reports per filesystem: how big it is and how full. -h prints human units and -T adds the type:

$ df -hT / /boot
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda1      ext4  9.6G  3.1G  6.5G  33% /
/dev/sda16     ext4  881M   88M  731M  11% /boot

A filesystem can also run out of inodes: the fixed table of entries that describe files. Millions of tiny files (session files, cache fragments, mail queues) can exhaust inodes while gigabytes remain free, and every write fails with the same “No space left on device”. Check with -i:

$ df -i /
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda1       647168 121933  525235   19% /

Find what fills a disk with du

df answers “how full”; du answers “full of what”. -s summarises a directory, -h uses human units, -x stays on one filesystem and -d 1 stops one level deep. Pipe into sort -h, which understands K, M and G:

$ sudo du -xh -d 1 /var 2>/dev/null | sort -h | tail -5
48M     /var/cache
210M    /var/log
612M    /var/lib
874M    /var

Repeat on the biggest entry (/var/lib, then the biggest inside it) until you find the culprit. To list single large files, use find with a size filter: sudo find /var -xdev -type f -size +100M -exec ls -lh {} +. For interactive digging, install ncdu and run sudo ncdu -x /: it scans once, sorts by size and lets you walk the tree with the arrow keys.

Partitions and filesystems

A new disk is a blank row of blocks. You usually create a partition table (GPT today; MBR is limited to 2 TiB disks) and one or more partitions. Two tools do it:

  • fdisk /dev/nvme1n1 is interactive: g creates a GPT table, n a new partition, w writes and exits. Nothing changes until w.
  • parted can run non-interactively, which suits scripts: sudo parted --script /dev/nvme1n1 mklabel gpt mkpart data ext4 0% 100%.

On cloud data volumes many teams skip partitioning and put a filesystem or LVM directly on the whole disk. Both work; just be consistent.

Choosing a filesystem

FilesystemDefault onStrengthsWatch out
ext4Ubuntu, DebianMature, fast fsck, can grow online and shrink offlineFixed inode count chosen at mkfs time
XFSRHEL, Rocky, AlmaGreat with large files and parallel I/O, grows onlineCannot shrink, ever
BtrfsFedora Workstation, openSUSESnapshots, checksums, compression, subvolumesMore concepts to learn; tune before using it for databases

For Orbit’s servers, ext4 on Ubuntu and XFS on Rocky are both sound choices. Create a filesystem with mkfs.<type>, for example sudo mkfs.ext4 -L orbit-data /dev/nvme1n1p1 or sudo mkfs.xfs -L orbit-data /dev/nvme1n1p1; -L sets a label.

Warning: mkfs erases whatever was on the device without asking twice. Run lsblk -f first and check that the target has no FSTYPE and no mount point. Typing nvme0n1 instead of nvme1n1 formats your root disk.

Mounting filesystems

mount DEVICE DIR attaches a filesystem; umount DIR (no “n” after the “u”) detaches it. findmnt shows what is mounted where: alone it prints a tree, findmnt /boot shows the source, type and options of one mount, and findmnt -t ext4,xfs filters by type.

Two surprises catch everyone once. Mounting over a directory that already contains files hides them until you unmount, they are not deleted. And umount refuses with “target is busy” while any process has a file open or a shell sits inside the directory: sudo fuser -vm /var/lib/orbit or sudo lsof +f -- /var/lib/orbit names the culprit.

Make it permanent with /etc/fstab

Mounts made with mount disappear at reboot. /etc/fstab lists filesystems to mount at boot, one per line with six fields:

# device                                   mount point     type  options          dump  pass
UUID=0b9f3e1c-5a7d-4f2e-8c61-d4e2a9b7f310  /var/lib/orbit  ext4  defaults,nofail   0     2
  • device: always UUID= (or LABEL=), never /dev/nvme1n1, because device names can change between boots.
  • options: defaults plus nofail, which tells systemd to keep booting if this disk is missing. Without it, a detached volume drops the server into emergency mode, and on a cloud VM without a console that is a very bad day.
  • dump: legacy, always 0. pass: fsck order; 1 for /, 2 for other ext4 filesystems, 0 to skip (usual for XFS).

Never reboot to test fstab. Validate it, reload systemd and mount everything that is not mounted yet:

$ sudo findmnt --verify
Success, no errors or warnings detected
$ sudo systemctl daemon-reload
$ sudo mount -a

If mount -a prints an error, fix the line now, while you still have a working shell.

Hands-on: a data disk for Orbit on a loop device

Time to do it for real without risking a real disk. A loop device makes a regular file behave like a block device. You will give Orbit a dedicated 1 GiB filesystem at /var/lib/orbit.

If you skipped ahead, create the service account and directory first:

$ sudo useradd --system --home-dir /srv/orbit --shell /usr/sbin/nologin orbit
$ sudo mkdir -p /var/lib/orbit

Check that /var/lib/orbit is empty (sudo ls -A /var/lib/orbit); move anything there aside first, or it will be hidden by the mount. Then create the “disk”, attach it and format it:

$ sudo fallocate -l 1G /var/tmp/orbit-data.img
$ LOOP=$(sudo losetup --find --show /var/tmp/orbit-data.img)
$ sudo mkfs.ext4 -q -L orbit-data "$LOOP"
$ sudo mount "$LOOP" /var/lib/orbit
$ sudo chown orbit:orbit /var/lib/orbit
$ df -h /var/lib/orbit
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop4      974M   24K  907M   1% /var/lib/orbit

fallocate -l 1G reserves a 1 GiB file instantly. losetup --find --show attaches it to the first free loop device and prints its name (here /dev/loop4), which the shell stores in $LOOP. The chown comes after mounting because the new filesystem’s root directory belongs to root. A fresh ext4 also contains a lost+found directory that fsck uses; leave it alone.

To mount it at boot, add a line to /etc/fstab. A loop file has no stable device to point a UUID at, so this entry names the file and adds the loop option, which sets up the loop device automatically:

/var/tmp/orbit-data.img  /var/lib/orbit  ext4  loop,nofail  0  2

Test it the safe way: sudo umount /var/lib/orbit, sudo losetup -d "$LOOP", then sudo findmnt --verify, sudo systemctl daemon-reload and sudo mount -a. findmnt /var/lib/orbit should show it back.

LVM: flexible volumes

Partitions are rigid. LVM (Logical Volume Manager) adds a layer so you can grow volumes, span several disks and take snapshots:

  • Physical volume (PV): a disk or partition handed to LVM (pvcreate).
  • Volume group (VG): a pool made of one or more PVs (vgcreate, vgextend).
  • Logical volume (LV): a slice of the pool that you format and mount like a partition (lvcreate, lvextend).

On orbit-db-01, a new 100 GB volume becomes a 60 GB volume for PostgreSQL, leaving room to grow:

$ sudo pvcreate /dev/nvme1n1
$ sudo vgcreate vg_data /dev/nvme1n1
$ sudo lvcreate -n pgdata -L 60G vg_data
$ sudo mkfs.xfs -L pgdata /dev/vg_data/pgdata

sudo vgs now reports the group’s size and about 40 GB in the VFree column. When the database grows, one command extends the volume and the filesystem, online. -r (--resizefs) runs the right grow tool for ext4 or XFS:

$ sudo lvextend -r -L +20G vg_data/pgdata

Use -l +100%FREE to take everything left in the group. Out of space in the VG? Attach another disk, pvcreate it and vgextend vg_data /dev/nvme2n1. The Ubuntu Server installer puts / on LVM (ubuntu-vg/ubuntu-lv) and often leaves part of the group unallocated: sudo vgs reveals free space you can claim with lvextend -r.

Growing a cloud disk

On a cloud VM you enlarge the volume in the console or with Terraform, but Linux does not use the extra space by itself. Three layers must grow in order: disk, partition, filesystem.

$ lsblk /dev/nvme0n1
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
nvme0n1     259:0    0   60G  0 disk
└─nvme0n1p1 259:1    0   30G  0 part /
$ sudo growpart /dev/nvme0n1 1
CHANGED: partition=1 start=2048 old: size=62912479 end=62914527 new: size=125827039 end=125829087
$ sudo resize2fs /dev/nvme0n1p1

growpart (package cloud-guest-utils) takes the disk and the partition number, separated by a space. resize2fs grows ext4 using the device; for XFS use sudo xfs_growfs /, which takes the mount point. Both work while mounted. Ubuntu cloud images also grow the root partition automatically at boot through cloud-init, so a reboot sometimes does the job, but you rarely want to reboot production for it.

Swap files

Swap is disk space the kernel uses when RAM runs short. It will not make a server fast, but on small VMs it turns a sudden out-of-memory kill into a slowdown you can react to. Ubuntu cloud images ship without swap; adding a swap file takes a minute:

$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

swapon --show and free -h confirm it. Make it permanent with the fstab line /swapfile none swap sw,nofail 0 0. The chmod 600 matters: swap can contain secrets from memory. Kubernetes nodes have traditionally run with swap off; recent releases support it, but only when configured on purpose.

Archives: tar, gzip and zip

Backups and releases move as archives. sudo tar -czf /var/backups/orbit/data.tar.gz -C / var/lib/orbit bundles a directory: -c creates, -z compresses with gzip (--zstd uses zstd, faster and often smaller), -f names the archive and -C / changes directory first, so paths inside stay relative. tar -tzf file lists the contents and tar -xzf file -C /tmp/restore extracts them; always list an archive from someone else before extracting it. gzip file compresses a single file in place; zip -r and unzip exist for sharing with Windows users. Lesson 12 turns tar into a proper backup script.

Red Hat and Alpine

  • RHEL, Rocky and Alma default to XFS with root on LVM (rhel-root); remember that XFS cannot shrink. growpart comes from cloud-utils-growpart.
  • Alpine’s BusyBox df and du accept the common flags, but lsblk, findmnt and mkfs.ext4 need the util-linux and e2fsprogs packages (apk add).

Incident: Orbit’s disk is full

At 09:20 the API on orbit-web-01 starts returning errors and the log shows “No space left on device”. Leo works through it calmly:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1   30G   30G     0 100% /

df -i / shows 10% of inodes used, so it is bytes. sudo du -xh -d 1 / | sort -h points to /var, then /var/log/orbit. A colleague already deleted a 14 GB api.log there an hour ago, yet df did not move. That is the clue: a deleted file keeps its space while any process holds it open.

$ sudo lsof +L1
COMMAND   PID  USER  FD   TYPE DEVICE    SIZE/OFF NLINK  NODE NAME
python3  1873 orbit   3w   REG  259,1 15032385536     0 77012 /var/log/orbit/api.log (deleted)

+L1 lists open files with fewer than one link, meaning deleted. Restarting the service releases the file: sudo systemctl restart orbit-api, and df -h / drops to 52%. If a restart is not acceptable, truncate it through the process instead: sudo truncate -s 0 /proc/1873/fd/3.

The fix is not done until it cannot recur: rotate /var/log/orbit/ with logrotate (lesson 18), cap the journal with journalctl --vacuum-size, alert at 80% disk and inode use, and next time empty a big log with truncate -s 0 instead of rm.

Practice lab

Orbit’s data now lives on its own filesystem. Prove you can manage it on orbit-lab:

  1. Create a second 512 MiB loop “disk”, format it as XFS with the label orbit-archive and mount it at /var/backups/orbit.
  2. Add an fstab entry so it mounts at boot, and test it without rebooting.
  3. Fill it with a 300 MiB file, find that file using only du and sort, then delete it while tail -f holds it open and show the space is still used.
  4. Add a 1 GiB swap file and confirm it with swapon --show.
Show a solution

Create the file with sudo fallocate -l 512M /var/tmp/orbit-archive.img, attach it with LOOP=$(sudo losetup --find --show /var/tmp/orbit-archive.img), format with sudo mkfs.xfs -L orbit-archive "$LOOP" (install xfsprogs if it is missing) and mount with sudo mkdir -p /var/backups/orbit && sudo mount "$LOOP" /var/backups/orbit. For boot, add /var/tmp/orbit-archive.img /var/backups/orbit xfs loop,nofail 0 0 to /etc/fstab, unmount, detach with sudo losetup -d "$LOOP", then run sudo findmnt --verify, sudo systemctl daemon-reload and sudo mount -a. Create the big file with sudo fallocate -l 300M /var/backups/orbit/big.bin, find it with sudo du -ah /var/backups/orbit | sort -h | tail -3, start sudo tail -f /var/backups/orbit/big.bin in a second shell, delete the file, and df -h /var/backups/orbit still shows it used until you stop tail (sudo lsof +L1 lists it meanwhile). The swap file follows the same steps as in the lesson with -l 1G.

Exercise

Disk usage on orbit-web-02 is 97%. You delete a 9 GB log in /var/log/orbit/, but df still shows 97% while du reports far less. What is going on?

  1. The filesystem needs fsck before freed space is counted
  2. A running process still has the deleted file open; find it with lsof +L1 and restart or truncate through /proc
  3. df caches its results for an hour
  4. The inode table is full, so the space cannot be released
Show answer

Answer: B. Space is freed only when the last open handle closes. lsof +L1 shows deleted-but-open files; restarting the process, or truncating /proc/PID/fd/N, releases the space.

Exercise

An upload directory starts failing with “No space left on device”, yet df -h shows the filesystem only 40% full. What do you check next?

  1. The swap file, because it is probably full
  2. The partition table, because it may be MBR
  3. Whether the disk is NVMe or SATA
  4. df -i, because millions of small files may have exhausted the inodes
Show answer

Answer: D. Each file needs an inode, and ext4 fixes their number when the filesystem is created. df -i shows inode usage; the fix is cleaning up the small files (or recreating the filesystem with more inodes).

Exercise

You resized orbit-db-01’s root EBS volume from 30 GB to 60 GB. lsblk shows nvme0n1 at 60G, but df -h / still shows 30G on an ext4 root. What do you run?

  1. mkfs.ext4 /dev/nvme0n1p1 to rebuild it at the new size
  2. lvextend -r on /dev/nvme0n1
  3. growpart /dev/nvme0n1 1, then resize2fs /dev/nvme0n1p1
  4. Nothing; df updates after the next cron run
Show answer

Answer: C. The disk grew, but the partition and the filesystem did not. growpart extends partition 1, resize2fs grows ext4 online (xfs_growfs / for XFS). mkfs would wipe the data.

Key takeaways

  • Disks become directories through four layers: block device, partition or LV, filesystem, mount point.
  • df -h and df -i tell you how full; du -xh -d 1 | sort -h and ncdu tell you what with.
  • In /etc/fstab use UUID= and nofail, and test with findmnt --verify and mount -a before any reboot.
  • LVM makes growth a single lvextend -r; cloud disks grow with growpart plus resize2fs or xfs_growfs.
  • Deleted-but-open files keep their space: lsof +L1, then restart or truncate.

FAQ

Why do df and du show different numbers?

They measure different things. du walks the directory tree and adds up the files it can see. df asks the filesystem how many blocks are allocated. Deleted files that are still open, files hidden under a mount point and filesystem metadata count for df but not for du. A large gap after deleting files almost always means a process still holds them; check with sudo lsof +L1.

How do I find out what is using all my disk space?

Start with df -h to find the full filesystem, then sudo du -xh -d 1 /mountpoint | sort -h and repeat on the biggest directory until you reach the culprit. sudo ncdu -x / does the same interactively. For single big files, sudo find / -xdev -type f -size +500M -exec ls -lh {} + is quick. The usual suspects are logs, old releases, container images in /var/lib/docker and forgotten backups.

Is it safe to delete files in /var/log?

Old rotated logs (.1, .gz) are safe to remove. For a log a service is still writing, empty it with sudo truncate -s 0 file instead of rm, so the space is released immediately and the service keeps writing to the same file. For the systemd journal, use sudo journalctl --vacuum-size=500M rather than deleting its files.

Should I use ext4 or XFS?

Use your distribution’s default unless you have a reason: ext4 on Ubuntu and Debian, XFS on RHEL and its clones. Both are fast, stable and grow online. ext4 can shrink while unmounted, XFS never can, which matters if you might need to give space back. XFS handles very large files and highly parallel writes particularly well.

How much swap does a server need?

There is no single rule anymore. Many teams add a small swap file (1 to 4 GB) to VMs as a safety buffer so that a memory spike causes a slowdown rather than an out-of-memory kill. Databases and latency-sensitive services are usually sized to never swap. Kubernetes nodes have traditionally disabled swap and only enable it deliberately.

↑ ↓ navigate Enter open Esc close