“No space left on device” is one of the most common pages a DevOps engineer receives, and it is rarely about buying more disk. A log that nobody rotates, a deleted file a process still holds open, a volume that was attached but never mounted: storage problems are usually small mistakes that took months to fill up. Knowing where the bytes live, and how disks become directories, turns a stressful outage into a ten-minute fix.
In this lesson you learn how Linux names block devices, how to measure space and inodes, how partitions, filesystems and mounts fit together, and how to make a mount permanent without risking the next boot. You then build a real filesystem on a loop device (a file pretending to be a disk), grow volumes with LVM and in the cloud, add swap, and walk through a disk-full incident on Orbit.
Note: this lesson needs the Multipass VM from lesson 1. Containers cannot create loop devices or mount filesystems without extra privileges.
From disk to directory
Four layers sit between a physical disk and the path /var/lib/orbit:
| Layer | What it is | Tools |
|---|---|---|
| Block device | A disk the kernel can read in fixed-size blocks | lsblk, blkid |
| Partition or logical volume | A slice of a disk (or of several disks, with LVM) | fdisk, parted, lvcreate |
| Filesystem | The structure that turns blocks into files, directories and permissions | mkfs.ext4, mkfs.xfs |
| Mount point | The directory where that filesystem appears in the single tree | mount, findmnt, /etc/fstab |
Linux has one tree starting at / (lesson 2). A second disk does not get a drive letter; you mount its filesystem on an empty directory and it becomes part of the tree.
How block devices are named
Every block device appears under /dev. The name tells you the driver, not the size or importance of the disk:
| Name | Where you see it | Partitions |
|---|---|---|
sda, sdb | SATA, SAS, USB, many hypervisors (Hyper-V, VMware) | sda1, sda2 |
nvme0n1 | NVMe drives; AWS Nitro and most modern cloud instances | nvme0n1p1 (note the p) |
vda, vdb | virtio disks on KVM, QEMU and OpenStack | vda1 |
xvda | Xen guests, older AWS instance types | xvda1 |
loop0 | A file used as a disk (snaps use these too) | loop0p1 |
On nvme0n1, 0 is the controller and n1 the namespace. Names are handed out in the order devices are discovered, so a disk that was nvme1n1 today can be nvme2n1 after a reboot. On AWS, a volume you attach as /dev/sdf shows up as an NVMe name. That is why configuration should never rely on these names, as you will see in the fstab section.
Inspect disks and filesystems
lsblk draws the tree of devices; -f adds filesystem type, label, UUID and usage:
$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 ext4 1.0 cloudimg-rootfs 3f1c2a9e-7d4b-4c55-9a51-0b8e6c2d1f44 6.1G 33% /
├─sda14
├─sda15 vfat FAT32 UEFI 7A1B-3C2D 98.2M 6% /boot/efi
└─sda16 ext4 1.0 BOOT b5e0c7d2-1a3f-4e8b-9c6d-2f4a8e1b7c90 731.4M 10% /boot
Your device names may be vda instead of sda, depending on how Multipass runs on your host, and you will also see loop devices used by snaps. sudo blkid /dev/sda1 prints the label, UUID and type of a single device in KEY="value" form, which is handy in scripts.
Space and inodes with df
df reports per filesystem: how big it is and how full. -h prints human units and -T adds the type:
$ df -hT / /boot
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 9.6G 3.1G 6.5G 33% /
/dev/sda16 ext4 881M 88M 731M 11% /boot
A filesystem can also run out of inodes: the fixed table of entries that describe files. Millions of tiny files (session files, cache fragments, mail queues) can exhaust inodes while gigabytes remain free, and every write fails with the same “No space left on device”. Check with -i:
$ df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 647168 121933 525235 19% /
Find what fills a disk with du
df answers “how full”; du answers “full of what”. -s summarises a directory, -h uses human units, -x stays on one filesystem and -d 1 stops one level deep. Pipe into sort -h, which understands K, M and G:
$ sudo du -xh -d 1 /var 2>/dev/null | sort -h | tail -5
48M /var/cache
210M /var/log
612M /var/lib
874M /var
Repeat on the biggest entry (/var/lib, then the biggest inside it) until you find the culprit. To list single large files, use find with a size filter: sudo find /var -xdev -type f -size +100M -exec ls -lh {} +. For interactive digging, install ncdu and run sudo ncdu -x /: it scans once, sorts by size and lets you walk the tree with the arrow keys.
Partitions and filesystems
A new disk is a blank row of blocks. You usually create a partition table (GPT today; MBR is limited to 2 TiB disks) and one or more partitions. Two tools do it:
fdisk /dev/nvme1n1is interactive:gcreates a GPT table,na new partition,wwrites and exits. Nothing changes untilw.partedcan run non-interactively, which suits scripts:sudo parted --script /dev/nvme1n1 mklabel gpt mkpart data ext4 0% 100%.
On cloud data volumes many teams skip partitioning and put a filesystem or LVM directly on the whole disk. Both work; just be consistent.
Choosing a filesystem
| Filesystem | Default on | Strengths | Watch out |
|---|---|---|---|
| ext4 | Ubuntu, Debian | Mature, fast fsck, can grow online and shrink offline | Fixed inode count chosen at mkfs time |
| XFS | RHEL, Rocky, Alma | Great with large files and parallel I/O, grows online | Cannot shrink, ever |
| Btrfs | Fedora Workstation, openSUSE | Snapshots, checksums, compression, subvolumes | More concepts to learn; tune before using it for databases |
For Orbit’s servers, ext4 on Ubuntu and XFS on Rocky are both sound choices. Create a filesystem with mkfs.<type>, for example sudo mkfs.ext4 -L orbit-data /dev/nvme1n1p1 or sudo mkfs.xfs -L orbit-data /dev/nvme1n1p1; -L sets a label.
Warning: mkfs erases whatever was on the device without asking twice. Run lsblk -f first and check that the target has no FSTYPE and no mount point. Typing nvme0n1 instead of nvme1n1 formats your root disk.
Mounting filesystems
mount DEVICE DIR attaches a filesystem; umount DIR (no “n” after the “u”) detaches it. findmnt shows what is mounted where: alone it prints a tree, findmnt /boot shows the source, type and options of one mount, and findmnt -t ext4,xfs filters by type.
Two surprises catch everyone once. Mounting over a directory that already contains files hides them until you unmount, they are not deleted. And umount refuses with “target is busy” while any process has a file open or a shell sits inside the directory: sudo fuser -vm /var/lib/orbit or sudo lsof +f -- /var/lib/orbit names the culprit.
Make it permanent with /etc/fstab
Mounts made with mount disappear at reboot. /etc/fstab lists filesystems to mount at boot, one per line with six fields:
# device mount point type options dump pass
UUID=0b9f3e1c-5a7d-4f2e-8c61-d4e2a9b7f310 /var/lib/orbit ext4 defaults,nofail 0 2
- device: always
UUID=(orLABEL=), never/dev/nvme1n1, because device names can change between boots. - options:
defaultsplusnofail, which tells systemd to keep booting if this disk is missing. Without it, a detached volume drops the server into emergency mode, and on a cloud VM without a console that is a very bad day. - dump: legacy, always
0. pass:fsckorder;1for/,2for other ext4 filesystems,0to skip (usual for XFS).
Never reboot to test fstab. Validate it, reload systemd and mount everything that is not mounted yet:
$ sudo findmnt --verify
Success, no errors or warnings detected
$ sudo systemctl daemon-reload
$ sudo mount -a
If mount -a prints an error, fix the line now, while you still have a working shell.
Hands-on: a data disk for Orbit on a loop device
Time to do it for real without risking a real disk. A loop device makes a regular file behave like a block device. You will give Orbit a dedicated 1 GiB filesystem at /var/lib/orbit.
If you skipped ahead, create the service account and directory first:
$ sudo useradd --system --home-dir /srv/orbit --shell /usr/sbin/nologin orbit
$ sudo mkdir -p /var/lib/orbit
Check that /var/lib/orbit is empty (sudo ls -A /var/lib/orbit); move anything there aside first, or it will be hidden by the mount. Then create the “disk”, attach it and format it:
$ sudo fallocate -l 1G /var/tmp/orbit-data.img
$ LOOP=$(sudo losetup --find --show /var/tmp/orbit-data.img)
$ sudo mkfs.ext4 -q -L orbit-data "$LOOP"
$ sudo mount "$LOOP" /var/lib/orbit
$ sudo chown orbit:orbit /var/lib/orbit
$ df -h /var/lib/orbit
Filesystem Size Used Avail Use% Mounted on
/dev/loop4 974M 24K 907M 1% /var/lib/orbit
fallocate -l 1G reserves a 1 GiB file instantly. losetup --find --show attaches it to the first free loop device and prints its name (here /dev/loop4), which the shell stores in $LOOP. The chown comes after mounting because the new filesystem’s root directory belongs to root. A fresh ext4 also contains a lost+found directory that fsck uses; leave it alone.
To mount it at boot, add a line to /etc/fstab. A loop file has no stable device to point a UUID at, so this entry names the file and adds the loop option, which sets up the loop device automatically:
/var/tmp/orbit-data.img /var/lib/orbit ext4 loop,nofail 0 2
Test it the safe way: sudo umount /var/lib/orbit, sudo losetup -d "$LOOP", then sudo findmnt --verify, sudo systemctl daemon-reload and sudo mount -a. findmnt /var/lib/orbit should show it back.
LVM: flexible volumes
Partitions are rigid. LVM (Logical Volume Manager) adds a layer so you can grow volumes, span several disks and take snapshots:
- Physical volume (PV): a disk or partition handed to LVM (
pvcreate). - Volume group (VG): a pool made of one or more PVs (
vgcreate,vgextend). - Logical volume (LV): a slice of the pool that you format and mount like a partition (
lvcreate,lvextend).
On orbit-db-01, a new 100 GB volume becomes a 60 GB volume for PostgreSQL, leaving room to grow:
$ sudo pvcreate /dev/nvme1n1
$ sudo vgcreate vg_data /dev/nvme1n1
$ sudo lvcreate -n pgdata -L 60G vg_data
$ sudo mkfs.xfs -L pgdata /dev/vg_data/pgdata
sudo vgs now reports the group’s size and about 40 GB in the VFree column. When the database grows, one command extends the volume and the filesystem, online. -r (--resizefs) runs the right grow tool for ext4 or XFS:
$ sudo lvextend -r -L +20G vg_data/pgdata
Use -l +100%FREE to take everything left in the group. Out of space in the VG? Attach another disk, pvcreate it and vgextend vg_data /dev/nvme2n1. The Ubuntu Server installer puts / on LVM (ubuntu-vg/ubuntu-lv) and often leaves part of the group unallocated: sudo vgs reveals free space you can claim with lvextend -r.
Growing a cloud disk
On a cloud VM you enlarge the volume in the console or with Terraform, but Linux does not use the extra space by itself. Three layers must grow in order: disk, partition, filesystem.
$ lsblk /dev/nvme0n1
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme0n1 259:0 0 60G 0 disk
└─nvme0n1p1 259:1 0 30G 0 part /
$ sudo growpart /dev/nvme0n1 1
CHANGED: partition=1 start=2048 old: size=62912479 end=62914527 new: size=125827039 end=125829087
$ sudo resize2fs /dev/nvme0n1p1
growpart (package cloud-guest-utils) takes the disk and the partition number, separated by a space. resize2fs grows ext4 using the device; for XFS use sudo xfs_growfs /, which takes the mount point. Both work while mounted. Ubuntu cloud images also grow the root partition automatically at boot through cloud-init, so a reboot sometimes does the job, but you rarely want to reboot production for it.
Swap files
Swap is disk space the kernel uses when RAM runs short. It will not make a server fast, but on small VMs it turns a sudden out-of-memory kill into a slowdown you can react to. Ubuntu cloud images ship without swap; adding a swap file takes a minute:
$ sudo fallocate -l 2G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
swapon --show and free -h confirm it. Make it permanent with the fstab line /swapfile none swap sw,nofail 0 0. The chmod 600 matters: swap can contain secrets from memory. Kubernetes nodes have traditionally run with swap off; recent releases support it, but only when configured on purpose.
Archives: tar, gzip and zip
Backups and releases move as archives. sudo tar -czf /var/backups/orbit/data.tar.gz -C / var/lib/orbit bundles a directory: -c creates, -z compresses with gzip (--zstd uses zstd, faster and often smaller), -f names the archive and -C / changes directory first, so paths inside stay relative. tar -tzf file lists the contents and tar -xzf file -C /tmp/restore extracts them; always list an archive from someone else before extracting it. gzip file compresses a single file in place; zip -r and unzip exist for sharing with Windows users. Lesson 12 turns tar into a proper backup script.
Red Hat and Alpine
- RHEL, Rocky and Alma default to XFS with root on LVM (
rhel-root); remember that XFS cannot shrink.growpartcomes fromcloud-utils-growpart. - Alpine’s BusyBox
dfandduaccept the common flags, butlsblk,findmntandmkfs.ext4need theutil-linuxande2fsprogspackages (apk add).
Incident: Orbit’s disk is full
At 09:20 the API on orbit-web-01 starts returning errors and the log shows “No space left on device”. Leo works through it calmly:
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 30G 30G 0 100% /
df -i / shows 10% of inodes used, so it is bytes. sudo du -xh -d 1 / | sort -h points to /var, then /var/log/orbit. A colleague already deleted a 14 GB api.log there an hour ago, yet df did not move. That is the clue: a deleted file keeps its space while any process holds it open.
$ sudo lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
python3 1873 orbit 3w REG 259,1 15032385536 0 77012 /var/log/orbit/api.log (deleted)
+L1 lists open files with fewer than one link, meaning deleted. Restarting the service releases the file: sudo systemctl restart orbit-api, and df -h / drops to 52%. If a restart is not acceptable, truncate it through the process instead: sudo truncate -s 0 /proc/1873/fd/3.
The fix is not done until it cannot recur: rotate /var/log/orbit/ with logrotate (lesson 18), cap the journal with journalctl --vacuum-size, alert at 80% disk and inode use, and next time empty a big log with truncate -s 0 instead of rm.
Practice lab
Orbit’s data now lives on its own filesystem. Prove you can manage it on orbit-lab:
- Create a second 512 MiB loop “disk”, format it as XFS with the label
orbit-archiveand mount it at/var/backups/orbit. - Add an fstab entry so it mounts at boot, and test it without rebooting.
- Fill it with a 300 MiB file, find that file using only
duandsort, then delete it whiletail -fholds it open and show the space is still used. - Add a 1 GiB swap file and confirm it with
swapon --show.
Show a solution
Create the file with sudo fallocate -l 512M /var/tmp/orbit-archive.img, attach it with LOOP=$(sudo losetup --find --show /var/tmp/orbit-archive.img), format with sudo mkfs.xfs -L orbit-archive "$LOOP" (install xfsprogs if it is missing) and mount with sudo mkdir -p /var/backups/orbit && sudo mount "$LOOP" /var/backups/orbit. For boot, add /var/tmp/orbit-archive.img /var/backups/orbit xfs loop,nofail 0 0 to /etc/fstab, unmount, detach with sudo losetup -d "$LOOP", then run sudo findmnt --verify, sudo systemctl daemon-reload and sudo mount -a. Create the big file with sudo fallocate -l 300M /var/backups/orbit/big.bin, find it with sudo du -ah /var/backups/orbit | sort -h | tail -3, start sudo tail -f /var/backups/orbit/big.bin in a second shell, delete the file, and df -h /var/backups/orbit still shows it used until you stop tail (sudo lsof +L1 lists it meanwhile). The swap file follows the same steps as in the lesson with -l 1G.
Exercise
Disk usage on orbit-web-02 is 97%. You delete a 9 GB log in /var/log/orbit/, but df still shows 97% while du reports far less. What is going on?
Show answer
Answer: B. Space is freed only when the last open handle closes. lsof +L1 shows deleted-but-open files; restarting the process, or truncating /proc/PID/fd/N, releases the space.
Exercise
An upload directory starts failing with “No space left on device”, yet df -h shows the filesystem only 40% full. What do you check next?
Show answer
Answer: D. Each file needs an inode, and ext4 fixes their number when the filesystem is created. df -i shows inode usage; the fix is cleaning up the small files (or recreating the filesystem with more inodes).
Exercise
You resized orbit-db-01’s root EBS volume from 30 GB to 60 GB. lsblk shows nvme0n1 at 60G, but df -h / still shows 30G on an ext4 root. What do you run?
Show answer
Answer: C. The disk grew, but the partition and the filesystem did not. growpart extends partition 1, resize2fs grows ext4 online (xfs_growfs / for XFS). mkfs would wipe the data.
Key takeaways
- Disks become directories through four layers: block device, partition or LV, filesystem, mount point.
df -handdf -itell you how full;du -xh -d 1 | sort -handncdutell you what with.- In
/etc/fstabuseUUID=andnofail, and test withfindmnt --verifyandmount -abefore any reboot. - LVM makes growth a single
lvextend -r; cloud disks grow withgrowpartplusresize2fsorxfs_growfs. - Deleted-but-open files keep their space:
lsof +L1, then restart or truncate.
FAQ
Why do df and du show different numbers?
They measure different things. du walks the directory tree and adds up the files it can see. df asks the filesystem how many blocks are allocated. Deleted files that are still open, files hidden under a mount point and filesystem metadata count for df but not for du. A large gap after deleting files almost always means a process still holds them; check with sudo lsof +L1.
How do I find out what is using all my disk space?
Start with df -h to find the full filesystem, then sudo du -xh -d 1 /mountpoint | sort -h and repeat on the biggest directory until you reach the culprit. sudo ncdu -x / does the same interactively. For single big files, sudo find / -xdev -type f -size +500M -exec ls -lh {} + is quick. The usual suspects are logs, old releases, container images in /var/lib/docker and forgotten backups.
Is it safe to delete files in /var/log?
Old rotated logs (.1, .gz) are safe to remove. For a log a service is still writing, empty it with sudo truncate -s 0 file instead of rm, so the space is released immediately and the service keeps writing to the same file. For the systemd journal, use sudo journalctl --vacuum-size=500M rather than deleting its files.
Should I use ext4 or XFS?
Use your distribution’s default unless you have a reason: ext4 on Ubuntu and Debian, XFS on RHEL and its clones. Both are fast, stable and grow online. ext4 can shrink while unmounted, XFS never can, which matters if you might need to give space back. XFS handles very large files and highly parallel writes particularly well.
How much swap does a server need?
There is no single rule anymore. Many teams add a small swap file (1 to 4 GB) to VMs as a safety buffer so that a memory spike causes a slowdown rather than an out-of-memory kill. Databases and latency-sensitive services are usually sized to never swap. Kubernetes nodes have traditionally disabled swap and only enable it deliberately.