Lesson 09 intermediate

Kubernetes Storage

Keep data alive beyond a Pod: emptyDir and hostPath, PersistentVolumes and claims, StorageClasses, access modes, reclaim policies, CSI drivers and how to debug a Pending PVC.

Pods are disposable, and so is everything a container writes to its own filesystem. Kubernetes storage objects let data outlive a Pod, and they let application teams ask for a disk without knowing whether it comes from AWS, Google Cloud or an NFS server.

Everything a container writes is temporary

A container’s writable layer is thrown away when the container restarts, not just when the Pod is deleted. A Pod that shows Running with RESTARTS 1 may already have lost every file it wrote.

$ kubectl exec shop-api-6d9f7b8c4-kq2xv -- sh -c 'echo hello > /tmp/note.txt'
$ kubectl exec shop-api-6d9f7b8c4-kq2xv -- kill 1
$ kubectl exec shop-api-6d9f7b8c4-kq2xv -- cat /tmp/note.txt
cat: can't open '/tmp/note.txt': No such file or directory

Storage options, from shortest-lived to longest-lived

OptionSurvives container restartSurvives Pod deletionUse it for
Container filesystemNoNoNothing you care about
emptyDirYesNoScratch space, caches, sharing files between containers
emptyDir in memoryYesNoFast tmpfs scratch, counts against the memory limit
hostPathYesYes, but only on that one nodeNode agents that need node files
PersistentVolumeClaimYesYesDatabases, uploads, anything durable

emptyDir

An emptyDir is created empty when the Pod lands on a node and deleted with the Pod. It is the standard way to share files between containers in one Pod, such as an app writing logs and a sidecar shipping them:

spec:
  containers:
    - name: api
      image: ghcr.io/jeffxl/shop-api:1.2.0
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
    - name: shipper
      image: fluent/fluent-bit:3.1
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
          readOnly: true
  volumes:
    - name: logs
      emptyDir:
        sizeLimit: 1Gi

Always set sizeLimit. Without it, a runaway process can fill the node’s disk and trigger evictions of unrelated Pods.

hostPath

hostPath mounts a folder of the node into the Pod. If the Pod moves to another node, the data stays behind. It is also a security risk: mounting / or the container runtime socket effectively gives the Pod control of the node. Keep it for node-level agents such as log collectors and monitoring exporters.

PersistentVolumes and PersistentVolumeClaims

Durable storage uses two objects:

  • A PersistentVolume (PV) is a piece of real storage, such as an EBS disk. It is cluster-wide and usually created automatically.
  • A PersistentVolumeClaim (PVC) is a request for storage in a namespace: “I need 20 GiB, read-write, of class gp3”.

The split lets the platform team decide how storage is provided while app teams only say what they need. The same manifest works on any cluster.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: orders-db
spec:
  containers:
    - name: postgres
      image: postgres:17
      env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: orders-db
              key: password
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: orders-data

PGDATA points at a subfolder because a fresh disk often contains a lost+found folder, and Postgres refuses to initialize a data directory that is not empty.

A StorageClass describes a type of storage and the driver that provisions it. When a PVC names a StorageClass, the matching PV is created on demand. That is dynamic provisioning, and it is why you almost never write PVs by hand.

$ kubectl get storageclass
NAME            PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION
gp3 (default)   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true
$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
orders-data   Bound    pvc-5d2c8e41-7a3b-4f6e-9c1d-2b8a7e6f4c30   20Gi       RWO            gp3

Access modes

ModeShortMeaning
ReadWriteOnceRWORead-write from a single node
ReadOnlyManyROXRead-only from many nodes
ReadWriteManyRWXRead-write from many nodes
ReadWriteOncePodRWOPRead-write from exactly one Pod

Two facts trip people up:

  • RWO is per node, not per Pod. Several Pods on the same node can share it; a Pod on another node cannot attach it.
  • Block storage (EBS, Google Persistent Disk, Azure Disk) only supports RWO. For shared read-write access you need a file system: EFS, Azure Files, Filestore or NFS.

If each replica needs its own disk, use a StatefulSet with volumeClaimTemplates instead of a Deployment sharing one PVC.

Reclaim policy: what happens when the claim is deleted

PolicyWhen the PVC is deleted
DeleteThe underlying disk is destroyed. The usual cloud default
RetainThe PV and its data are kept for an admin to handle
RecycleDeprecated, do not use

Warning: because Delete is the default on most clouds, deleting a namespace deletes its PVCs and destroys the disks. For data you care about, use a StorageClass with reclaimPolicy: Retain and take volume snapshots.

CSI drivers

Kubernetes does not talk to EBS or Ceph itself. Storage plugs in through the Container Storage Interface (CSI), with a driver installed per storage system, such as the EBS CSI driver on EKS. Drivers also provide snapshots and volume expansion:

$ kubectl get csidrivers
NAME              ATTACHREQUIRED   PODINFOONMOUNT
ebs.csi.aws.com   true             false
$ kubectl patch pvc orders-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'
persistentvolumeclaim/orders-data patched

Growing a volume needs allowVolumeExpansion: true on the StorageClass. Shrinking is never supported.

Debugging a Pending PVC

$ kubectl describe pvc orders-data
...
Events:
  Normal  WaitForFirstConsumer  waiting for first consumer to be created before binding

That event is normal. With volumeBindingMode: WaitForFirstConsumer, the disk is created only after a Pod using it is scheduled, so it lands in the same availability zone as the Pod. Real reasons a PVC stays Pending:

  • No storageClassName and no default StorageClass.
  • The CSI driver for that class is not installed.
  • The access mode is not supported, such as RWX on block storage.
  • A zone mismatch between an existing volume and where the Pod can run.

Quick reference

TaskCommand
List claimskubectl get pvc
Why a claim is Pendingkubectl describe pvc <name>
List volumeskubectl get pv
Storage classeskubectl get storageclass
Installed CSI driverskubectl get csidrivers
Which Pod uses a claimkubectl describe pvc <name> (the Used By line)

Exercise

An app container and a log-shipping sidecar must share a folder that can disappear with the Pod. Which volume fits?

  1. hostPath
  2. emptyDir
  3. A PersistentVolumeClaim
  4. A ConfigMap
Show answer

Answer: B. emptyDir is created with the Pod, is visible to all its containers and is deleted with it. Set a sizeLimit so it cannot fill the node’s disk.

Exercise

You need to grow PVC orders-data from 20Gi to 50Gi. What must be true?

  1. The access mode must be ReadWriteMany
  2. The reclaim policy must be Retain
  3. The Pod using it must be deleted first
  4. Its StorageClass must have allowVolumeExpansion: true
Show answer

Answer: D. With expansion allowed, you raise spec.resources.requests.storage and the CSI driver grows the disk, often while it is in use. Shrinking is never possible.

Key takeaways

  • The container filesystem is lost when the container restarts.
  • emptyDir lives as long as the Pod; always set sizeLimit.
  • hostPath is tied to one node and is a security risk; never use it for app data.
  • Write PVCs and let the StorageClass provision the PV.
  • RWO means one node, not one Pod; block storage cannot do RWX.
  • The default Delete reclaim policy means deleting a namespace can destroy data.
  • WaitForFirstConsumer Pending is expected, not a bug.

FAQ

What is the difference between a PV and a PVC?

A PV is the actual storage in the cluster. A PVC is a request for storage made in a namespace. With dynamic provisioning you write only the PVC and the PV is created for you.

Can several Pods share one volume?

It depends on the access mode and backend. RWO is shared only by Pods on the same node. Many nodes need RWX, which requires file storage such as EFS, Azure Files, Filestore or NFS.

What happens to my data when I delete a PVC?

The reclaim policy decides. With Delete the disk is destroyed; with Retain the PV and data are kept for manual recovery.

When should I use emptyDir?

For temporary data that only needs to live as long as the Pod: scratch files, caches and files shared between containers. Use medium: Memory if you need speed and the memory limit allows it.

↑ ↓ navigate Enter open Esc close