Lesson 08 intermediate

ConfigMaps and Secrets

Keep configuration out of your images: ConfigMaps for settings, Secrets for credentials, environment variables vs mounted files, immutable versioned config and where real secret management lives.

The same image should run in development, staging and production. What changes between them is configuration: database addresses, feature flags, log levels, API keys. Kubernetes gives you two objects to supply it at runtime: ConfigMaps for ordinary settings and Secrets for sensitive values.

Why configuration does not belong in the image

If you bake DATABASE_URL into the image, the image only works in one environment. The thing you tested in staging is then not the thing you ship to production. Worse, rotating a password means building and deploying a new image. Keeping configuration outside the build is the third factor of the well-known twelve-factor app methodology.

ConfigMap or Secret?

ConfigMapSecret
ForNon-sensitive settingsPasswords, tokens, keys, certificates
Stored asPlain text in etcdBase64 in the API, optionally encrypted at rest
kubectl get -o yaml showsThe valuesBase64 that anyone can decode
Maximum size1 MiB1 MiB
Access controlIts own RBAC rulesIts own RBAC rules, and this is what matters

Important: base64 is an encoding, not encryption. echo cGFzc3dvcmQ= | base64 -d prints password. A Secret is a ConfigMap with safer defaults, not a vault.

What Secrets really give you: separate RBAC (people can read ConfigMaps without reading Secrets), values hidden from kubectl describe and most logs, optional encryption in etcd (often with a cloud KMS) and in-memory (tmpfs) volumes on the node. They do not protect anything from someone who is allowed to read Secrets in that namespace.

Creating them

$ kubectl create configmap shop-config --from-literal=LOG_LEVEL=info --from-file=app.yaml
configmap/shop-config created
$ kubectl create secret generic shop-db --from-literal=password='S3cure!pass'
secret/shop-db created
$ kubectl get secret shop-db -o jsonpath='{.data.password}' | base64 -d
S3cure!pass

The same ConfigMap as a manifest:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shop-config
data:
  LOG_LEVEL: info
  app.yaml: |
    cache:
      ttlSeconds: 300
    features:
      newCheckout: true

Two ways to use them, and one big difference

As environment variables:

containers:
  - name: api
    image: ghcr.io/jeffxl/shop-api:1.2.0
    env:
      - name: LOG_LEVEL
        valueFrom:
          configMapKeyRef:
            name: shop-config
            key: LOG_LEVEL
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: shop-db
            key: password

Use envFrom to load every key of a ConfigMap or Secret at once.

As files in a volume:

containers:
  - name: api
    image: ghcr.io/jeffxl/shop-api:1.2.0
    volumeMounts:
      - name: config
        mountPath: /etc/shop
        readOnly: true
volumes:
  - name: config
    configMap:
      name: shop-config

Each key becomes a file, so the app reads /etc/shop/app.yaml.

The difference that surprises everyone:

  • Mounted files update in running Pods, usually within a minute of changing the ConfigMap.
  • Environment variables never update. They are read once when the process starts.

After changing config that is used as environment variables, restart the Pods:

$ kubectl rollout restart deployment/shop-api
deployment.apps/shop-api restarted

Watch out: a volume mounted with subPath (a single file placed into an existing folder) never receives updates either.

Immutable, versioned configuration

Set immutable: true and Kubernetes refuses any change to the data:

apiVersion: v1
kind: ConfigMap
metadata:
  name: shop-config-v3
immutable: true
data:
  LOG_LEVEL: info

This has two benefits: nobody can change production config in place by accident, and kubelets stop watching the object, which takes load off the API server in big clusters. The pattern is to put a version in the name (shop-config-v4) and update the Deployment to point at it. The config change then becomes a normal rollout, with history and a rollback.

Where real secret management lives

Native Secrets are the delivery mechanism. The source of truth should live somewhere better:

  • External Secrets Operator syncs values from AWS Secrets Manager, Parameter Store, HashiCorp Vault, Azure Key Vault or Google Secret Manager into native Secrets. This is the most common choice.
  • Secrets Store CSI Driver mounts values straight from the external store as files, without storing them in etcd unless you ask it to.
  • Sealed Secrets encrypts Secrets with a key only the cluster has, so the encrypted file is safe to commit. Popular with GitOps.
  • Vault Agent Injector adds a sidecar that fetches secrets, including short-lived dynamic database credentials.

Rule: never commit a Secret manifest with real base64 values to Git. Anyone with read access to the repository has the password.

Quick reference

TaskCommand
ConfigMap from valueskubectl create configmap <name> --from-literal=KEY=value
ConfigMap from a filekubectl create configmap <name> --from-file=app.yaml
Secret from valueskubectl create secret generic <name> --from-literal=KEY=value
Registry credentialskubectl create secret docker-registry <name> --docker-server=ghcr.io --docker-username=<u> --docker-password=<token>
TLS certificatekubectl create secret tls <name> --cert=tls.crt --key=tls.key
Read a Secret valuekubectl get secret <name> -o jsonpath='{.data.KEY}' then decode with base64 -d
Apply env var changeskubectl rollout restart deployment/<name>

Exercise

An app reads /etc/shop/app.yaml from a mounted ConfigMap, without subPath. You change the ConfigMap. What happens?

  1. The file is updated inside the running Pods, usually within a minute; the app still has to re-read it
  2. Nothing until the Deployment is applied again
  3. Kubernetes restarts the Pods automatically
  4. The Pods fail because the file changed under them
Show answer

Answer: A. Mounted ConfigMaps are refreshed by the kubelet. Whether the new values take effect depends on the app watching or re-reading the file; environment variables, by contrast, never change.

Exercise

Why give a ConfigMap immutable: true and a version in its name, such as shop-config-v4?

  1. It encrypts the values in etcd
  2. It is required when a namespace has a ResourceQuota
  3. It blocks accidental in-place edits and turns each config change into a normal, reversible rollout
  4. It makes the ConfigMap available in every namespace
Show answer

Answer: C. Pointing the Deployment at a new versioned ConfigMap creates a new revision, so kubectl rollout undo also brings back the old config. Kubelets also stop watching immutable objects, which reduces API load.

Key takeaways

  • Keep configuration outside the image so one image runs everywhere.
  • ConfigMaps for settings, Secrets for credentials, and RBAC is the real protection.
  • Secrets are not encrypted by default; base64 is only an encoding.
  • Mounted files update live, environment variables do not. Restart Pods after env changes.
  • subPath mounts never update.
  • Prefer immutable, versioned ConfigMaps and Secrets.
  • Use External Secrets, Sealed Secrets or Vault, and never commit real Secrets to Git.

FAQ

Are Kubernetes Secrets encrypted?

Not by default. They are base64-encoded in the API and stored in etcd. Encryption at rest must be enabled on the API server (managed services usually offer it with a KMS key), and RBAC controls who can read them.

How do I store secrets in Git safely?

Commit only something that is useless without the cluster: a SealedSecret encrypted with the cluster’s key, or an ExternalSecret that only references a path in your secret manager.

Should I use environment variables or mounted files?

Files for anything structured or that may change while the app runs, because they update live. Environment variables for small values that only change with a deploy.

Is there a size limit?

Yes, 1 MiB per ConfigMap or Secret, because they live in etcd. Larger data belongs in object storage or a volume.

↑ ↓ navigate Enter open Esc close