Lesson 11 advanced

Kubernetes Resource Management

Requests vs limits, CPU and memory units, throttling and OOMKilled, the CPU limits debate, QoS classes and eviction, LimitRange and ResourceQuota, and how to choose the right numbers.

Most “mysterious” Kubernetes problems are resource problems: Pods that will not schedule on a node that looks idle, containers that restart every few hours with clean logs, APIs that get slow without any errors. They all come from two settings, requests and limits, and from the fact that they do completely different things.

Requests and limits are different mechanisms

  • Requests are read by the scheduler. They reserve capacity on a node. A Pod is placed only on a node with enough unreserved requests left.
  • Limits are enforced at runtime by the kubelet and the Linux kernel (cgroups). They cap what the container can actually use.
resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi

The scheduler adds up the requests on each node. It never looks at real usage. That is why a node can show 20% CPU usage and still refuse new Pods:

$ kubectl describe node lab-worker
...
Allocated resources:
  Resource           Requests      Limits
  --------           --------      ------
  cpu                3850m (96%)   6 (150%)
  memory             7Gi (90%)     11Gi (141%)

Limits above 100% are a deliberate overcommit. Requests near 100% mean the node is full, whatever the usage graphs say.

Units

  • CPU is measured in cores. 1 is one core, 500m (500 millicores) is half a core, 100m is a tenth.
  • Memory is measured in bytes. Mi and Gi are binary units (1024-based). M and G are decimal (1000-based). 128M is about 122Mi, so stick to Mi and Gi.

The most important difference: CPU is compressible, memory is not. A container can be slowed down when it wants more CPU, but memory it already holds cannot be taken back without killing it.

What happens at the limit

Over the CPU limit: throttling. The kernel gives the container its quota in 100 ms periods. Once the quota is used, the container waits for the next period. Nothing crashes, but latency goes up. You can see it in the cgroup statistics:

$ kubectl exec shop-api-6d9f7b8c4-kq2xv -- cat /sys/fs/cgroup/cpu.stat
usage_usec 81234567
nr_periods 152340
nr_throttled 41872
throttled_usec 912345678

Here more than a quarter of all periods were throttled, which is a real cost in response time.

Over the memory limit: OOMKilled. The kernel kills the container immediately. The app never gets a chance to log an error, so the classic sign is a container that restarts every few hours with clean logs:

$ kubectl get pod shop-api-6d9f7b8c4-kq2xv \
    -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
OOMKilled

The CPU limit debate

Teams argue about CPU limits, and both sides have a point.

Against: throttling adds latency even when the node has idle CPU. Multi-threaded apps hit the quota fast: with 8 busy threads and a 1000m limit, the container uses its whole 100 ms quota in about 12 ms and then sits frozen for the remaining 88 ms.

For: without CPU limits, one misbehaving workload can starve its neighbours.

A practical middle ground:

  • Always set memory limits.
  • Always set CPU requests, which also give each container its fair share when CPU is contended.
  • Be careful with CPU limits on latency-sensitive services. If you set them, set them well above the observed peak.

QoS classes and eviction

Kubernetes puts every Pod in a Quality of Service class, based on its requests and limits. When a node runs low on memory, the class decides who is evicted first.

ClassConditionEvicted
GuaranteedEvery container has requests equal to limits, for both CPU and memoryLast
BurstableAt least one request or limit is set, but not GuaranteedIn between
BestEffortNo requests or limits at allFirst
$ kubectl get pod shop-api-6d9f7b8c4-kq2xv -o jsonpath='{.status.qosClass}'
Burstable

BestEffort Pods are also scheduled blindly, since they reserve nothing. Every production container should at least be Burstable.

Namespace guardrails: LimitRange and ResourceQuota

A LimitRange gives containers in a namespace default values and maximums, so a forgotten resources block does not create a BestEffort Pod:

apiVersion: v1
kind: LimitRange
metadata:
  name: shop-defaults
  namespace: shop
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      default:
        memory: 512Mi
      max:
        cpu: "4"
        memory: 8Gi

A ResourceQuota caps the total for the whole namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: shop-quota
  namespace: shop
spec:
  hard:
    requests.cpu: "16"
    requests.memory: 32Gi
    limits.memory: 64Gi
    pods: "60"

Once a quota covers a resource, every Pod must declare it. Deploy the LimitRange with the quota so existing manifests keep working.

Choosing the numbers

Measure before you guess. kubectl top needs metrics-server and shows current usage:

$ kubectl top pods --containers -n shop
POD                        NAME   CPU(cores)   MEMORY(bytes)
shop-api-6d9f7b8c4-kq2xv   api    85m          190Mi
shop-api-6d9f7b8c4-p8mzt   api    92m          201Mi

A single reading misses daily peaks, so look at a week of history in Prometheus (lesson 13) when you can. Then:

  • Memory request: steady-state usage plus about 25%.
  • Memory limit: the request plus room for real peaks. Tell runtimes about it: JVM, Node.js and Go size their heaps from the memory they think they have.
  • CPU request: the average usage.
  • CPU limit: well above the peak, or none for latency-sensitive services.

The Vertical Pod Autoscaler in recommendation mode watches usage and suggests values without changing anything, which is a good second opinion.

Quick reference

TaskCommand
Container usagekubectl top pods --containers
Node usagekubectl top nodes
Requests vs capacitykubectl describe node <name> (Allocated resources)
QoS classkubectl get pod <name> -o jsonpath='{.status.qosClass}'
Was it OOMKilled?kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
CPU throttlingkubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat
Quota usagekubectl describe resourcequota -n <namespace>

Exercise

Every container in a Pod has CPU and memory requests equal to its limits. Which QoS class does the Pod get?

  1. Guaranteed
  2. Burstable
  3. BestEffort
  4. Critical
Show answer

Answer: A. Requests equal to limits for both resources in every container means Guaranteed, the last class to be evicted under memory pressure. There is no “Critical” QoS class.

Exercise

A latency-sensitive API has a CPU limit of 500m. cpu.stat shows a high nr_throttled while the node is mostly idle. What is the best change?

  1. Lower its CPU request
  2. Raise its memory limit
  3. Raise the CPU limit well above the peak, or remove it, while keeping the CPU request
  4. Switch the image to a smaller base
Show answer

Answer: C. Throttling happens at the limit even when the node has spare CPU. The request still guarantees the container’s fair share; the limit is what is slowing it down.

Key takeaways

  • The scheduler reads requests, not usage: a quiet node can still be full.
  • Mi is not M.
  • CPU over the limit is throttled; memory over the limit is killed.
  • Restarts every few hours with clean logs usually mean OOMKilled.
  • Always set memory limits; be careful with CPU limits.
  • No requests means BestEffort, which is evicted first; requests equal to limits gives Guaranteed.
  • A ResourceQuota makes requests mandatory, so add a LimitRange with it.

FAQ

What is the difference between requests and limits?

Requests decide where a Pod can be placed and guarantee its share. Limits are the ceiling at runtime: CPU above the limit is throttled and memory above the limit gets the container killed.

Should I set CPU limits?

Always set CPU requests. For latency-sensitive services, many teams leave CPU limits off or set them well above the peak, because throttling causes slowness that is hard to see.

How do I fix OOMKilled?

First check real usage with kubectl top or Prometheus. If the limit is too tight, raise it; if usage keeps growing, look for a leak; and make sure the runtime’s heap size is based on the container limit.

Why does kubectl top not work?

It needs metrics-server. Install it, or enable it with minikube addons enable metrics-server. On kind you may need to add the --kubelet-insecure-tls flag to its container.

↑ ↓ navigate Enter open Esc close