Most “mysterious” Kubernetes problems are resource problems: Pods that will not schedule on a node that looks idle, containers that restart every few hours with clean logs, APIs that get slow without any errors. They all come from two settings, requests and limits, and from the fact that they do completely different things.
Requests and limits are different mechanisms
- Requests are read by the scheduler. They reserve capacity on a node. A Pod is placed only on a node with enough unreserved requests left.
- Limits are enforced at runtime by the kubelet and the Linux kernel (cgroups). They cap what the container can actually use.
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
memory: 512Mi
The scheduler adds up the requests on each node. It never looks at real usage. That is why a node can show 20% CPU usage and still refuse new Pods:
$ kubectl describe node lab-worker
...
Allocated resources:
Resource Requests Limits
-------- -------- ------
cpu 3850m (96%) 6 (150%)
memory 7Gi (90%) 11Gi (141%)
Limits above 100% are a deliberate overcommit. Requests near 100% mean the node is full, whatever the usage graphs say.
Units
- CPU is measured in cores.
1is one core,500m(500 millicores) is half a core,100mis a tenth. - Memory is measured in bytes.
MiandGiare binary units (1024-based).MandGare decimal (1000-based).128Mis about 122Mi, so stick toMiandGi.
The most important difference: CPU is compressible, memory is not. A container can be slowed down when it wants more CPU, but memory it already holds cannot be taken back without killing it.
What happens at the limit
Over the CPU limit: throttling. The kernel gives the container its quota in 100 ms periods. Once the quota is used, the container waits for the next period. Nothing crashes, but latency goes up. You can see it in the cgroup statistics:
$ kubectl exec shop-api-6d9f7b8c4-kq2xv -- cat /sys/fs/cgroup/cpu.stat
usage_usec 81234567
nr_periods 152340
nr_throttled 41872
throttled_usec 912345678
Here more than a quarter of all periods were throttled, which is a real cost in response time.
Over the memory limit: OOMKilled. The kernel kills the container immediately. The app never gets a chance to log an error, so the classic sign is a container that restarts every few hours with clean logs:
$ kubectl get pod shop-api-6d9f7b8c4-kq2xv \
-o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
OOMKilled
The CPU limit debate
Teams argue about CPU limits, and both sides have a point.
Against: throttling adds latency even when the node has idle CPU. Multi-threaded apps hit the quota fast: with 8 busy threads and a 1000m limit, the container uses its whole 100 ms quota in about 12 ms and then sits frozen for the remaining 88 ms.
For: without CPU limits, one misbehaving workload can starve its neighbours.
A practical middle ground:
- Always set memory limits.
- Always set CPU requests, which also give each container its fair share when CPU is contended.
- Be careful with CPU limits on latency-sensitive services. If you set them, set them well above the observed peak.
QoS classes and eviction
Kubernetes puts every Pod in a Quality of Service class, based on its requests and limits. When a node runs low on memory, the class decides who is evicted first.
| Class | Condition | Evicted |
|---|---|---|
| Guaranteed | Every container has requests equal to limits, for both CPU and memory | Last |
| Burstable | At least one request or limit is set, but not Guaranteed | In between |
| BestEffort | No requests or limits at all | First |
$ kubectl get pod shop-api-6d9f7b8c4-kq2xv -o jsonpath='{.status.qosClass}'
Burstable
BestEffort Pods are also scheduled blindly, since they reserve nothing. Every production container should at least be Burstable.
Namespace guardrails: LimitRange and ResourceQuota
A LimitRange gives containers in a namespace default values and maximums, so a forgotten resources block does not create a BestEffort Pod:
apiVersion: v1
kind: LimitRange
metadata:
name: shop-defaults
namespace: shop
spec:
limits:
- type: Container
defaultRequest:
cpu: 100m
memory: 128Mi
default:
memory: 512Mi
max:
cpu: "4"
memory: 8Gi
A ResourceQuota caps the total for the whole namespace:
apiVersion: v1
kind: ResourceQuota
metadata:
name: shop-quota
namespace: shop
spec:
hard:
requests.cpu: "16"
requests.memory: 32Gi
limits.memory: 64Gi
pods: "60"
Once a quota covers a resource, every Pod must declare it. Deploy the LimitRange with the quota so existing manifests keep working.
Choosing the numbers
Measure before you guess. kubectl top needs metrics-server and shows current usage:
$ kubectl top pods --containers -n shop
POD NAME CPU(cores) MEMORY(bytes)
shop-api-6d9f7b8c4-kq2xv api 85m 190Mi
shop-api-6d9f7b8c4-p8mzt api 92m 201Mi
A single reading misses daily peaks, so look at a week of history in Prometheus (lesson 13) when you can. Then:
- Memory request: steady-state usage plus about 25%.
- Memory limit: the request plus room for real peaks. Tell runtimes about it: JVM, Node.js and Go size their heaps from the memory they think they have.
- CPU request: the average usage.
- CPU limit: well above the peak, or none for latency-sensitive services.
The Vertical Pod Autoscaler in recommendation mode watches usage and suggests values without changing anything, which is a good second opinion.
Quick reference
| Task | Command |
|---|---|
| Container usage | kubectl top pods --containers |
| Node usage | kubectl top nodes |
| Requests vs capacity | kubectl describe node <name> (Allocated resources) |
| QoS class | kubectl get pod <name> -o jsonpath='{.status.qosClass}' |
| Was it OOMKilled? | kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' |
| CPU throttling | kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat |
| Quota usage | kubectl describe resourcequota -n <namespace> |
Exercise
Every container in a Pod has CPU and memory requests equal to its limits. Which QoS class does the Pod get?
Show answer
Answer: A. Requests equal to limits for both resources in every container means Guaranteed, the last class to be evicted under memory pressure. There is no “Critical” QoS class.
Exercise
A latency-sensitive API has a CPU limit of 500m. cpu.stat shows a high nr_throttled while the node is mostly idle. What is the best change?
Show answer
Answer: C. Throttling happens at the limit even when the node has spare CPU. The request still guarantees the container’s fair share; the limit is what is slowing it down.
Key takeaways
- The scheduler reads requests, not usage: a quiet node can still be full.
Miis notM.- CPU over the limit is throttled; memory over the limit is killed.
- Restarts every few hours with clean logs usually mean OOMKilled.
- Always set memory limits; be careful with CPU limits.
- No requests means BestEffort, which is evicted first; requests equal to limits gives Guaranteed.
- A ResourceQuota makes requests mandatory, so add a LimitRange with it.
FAQ
What is the difference between requests and limits?
Requests decide where a Pod can be placed and guarantee its share. Limits are the ceiling at runtime: CPU above the limit is throttled and memory above the limit gets the container killed.
Should I set CPU limits?
Always set CPU requests. For latency-sensitive services, many teams leave CPU limits off or set them well above the peak, because throttling causes slowness that is hard to see.
How do I fix OOMKilled?
First check real usage with kubectl top or Prometheus. If the limit is too tight, raise it; if usage keeps growing, look for a leak; and make sure the runtime’s heap size is based on the container limit.
Why does kubectl top not work?
It needs metrics-server. Install it, or enable it with minikube addons enable metrics-server. On kind you may need to add the --kubelet-insecure-tls flag to its container.