Out of the box, Kubernetes monitors almost nothing. There is no metric history, kubectl top does not work, container logs disappear with their Pods and events vanish after an hour. Everything in this lesson is something you install on purpose. The good news: a solid setup takes one Helm command and a few small manifests.
metrics-server is not a monitoring system
metrics-server collects the current CPU and memory usage from every kubelet and serves it through the Metrics API. Two things read that API: kubectl top and the Horizontal Pod Autoscaler.
$ kubectl top pods -n shop
NAME CPU(cores) MEMORY(bytes)
shop-api-6d9f7b8c4-kq2xv 85m 190Mi
shop-api-6d9f7b8c4-p8mzt 92m 201Mi
That is all it does. It keeps no history, cannot be queried and sends no alerts. If kubectl top fails or an HPA shows <unknown> targets, check that metrics-server is running:
$ kubectl get deployment metrics-server -n kube-system
NAME READY UP-TO-DATE AVAILABLE AGE
metrics-server 1/1 1 1 41d
Prometheus is the real answer
Do not assemble Prometheus piece by piece. Install kube-prometheus-stack, which brings Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics, already wired together with dashboards and alert rules:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install monitoring prometheus-community/kube-prometheus-stack \
--namespace monitoring --create-namespace
$ kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
Metrics come from three sources:
| Source | What it provides |
|---|---|
| kubelet (cAdvisor) | Real container usage: CPU, memory, network, filesystem |
| kube-state-metrics | Object state as the API sees it: desired vs ready replicas, Pod phase, restarts |
| node-exporter | Node operating system metrics: disk, load, filesystems |
kube-state-metrics is the one most people overlook, and it is the source of most useful alerts: “this Deployment wants 3 replicas and has 1” is a kube-state-metrics fact.
Scraping your own applications
Make your app expose metrics on an HTTP endpoint (most languages have a Prometheus client library), give the port a name in the Service and tell Prometheus about it with a ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: shop-api
namespace: shop
labels:
release: monitoring
spec:
selector:
matchLabels:
app: shop-api
endpoints:
- port: metrics
path: /metrics
interval: 30s
Two mistakes cause almost every “Prometheus scrapes nothing” problem:
portmust be the name of a port declared on the Service, not a number.- The ServiceMonitor’s labels must match what Prometheus selects. With kube-prometheus-stack that means
release: <your release name>.
Check the result on the Targets page:
$ kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090
Then open http://localhost:9090/targets and look for shop/shop-api.
What to watch
Two simple frameworks cover most needs:
- RED for applications: Rate of requests, Errors, Duration (latency).
- USE for resources: Utilization, Saturation, Errors.
At the cluster level, watch these signals:
| Signal | Why it matters |
|---|---|
| Container restart rate | Crash loops and OOM kills |
| Pods not Ready | Missing capacity behind your Services |
| Nodes NotReady | Lost machines |
| CPU throttling ratio | Hidden latency from CPU limits |
| Memory usage vs limit | OOMKilled coming soon |
| Requests vs allocatable | Room left for scheduling |
| PVC usage | Disks that are about to fill up |
| Certificate expiry | Outages that are completely avoidable |
Two PromQL queries to start with:
# Containers that restarted more than 3 times in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3
# Share of CPU periods that were throttled, per container
rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25
Alerts worth having
Alert on symptoms people feel, not on every number that moves. Every alert should mean “a human needs to act”.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-alerts
namespace: shop
labels:
release: monitoring
spec:
groups:
- name: shop
rules:
- alert: PodCrashLooping
expr: increase(kube_pod_container_status_restarts_total{namespace="shop"}[15m]) > 3
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.pod }} is restarting repeatedly"
- alert: DeploymentReplicasUnavailable
expr: kube_deployment_status_replicas_unavailable{namespace="shop"} > 0
for: 10m
labels:
severity: critical
annotations:
summary: "{{ $labels.deployment }} has unavailable replicas"
The for clause is what keeps you sane: the condition must stay true for that long before anyone is paged, so a short blip during a deploy does not wake you up.
A good starter set: crash-looping Pods, unavailable replicas, NotReady nodes, disks above 85%, certificates expiring within two weeks, and your application’s own error rate and latency targets.
Logs
Container logs live on the node and are deleted with the Pod. For live debugging, kubectl is enough:
$ kubectl logs shop-api-6d9f7b8c4-kq2xv
$ kubectl logs shop-api-6d9f7b8c4-kq2xv --previous
$ kubectl logs -l app=shop-api --tail=100 --prefix
$ kubectl logs deployment/shop-api --since=15m
To keep logs, run a log collector as a DaemonSet (one Pod per node). Loki with Grafana Alloy is the cheap, common partner for Prometheus and Grafana. Elasticsearch or OpenSearch fit when you need heavy full-text search. Either way, have your apps log to stdout in JSON so fields stay searchable.
Events expire
Events (Scheduled, Pulled, BackOff, FailedScheduling, …) are kept for only one hour by default. That is often not long enough to investigate last night’s incident.
$ kubectl get events -A --sort-by=.lastTimestamp
$ kubectl get events -n shop --field-selector involvedObject.name=shop-api-6d9f7b8c4-kq2xv
kube-prometheus-stack does not store events. Ship them to your log system with a tool such as an event exporter.
Tracing
Metrics tell you that something is slow. Traces tell you where: which service, which database call. OpenTelemetry is the standard way to instrument applications, with backends such as Grafana Tempo, Jaeger or a commercial tool. Add tracing once you have several services calling each other.
Quick reference
| Task | Command |
|---|---|
| Pod usage | kubectl top pods --containers |
| Node usage | kubectl top nodes |
| Logs | kubectl logs <pod> -c <container> |
| Logs of a crashed container | kubectl logs <pod> --previous |
| Logs by label | kubectl logs -l app=<label> --tail=100 --prefix |
| Recent events | kubectl get events -A --sort-by=.lastTimestamp |
| Open Prometheus | kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090 |
| Open Grafana | kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80 |
| metrics-server health | kubectl get deployment metrics-server -n kube-system |
Exercise
Which source tells you that a Deployment wants 3 replicas but only 1 is available?
Show answer
Answer: C. kube-state-metrics turns the state of Kubernetes objects into metrics. node-exporter and cAdvisor measure resource usage, and metrics-server only serves current CPU and memory.
Exercise
What does for: 10m do in a Prometheus alert rule?
Show answer
Answer: A. Short blips, like a Pod restarting once during a deploy, clear before the 10 minutes pass and never page anyone.
Key takeaways
- Kubernetes monitors almost nothing by default: install metrics, logs and alerting on purpose.
- metrics-server has no history; it serves
kubectl topand autoscaling only. - Install kube-prometheus-stack instead of assembling the parts.
- kube-state-metrics provides object state, the basis of most good alerts.
- ServiceMonitors fail on port names and label selectors.
- Watch CPU throttling: it causes latency nobody sees.
- Events expire after an hour; ship them and your logs somewhere durable.
- Alert on symptoms, and always use
for.
FAQ
Why does kubectl top not work?
metrics-server is missing or unhealthy. Check kubectl get deployment metrics-server -n kube-system and its logs. An HPA showing <unknown> targets usually has the same cause.
What is the difference between metrics-server and Prometheus?
metrics-server gives current CPU and memory for kubectl top and autoscaling. Prometheus stores history and supports queries, dashboards and alerts. Most clusters run both.
How do I keep logs after a Pod is deleted?
Run a log collector as a DaemonSet that ships container logs to a backend such as Loki or OpenSearch, and have apps log to stdout in JSON.
What should I alert on first?
Crash-looping Pods, unavailable replicas, NotReady nodes, disks above 85%, certificates close to expiry and your app’s error rate and latency. Always add a for duration so short blips do not page anyone.