Lesson 13 advanced

Kubernetes Monitoring

Turn a silent cluster into an observable one: metrics-server and kubectl top, Prometheus with kube-prometheus-stack, ServiceMonitors, what to watch, alerts worth having, logs, events and tracing.

Out of the box, Kubernetes monitors almost nothing. There is no metric history, kubectl top does not work, container logs disappear with their Pods and events vanish after an hour. Everything in this lesson is something you install on purpose. The good news: a solid setup takes one Helm command and a few small manifests.

metrics-server is not a monitoring system

metrics-server collects the current CPU and memory usage from every kubelet and serves it through the Metrics API. Two things read that API: kubectl top and the Horizontal Pod Autoscaler.

$ kubectl top pods -n shop
NAME                       CPU(cores)   MEMORY(bytes)
shop-api-6d9f7b8c4-kq2xv   85m          190Mi
shop-api-6d9f7b8c4-p8mzt   92m          201Mi

That is all it does. It keeps no history, cannot be queried and sends no alerts. If kubectl top fails or an HPA shows <unknown> targets, check that metrics-server is running:

$ kubectl get deployment metrics-server -n kube-system
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           41d

Prometheus is the real answer

Do not assemble Prometheus piece by piece. Install kube-prometheus-stack, which brings Prometheus, Alertmanager, Grafana, node-exporter and kube-state-metrics, already wired together with dashboards and alert rules:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install monitoring prometheus-community/kube-prometheus-stack \
    --namespace monitoring --create-namespace
$ kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

Metrics come from three sources:

SourceWhat it provides
kubelet (cAdvisor)Real container usage: CPU, memory, network, filesystem
kube-state-metricsObject state as the API sees it: desired vs ready replicas, Pod phase, restarts
node-exporterNode operating system metrics: disk, load, filesystems

kube-state-metrics is the one most people overlook, and it is the source of most useful alerts: “this Deployment wants 3 replicas and has 1” is a kube-state-metrics fact.

Scraping your own applications

Make your app expose metrics on an HTTP endpoint (most languages have a Prometheus client library), give the port a name in the Service and tell Prometheus about it with a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: shop-api
  namespace: shop
  labels:
    release: monitoring
spec:
  selector:
    matchLabels:
      app: shop-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s

Two mistakes cause almost every “Prometheus scrapes nothing” problem:

  • port must be the name of a port declared on the Service, not a number.
  • The ServiceMonitor’s labels must match what Prometheus selects. With kube-prometheus-stack that means release: <your release name>.

Check the result on the Targets page:

$ kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090

Then open http://localhost:9090/targets and look for shop/shop-api.

What to watch

Two simple frameworks cover most needs:

  • RED for applications: Rate of requests, Errors, Duration (latency).
  • USE for resources: Utilization, Saturation, Errors.

At the cluster level, watch these signals:

SignalWhy it matters
Container restart rateCrash loops and OOM kills
Pods not ReadyMissing capacity behind your Services
Nodes NotReadyLost machines
CPU throttling ratioHidden latency from CPU limits
Memory usage vs limitOOMKilled coming soon
Requests vs allocatableRoom left for scheduling
PVC usageDisks that are about to fill up
Certificate expiryOutages that are completely avoidable

Two PromQL queries to start with:

# Containers that restarted more than 3 times in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 3

# Share of CPU periods that were throttled, per container
rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]) > 0.25

Alerts worth having

Alert on symptoms people feel, not on every number that moves. Every alert should mean “a human needs to act”.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: shop-alerts
  namespace: shop
  labels:
    release: monitoring
spec:
  groups:
    - name: shop
      rules:
        - alert: PodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total{namespace="shop"}[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.pod }} is restarting repeatedly"
        - alert: DeploymentReplicasUnavailable
          expr: kube_deployment_status_replicas_unavailable{namespace="shop"} > 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.deployment }} has unavailable replicas"

The for clause is what keeps you sane: the condition must stay true for that long before anyone is paged, so a short blip during a deploy does not wake you up.

A good starter set: crash-looping Pods, unavailable replicas, NotReady nodes, disks above 85%, certificates expiring within two weeks, and your application’s own error rate and latency targets.

Logs

Container logs live on the node and are deleted with the Pod. For live debugging, kubectl is enough:

$ kubectl logs shop-api-6d9f7b8c4-kq2xv
$ kubectl logs shop-api-6d9f7b8c4-kq2xv --previous
$ kubectl logs -l app=shop-api --tail=100 --prefix
$ kubectl logs deployment/shop-api --since=15m

To keep logs, run a log collector as a DaemonSet (one Pod per node). Loki with Grafana Alloy is the cheap, common partner for Prometheus and Grafana. Elasticsearch or OpenSearch fit when you need heavy full-text search. Either way, have your apps log to stdout in JSON so fields stay searchable.

Events expire

Events (Scheduled, Pulled, BackOff, FailedScheduling, …) are kept for only one hour by default. That is often not long enough to investigate last night’s incident.

$ kubectl get events -A --sort-by=.lastTimestamp
$ kubectl get events -n shop --field-selector involvedObject.name=shop-api-6d9f7b8c4-kq2xv

kube-prometheus-stack does not store events. Ship them to your log system with a tool such as an event exporter.

Tracing

Metrics tell you that something is slow. Traces tell you where: which service, which database call. OpenTelemetry is the standard way to instrument applications, with backends such as Grafana Tempo, Jaeger or a commercial tool. Add tracing once you have several services calling each other.

Quick reference

TaskCommand
Pod usagekubectl top pods --containers
Node usagekubectl top nodes
Logskubectl logs <pod> -c <container>
Logs of a crashed containerkubectl logs <pod> --previous
Logs by labelkubectl logs -l app=<label> --tail=100 --prefix
Recent eventskubectl get events -A --sort-by=.lastTimestamp
Open Prometheuskubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090
Open Grafanakubectl port-forward -n monitoring svc/monitoring-grafana 3000:80
metrics-server healthkubectl get deployment metrics-server -n kube-system

Exercise

Which source tells you that a Deployment wants 3 replicas but only 1 is available?

  1. node-exporter
  2. metrics-server
  3. kube-state-metrics
  4. cAdvisor in the kubelet
Show answer

Answer: C. kube-state-metrics turns the state of Kubernetes objects into metrics. node-exporter and cAdvisor measure resource usage, and metrics-server only serves current CPU and memory.

Exercise

What does for: 10m do in a Prometheus alert rule?

  1. The condition must stay true for 10 minutes before the alert fires
  2. The alert repeats every 10 minutes
  3. The alert resolves itself after 10 minutes
  4. Prometheus scrapes the metric every 10 minutes
Show answer

Answer: A. Short blips, like a Pod restarting once during a deploy, clear before the 10 minutes pass and never page anyone.

Key takeaways

  • Kubernetes monitors almost nothing by default: install metrics, logs and alerting on purpose.
  • metrics-server has no history; it serves kubectl top and autoscaling only.
  • Install kube-prometheus-stack instead of assembling the parts.
  • kube-state-metrics provides object state, the basis of most good alerts.
  • ServiceMonitors fail on port names and label selectors.
  • Watch CPU throttling: it causes latency nobody sees.
  • Events expire after an hour; ship them and your logs somewhere durable.
  • Alert on symptoms, and always use for.

FAQ

Why does kubectl top not work?

metrics-server is missing or unhealthy. Check kubectl get deployment metrics-server -n kube-system and its logs. An HPA showing <unknown> targets usually has the same cause.

What is the difference between metrics-server and Prometheus?

metrics-server gives current CPU and memory for kubectl top and autoscaling. Prometheus stores history and supports queries, dashboards and alerts. Most clusters run both.

How do I keep logs after a Pod is deleted?

Run a log collector as a DaemonSet that ships container logs to a backend such as Loki or OpenSearch, and have apps log to stdout in JSON.

What should I alert on first?

Crash-looping Pods, unavailable replicas, NotReady nodes, disks above 85%, certificates close to expiry and your app’s error rate and latency. Always add a for duration so short blips do not page anyone.

↑ ↓ navigate Enter open Esc close