Lesson 02 intermediate

Kubernetes Architecture

What every control plane and node component does, what breaks when each one fails, and the full path from kubectl apply to a running Pod.

Every Kubernetes cluster has two halves. The control plane decides what should happen, and the worker nodes make it happen. Knowing which component owns which job is the fastest way to debug a cluster: the symptom usually tells you which piece is unhappy.

The control plane

kube-apiserver

The API server is the front door, and the only door. kubectl, the scheduler, controllers, kubelets and your CI pipeline all talk to it. No component talks to etcd or to another component directly.

Every request goes through the same pipeline: authenticate (who are you?), authorize with RBAC (are you allowed?), admission (should this object be changed or rejected by policy?), validate the object and finally persist it in etcd.

The API server keeps no state of its own, so you can run several copies behind a load balancer. If all copies are down, kubectl stops working and nothing can change, but running Pods keep serving.

etcd

etcd is a distributed key-value store that holds the entire cluster state: every Deployment, Secret, Service and node record. It is the only stateful component of the control plane. Lose etcd without a backup and you lose the cluster definition.

etcd uses the Raft consensus algorithm. Writes need a majority (a quorum) of members, which is why clusters always have an odd number of members:

MembersQuorumFailures tolerated
110
321
431
532

Four members tolerate the same single failure as three, while adding one more machine that can break. etcd is also very sensitive to disk latency: slow disks cause leader elections and an unstable cluster, so give it fast SSDs.

On a self-managed cluster you take snapshots with etcdctl:

$ ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key
Snapshot saved at /backup/etcd-2026-10-30.db

On EKS, GKE and AKS the provider runs and backs up etcd for you.

kube-scheduler

The scheduler watches for Pods that have no node assigned and picks one. It does not start anything: it only writes the chosen node name into the Pod. It works in two steps:

  • Filtering removes nodes that cannot run the Pod: not enough free CPU or memory requests, a taint the Pod does not tolerate, a node selector or affinity that does not match, or a volume that cannot attach there.
  • Scoring ranks the remaining nodes, for example to spread replicas or to balance resource usage.

If the scheduler is down, new Pods stay Pending forever while existing Pods are untouched. When a Pod is Pending, kubectl describe pod shows why:

$ kubectl describe pod api-6f7d9c8b5-x2k4j
...
Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/3 nodes are available: 1 node(s) had untolerated taint
           {node-role.kubernetes.io/control-plane: }, 2 Insufficient memory.

kube-controller-manager

One binary that runs many independent control loops. Each loop compares the desired state with the actual state and acts on the difference:

  • Deployment controller creates and scales ReplicaSets during rollouts.
  • ReplicaSet controller keeps the right number of Pods.
  • Node controller marks silent nodes as not ready and evicts their Pods.
  • Job and CronJob controllers run batch work to completion and on a schedule.
  • Namespace controller cleans up everything inside a namespace that is being deleted.

If the controller manager stops, self-healing stops. Workloads keep running, but a crashed node’s Pods are not replaced and rollouts freeze.

cloud-controller-manager

On a cloud provider this component connects Kubernetes to the cloud API: it creates load balancers for LoadBalancer Services, attaches disks and labels nodes with their region and zone. If it fails, new load balancers stay <pending> and volumes do not attach.

The node components

kubelet

The kubelet is the agent on every node. It watches the API server for Pods assigned to its node, asks the container runtime to start them, mounts their volumes, runs their health probes and reports their status back.

The kubelet is independent: if it loses contact with the API server, it keeps the containers it already has running. If the whole node goes silent, the node becomes NotReady, and after about five minutes its Pods are evicted and recreated elsewhere.

$ kubectl get nodes
NAME       STATUS     ROLES           AGE   VERSION
cp-1       Ready      control-plane   41d   v1.34.1
worker-1   Ready      <none>          41d   v1.34.1
worker-2   NotReady   <none>          41d   v1.34.1
$ kubectl describe node worker-2
...
Conditions:
  Type             Status    Reason
  MemoryPressure   Unknown   NodeStatusUnknown
  DiskPressure     Unknown   NodeStatusUnknown
  Ready            Unknown   NodeStatusUnknown

kube-proxy

kube-proxy makes Services work. It programs iptables or IPVS rules on every node so that traffic sent to a Service’s virtual IP is rewritten to one of the Pod IPs. It is not in the data path itself: the kernel does the forwarding. That is why Service load balancing is per connection and cannot look at HTTP paths.

Container runtime

The runtime pulls images and runs containers. Kubernetes talks to it through the Container Runtime Interface (CRI). containerd and CRI-O are the common choices today.

What happens when you run kubectl apply

This is the most useful picture to keep in your head:

  1. kubectl sends your manifest to the API server.
  2. The API server authenticates, authorizes, runs admission, validates and stores the Deployment in etcd.
  3. The Deployment controller sees the new Deployment and creates a ReplicaSet.
  4. The ReplicaSet controller creates Pod objects with no node assigned.
  5. The scheduler picks a node for each Pod and records it.
  6. The kubelet on that node pulls the image, starts the containers, mounts volumes and runs probes.
  7. The kubelet reports the Pod status back to the API server.
  8. When the Pod is Ready, its IP is added to the Service’s EndpointSlice and kube-proxy updates the routing rules.

Notice that no component calls another one directly. Each one watches the API server and reacts to changes. The system is eventually consistent: every piece converges toward the desired state on its own.

Control plane vs worker node

Control planeWorker node
JobDecideExecute
ComponentsAPI server, etcd, scheduler, controller managerkubelet, kube-proxy, container runtime
Runs your PodsNo, by defaultYes
If it is lost, traffic stopsNoYes, for the Pods on that node
If it is lost, changes stopYesNo
Who manages it on EKS/GKE/AKSThe providerYou

Tip: control plane nodes carry a taint that keeps your workloads off them. Removing it is fine on a one-node lab cluster, but in production your apps could starve the API server of CPU and memory.

Quick reference

TaskCommand
Node statuskubectl get nodes -o wide
Node conditions and capacitykubectl describe node <name>
API server health checkskubectl get --raw='/readyz?verbose'
Control plane Podskubectl get pods -n kube-system
Recent cluster eventskubectl get events -A --sort-by=.lastTimestamp
API server addresskubectl cluster-info
Every resource typekubectl api-resources
Node CPU and memory usagekubectl top nodes

Exercise

Which control plane component is the only one that stores the cluster’s state?

  1. kube-scheduler
  2. kube-apiserver
  3. kube-controller-manager
  4. etcd
Show answer

Answer: D. The API server, scheduler and controllers are stateless and can be restarted freely. Everything they know is read from etcd, which is why etcd is the component you must back up.

Exercise

A worker node loses its network connection for 10 minutes. What does the cluster do with the Pods that ran on it?

  1. Nothing: they are considered healthy until the node returns
  2. It marks the node NotReady and, after about five minutes, evicts the Pods and recreates them on other nodes
  3. It deletes the node object permanently
  4. It restarts the whole control plane
Show answer

Answer: B. The node controller notices the missing heartbeats. Pods managed by controllers are recreated elsewhere; bare Pods are simply lost, another reason to avoid them.

Key takeaways

  • The API server is the only entry point; every component talks to it and nothing else.
  • etcd is the only stateful part: back it up and give it fast disks.
  • Run an odd number of etcd members.
  • The scheduler assigns Pods to nodes, the kubelet runs them. A Pending Pod is a scheduling problem.
  • Controllers are reconciliation loops: without the controller manager there is no self-healing.
  • kube-proxy programs kernel rules and is not in the data path.
  • A control plane outage is a management outage, not a traffic outage.

FAQ

What are the main components of a Kubernetes cluster?

The control plane runs kube-apiserver, etcd, kube-scheduler, kube-controller-manager and, on clouds, cloud-controller-manager. Every worker node runs the kubelet, kube-proxy and a container runtime such as containerd.

What happens if the control plane goes down?

Running Pods keep working because each kubelet keeps its containers alive. You cannot deploy, scale or use kubectl, and failed Pods or nodes are not repaired until the control plane returns.

What is the difference between kubelet and kube-proxy?

The kubelet manages Pods on its node: it starts containers, mounts volumes and runs probes. kube-proxy manages networking: it programs the rules that send Service traffic to Pod IPs.

Does Kubernetes still use Docker?

Not as a runtime. Support for Docker Engine (dockershim) was removed in 1.24, and clusters use containerd or CRI-O. Images built with Docker are standard OCI images and run without changes.

Why is my Pod stuck in Pending?

The scheduler could not find a suitable node. Run kubectl describe pod <name> and read the FailedScheduling event: common reasons are insufficient CPU or memory requests, taints, node selectors that match no node and volumes in another zone.

↑ ↓ navigate Enter open Esc close