Every cluster needs maintenance: update the OS, the kubelet, increase RAM, replace disks. The difference between a smooth window and an incident is the process.

This chapter closes Season 2 with the upgrade we actually did: workers from 8 GiB to 16 GiB.

Course seal: tested on Kubernetes 1.35 and Ubuntu 24.04.

Before touching any node

Pre-flight checklist:

• Free capacity on other nodes to receive pods.
• PDBs for critical workloads.
• Anti-affinity (databases, Elasticsearch, RabbitMQ).
• Quorum (etcd, RabbitMQ, Redis Sentinel).
• Node-local storage (local-path) and PVCs.
• DaemonSets that need to run on the node.
• Recent backup and tested restore.

kubectl get nodes -o wide
kubectl top nodes
kubectl get pdb -A
kubectl get pods -A -o wide | grep <node>

The process, one node at a time

1. Cordon

kubectl cordon k8s-worker-01

No new pods are scheduled there; existing ones continue.

2. Drain

kubectl drain k8s-worker-01 --ignore-daemonsets --delete-emptydir-data

The drain respects PDBs. If it gets stuck, do not use --force without understanding: find out which PDB is blocking.

kubectl get pdb -A
kubectl describe pdb <pdb>

3. Validate rescheduling

kubectl get pods -A -o wide | grep -v Running
kubectl -n <ns> get pods -o wide

Pods with mandatory anti-affinity may stay Pending until the node returns. It's expected — but you need to know which ones.

4. Shut down/update

• Increase RAM on the VM/host.
• Update the OS and kubelet, if applicable.
• Reboot.

5. Bring up and uncordon

kubectl uncordon k8s-worker-01
kubectl get nodes
kubectl get pods -A -o wide | grep k8s-worker-01

6. Validate before the next one

• Nodes Ready.
• Critical workloads healthy.
• Quorum restored.
• PDBs with allowed disruptions.
• Metrics without anomalies.

Order matters

In a cluster with data:

1. Node without database/state first.
2. Replicas before the primary.
3. Primary last (with planned failover).
4. Never two nodes at the same time.

In our case, the worker with the Postgres primary was the last one.

What went wrong in our upgrade

• MinIO all on one node. All 4 pods were on the same worker; draining that node would bring down the object storage. We moved the pods beforehand (cordon + recycling), but found the image was not cached on another node — ErrImagePull.
• MinIO PDB minAvailable: 3. Draining a node with 2 pods violated the PDB. We temporarily adjusted it to 2.
• Postgres primary on the node. The drain required failover; the primary PDB was blocking. We performed a controlled failover beforehand.
• Pods Pending due to anti-affinity. With one node down, Elasticsearch and RabbitMQ could not maintain 3 replicas. Expected; we documented it.
• Aggressive HPA. In another incident, 10 backend pods stressed memory and brought down Postgres. Capacity is not just CPU.

Kubernetes version upgrade

For the kubelet:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt install -y kubeadm=1.35.x-1.1 kubelet=1.35.x-1.1 kubectl=1.35.x-1.1
sudo kubeadm upgrade node
sudo systemctl restart kubelet
kubectl uncordon <node>

Masters first (one at a time), then workers. Never skip minor versions.

Common errors

Symptom	Cause
Drain stuck	PDB with no disruptions
Pods Pending after drain	anti-affinity/capacity
local-path PVC stuck	volume is from old node
DaemonSet missing	taint without toleration
Postgres without quorum	primary/replica on same node
Node does not return Ready	kubelet/CNI/clock
HPA scales too much	memory budget ignored

Checklist

• [ ] Capacity on remaining nodes.
• [ ] PDBs reviewed.
• [ ] Anti-affinity/quorum mapped.
• [ ] Recent backup.
• [ ] Cordon + drain + validation.
• [ ] One node at a time.
• [ ] Uncordon and validation before the next.
• [ ] PDBs restored to original value.

Glossary

• cordon: prevents new pods on the node.
• drain: evicts pods from the node.
• uncordon: frees the node.
• eviction: removal respecting PDB.
• quorum: majority for consensus.
• anti-affinity: distribution rule.

Next

Season 2 concluded. In Season 3, we will package the application with Helm, version it in GitOps, set up CI/CD, harden security, measure capacity, and close with SLO and DR.

Next: Kubernetes Course 21 — Helm: packaging the application