This is the first postmortem of the course. It is not a hero story: it is the record of a load test that got out of control and what I learned while fixing it.

Summary

During a capacity rehearsal on the staging cluster, I increased the backend HPA to 10 replicas and triggered a ramp of 50 authenticated users. The test did not return HTTP errors, but p95 latency went to 8.4 seconds. Minutes later, the PostgreSQL primary lost liveness probes and CloudNativePG initiated a failover. The old primary got stuck in shutdown and I had to remove it for the cluster to promote the replica.

No data was lost. The service continued responding via old pods during most of the incident. But the scare was big — and the lessons, valuable.

Timeline (summarized)

Time	Event
T+0	HPA adjusted to maxReplicas: 10; test with 50 authenticated VUs starts
T+1 min	Workers reach 92% and 82% memory usage
T+2 min	Backend scales to 10 pods; p95 rises to 8.4 s
T+3 min	Postgres probes start failing due to timeout
T+5 min	CNPG initiates failover; old primary enters shutdown
T+8 min	Old primary does not finish shutdown; cluster becomes Failing over
T+10 min	I remove the old primary pod; CNPG promotes the replica
T+12 min	Cluster healthy; HPA reduced to 6; latency normalizes

Root cause

It was not “Kubernetes”. It was a combination of decisions:

1. HPA without memory budget. The limit of 10 pods was defined thinking about CPU, but each backend pod used ~1.2–1.3 GiB. On 8 GiB workers, 10 pods competed with database, Elasticsearch, MinIO, and RabbitMQ.
2. Memory pressure on nodes. With 92% usage, kubelet and probes start to suffer. Postgres was not killed by OOM, but liveness/readiness probes began to time out.
3. Aggressive probe on a database under load. CNPG's liveness uses the instance manager; under CPU/network contention, the 25 s timeout was enough to mark the primary as unhealthy.
4. Failover without fencing plan. The old primary did not finish shutdown; the cluster remained in Failing over until manual removal of the pod.

What worked

• The CNPG promoted the replica correctly as soon as the old primary went away.
• The backend continued serving with the old pods; there was no total outage.
• The Postgres PDB prevented accidental evictions ahead of time (and later showed why it exists).
• The failover runbook was written from the incident and placed in the repository.

What didn't work

• Adjusting HPA only by CPU, ignoring per-pod memory.
• Running authenticated load without checking kubectl top nodes every minute.
• Testing via direct NodePort: rate limit saw the NAT IP (10.244.1.0) and generated 429, masking the real result. Through ingress, the same test had 0% error.
• Leaving HPA at maximum during failover, further pressuring the nodes.

Actions taken

Action	Result
HPA reduced to maxReplicas: 6	Immediate stability
Workers expanded to 16 GiB	Margin for 8–10 pods with data
Test redone through ingress	0% error, p95 83.9 ms with 50 VUs
CNPG failover runbook	Documented procedure
Capacity plan	Maximum HPA by memory budget, not just CPU

Lessons

1. HPA is a capacity decision, not a magic number. Calculate memory per pod × replicas × nodes.
2. PDB protects, but does not save from resource shortage. It prevents evictions; it does not prevent the node from running out of memory.
3. Database probes are sensitive to contention. Adjust timeout and failureThreshold with real data.
4. Failover needs fencing. If the old primary does not die, the cluster does not promote the replica on its own.
5. Test via the real path. Direct NodePort does not represent ingress traffic.
6. Blameless postmortem. The goal is to fix the system, not the person who pressed the button.

Checklist for your cluster

• [ ] I know how much memory each backend replica uses.
• [ ] HPA has maxReplicas compatible with worker memory.
• [ ] Postgres has PDB and anti-affinity.
• [ ] I know what to do if the old primary does not finish shutdown.
• [ ] I test load via ingress, not via NodePort.
• [ ] I have failover and restore runbook.

Next

In chapter 05 we will set up the control plane in HA — and understand why etcd with quorum is the difference between “high availability” and “three machines that fall together”.

Next: Kubernetes Course 05 — HA: 3 masters, VIP and etcd