Capacity is not "how many users fit." It's the point at which the system stops meeting the SLO — and which component breaks first.

In most clusters I've seen, it's not the CPU. It's the database, the connection, the disk, or NAT.

Course seal: tested on Kubernetes 1.35 with k6 and HPA in staging.

What to measure

Metric	Why
p50/p95/p99	average hides the tail
error rate	200 with business error doesn't count
throughput	sustained req/s
saturation	CPU, memory, connections, disk
queue time	where the request waits

Average latency is the most useless capacity metric. A p95 of 83 ms with 0% error is a result; an average of 40 ms with p99 of 4 s is a problem.

Baseline before the test

kubectl top nodes
kubectl -n portal top pods
kubectl -n portal get hpa
kubectl -n portal get pdb
kubectl -n portal get pods -o wide

Record: replicas, requests/limits, Postgres connections, node memory usage. Without a baseline, the test becomes anecdote.

k6: ramp, not a single shot

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 10 },
    { duration: '5m', target: 50 },
    { duration: '3m', target: 100 },
    { duration: '2m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<500'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/fornecedores`, {
    headers: { Authorization: `Bearer ${__ENV.TOKEN}` },
  });
  check(res, { 'status 200': (r) => r.status === 200 });
  sleep(1);
}

k6 run -e BASE_URL=https://homolog.example.com -e TOKEN=... k6.js

Ramp finds the knee of the curve: the point where latency rises and throughput stops growing.

What happened in our test

With 50 VUs against the ingress:

Metric	Result
Errors	0%
p95	83,9 ms
p99	within acceptable range

The next, more aggressive test took down Postgres: the primary restarted and failover moved the database to another instance. The app was healthy; the database was not.

Practical conclusion: scaling the app without sizing the database transfers the problem. 10 backend pods with a pool of 10 connections = 100 connections, and Postgres has a limit.

HPA: what it sees and what it ignores

metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

• HPA uses requests as the denominator. Without requests, it doesn't scale.
• CPU doesn't see database queue, latency, or connections.
• For queues, use custom metrics (KEDA) — RabbitMQ queue size, for example.
• maxReplicas must fit in the cluster. HPA requesting 8 pods with 4 full nodes = Pending.

kubectl -n portal describe hpa portal-backend

The output shows unable to fetch metrics or missing request for cpu when requests are missing.

Node headroom

Real capacity = sum of allocatable minus what is already reserved, with margin for:

• one lost node (N+1);
• rollout surge (maxSurge);
• DaemonSets.

kubectl describe node <node> | sed -n '/Allocated resources/,/Events/p'

If each node is at 85%, you don't have HA: you have a cluster waiting for the next drain.

Rate limit and protection

Load testing also reveals lack of protection:

• Rate limit at the ingress (nginx limit_req) or in the app.
• Connection and read timeout.
• Circuit breaker for slow dependencies.
• Connection pool with limit and short timeout.

Without this, a spike becomes an infinite queue, and the queue becomes a cascading timeout.

Safe testing

Before running load:

1. Isolated environment (staging), never prod without a window.
2. Recent database backup.
3. On-call alert notified.
4. Defined stop criterion (e.g., error > 5% for 1 min).
5. Observability enabled: Prometheus, Grafana, logs.
6. Rollback plan.

And after:

kubectl -n portal get pods
kubectl -n portal logs deploy/portal-backend --tail=100
kubectl -n portal get events --sort-by=.lastTimestamp | tail -20

Lab

1. Run a short test (10 VUs, 2 min) and validate the environment.
2. Increase the ramp and monitor kubectl get hpa -w and Grafana.
3. Identify the first saturated component.
4. Reduce maxReplicas and repeat: see the error appear.
5. Adjust the app's connection pool and repeat.
6. Document: maximum throughput, p95 at the limit, and bottleneck.

Common errors

Symptom	Cause
p95 explodes under low load	small connection pool
HPA doesn't scale	missing requests or metrics-server down
Pods Pending at peak	maxReplicas above capacity
Database restarts under load	Postgres connections/memory
Unexpected 429	ingress rate limit
Test "passed" and prod went down	test environment with more resources
Unreproducible numbers	no baseline or versioned script

Checklist

• [ ] Baseline recorded before the test.
• [ ] k6 script versioned in the repository.
• [ ] Error and p95 thresholds defined.
• [ ] HPA with requests and maxReplicas that fits in the cluster.
• [ ] Connection pool sized with the database.
• [ ] N+1 headroom validated.
• [ ] Rate limit and timeouts configured.
• [ ] Stop criterion and rollback agreed upon.
• [ ] Result documented with date and commit.

Glossary

• VU: virtual user in k6.
• ramp: gradual load increase.
• knee: point where latency grows without throughput gain.
• headroom: capacity slack.
• pool: set of reusable connections.
• KEDA: event/queue autoscaling.

Next

You've measured capacity and know where it breaks. What's left is to combine that with what the business expects: SLO, SLI, on-call, and blameless postmortem.

Next: Kubernetes Course 27 — SLO/SLI, on-call, and blameless postmortem