A DR plan is not a document. It is a procedure that someone has already executed — with measured time.

Last chapter of the course: disaster scenarios, the drill that proves the restore, and the capstone to wrap everything up.

Course badge: validated with a restore drill in 85 seconds and real Postgres failover.

What is a disaster

Not every incident is a disaster. The distinction matters because the response changes:

Scenario	Impact	Response
Pod restarts	none (HA)	automatic
Node goes down	load redistributes	automatic
Primary database fails	failover	automatic + validation
Namespace deleted	service down	restore objects
Cluster lost	everything down	rebuild + restore
Region/zone lost	everything down	DR site
Corrupted data	partial loss	logical restore with PITR

The last three require a written procedure. The first three, automation.

RPO and RTO by layer

Layer	RPO	RTO	Mechanism
etcd	24h	1–2h	scheduled snapshot
Cluster objects	24h	1–4h	Velero
Postgres	minutes	30–60 min	WAL + base backup
Logical dump	24h	1h	pg_dump
Object storage	24h	hours	replication/backup
Configuration	0	minutes	Git

If the business requires a 5-minute RPO and backup is daily, the problem is not Kubernetes. It's the design.

Four drills worth doing

1. Namespace restore (Velero)

velero backup create drill-ns --include-namespaces portal --wait
velero restore create --from-backup drill-ns --namespace-mappings portal:drill
kubectl -n drill get all

Measure: restore time, pods ready, data accessible.

2. Logical database restore

pg_dump -Fc -h portal-postgres-rw -U portal -d portal -f /tmp/portal.dump
createdb -h portal-postgres-rw -U portal portal_drill
pg_restore -h portal-postgres-rw -U portal -d portal_drill /tmp/portal.dump
psql -h portal-postgres-rw -U portal -d portal_drill -c "select count(*) from tickets;"

Measure the time and compare counts with production.

3. etcd snapshot and restore

In a disposable environment, restore the snapshot and validate that the API responds with the expected objects. In HA, the procedure is surgical — have a runbook and practice it.

4. Rebuilding a service from scratch

Delete the Deployment and Service, and rebuild only with Git + Helm/GitOps:

kubectl -n drill delete deploy portal-backend
argocd app sync portal-drill
kubectl -n drill rollout status deploy/portal-backend

If this doesn't work, your configuration "backup" is an illusion.

DR runbook: what needs to be written down

1. Who decides to declare disaster and who executes.
2. How to access the cluster/console with credentials not stored in the cluster.
3. Recovery order: network → storage → database → apps → ingress.
4. Exact commands, not "restore the backup".
5. Validation criteria for each step.
6. Communication for users and stakeholders.
7. How to record the incident and the time of each step.

Rule: the runbook must be executable by someone tired at 3 AM, without the original author available.

What kills DR in practice

• Backup on the same storage as the cluster.
• Restore credential only in one person's head.
• Snapshot that was never restored.
• Short-lived DNS/TLS not renewed in the new environment.
• Image outside the registry (local build).
• Secret not included in the backup.
• Outdated runbook after architecture change.

Capstone

Objective: prove that you can lose and recover the environment.

Deliverables:

1. Inventory: applications, dependencies, data, and owners.
2. RPO/RTO matrix per service, agreed with the business.
3. Automated backups: etcd, Velero, database, object storage.
4. Executed drill with measured times and evidence (logs, screenshots, counts).
5. DR runbook reviewed by another person.
6. SLO dashboard + backup panel.
7. Drill postmortem: what failed during the rehearsal.

Acceptance criteria: restore the environment in a clean environment, within the declared RTO, with verified data.

Complete drill script

# 1. Estado inicial
kubectl -n portal get pods,pvc
psql -h portal-postgres-rw -U portal -d portal -c "select count(*) from tickets;"

# 2. Backup
velero backup create capstone --include-namespaces portal --wait
pg_dump -Fc -h portal-postgres-rw -U portal -d portal -f /tmp/capstone.dump

# 3. Destruir (ambiente de teste)
kubectl delete namespace portal-drill --ignore-not-found
kubectl create namespace portal-drill

# 4. Restaurar
velero restore create --from-backup capstone --namespace-mappings portal:portal-drill
pg_restore -h portal-postgres-rw -U portal -d portal_drill /tmp/capstone.dump

# 5. Validar
kubectl -n portal-drill get pods
psql -h portal-postgres-rw -U portal -d portal_drill -c "select count(*) from tickets;"

Time each step. The number that goes into the report is the real time, not the estimate.

Common mistakes

Symptom	Cause
Restore "ok" without data	partial backup/exclusions
PVC not recreated	storageClass missing in destination
App starts but doesn't connect	secret/DNS not restored
RTO exceeded	restore never measured
Corrupted backup	storage without verification
Useless runbook	written by someone who never executed it
Drill forgotten	no calendar and owner

Final course checklist

• [ ] HA cluster with etcd, VIP, and PDBs.
• [ ] Workloads with probes, resources, and HPA.
• [ ] Storage, database, queues, cache, and search operated by an operator.
• [ ] Layered backup and tested restore.
• [ ] GitOps as source of truth.
• [ ] CI/CD with immutable image, SBOM, and signing.
• [ ] RBAC, PSA, and NetworkPolicy applied.
• [ ] Capacity measured with load testing.
• [ ] SLO, burn rate alerts, and on-call.
• [ ] DR with RPO/RTO and executed drill.

Glossary

• RPO: maximum acceptable data loss.
• RTO: maximum recovery time.
• drill: recovery rehearsal.
• PITR: point-in-time recovery.
• runbook: operational procedure.
• capstone: final integrative project.

Closing

You started with an empty cluster and arrive here with: HA, data, observability, GitOps, security, measured capacity, SLO, and a tested recovery plan.

What separates a working cluster from a reliable cluster is not the tool. It's having procedures, measuring, and rehearsing before you need it.

Return to the course index to review any chapter — and keep drills on the calendar. Reliability is practice, not a certificate.

Next: revisit Chapter 19 — Backup and restore and run the drill again. Now with a stopwatch.