Backup is easy. Restore is what matters. And restore is only real when you rehearse it.

This chapter closes Season 2 with a layered strategy — and a drill that actually measures RPO/RTO.

Course seal: tested on Kubernetes 1.35 with Velero, CloudNativePG, and etcd 3.5.

Layered Backup

Layer	What it saves	Tool
Cluster state	API objects	etcd snapshot / Velero
Volumes	PVs	Velero (Restic/Kopia) / CSI snapshots
Database data	logical	pg_dump / CNPG backup
Configuration	manifests	Git
Secrets	secrets	Vault / secret manager

Doing only one layer is an illusion of security. I've seen clusters "with backup" that didn't have the database dump — and vice versa.

etcd: The Cluster State

etcd stores all objects. Snapshot:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%F).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Schedule via CronJob on the master (as in our cluster) and test the restore in a disposable environment.

etcd restore is surgery: stop the apiserver, restore the snapshot to a new directory, adjust --data-dir, and bring the cluster back. In HA, the procedure is more delicate. Have a runbook.

Velero: Objects and Volumes

Velero backs up resources and, with Restic/Kopia, volumes.

Installation (lab):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.10.0 \
  --bucket portal-k8s-backup \
  --secret-file ./credentials-velero \
  --backup-location-config region=us-east-1,s3ForcePathStyle=true,s3Url=http://minio-s3:9000

Schedule:

velero schedule create portal-daily --schedule="0 3 * * *" --include-namespaces portal

Manual backup:

velero backup create portal-manual --include-namespaces portal --wait
velero backup describe portal-manual --details

Restore to a new namespace (avoids collision):

velero restore create --from-backup portal-manual \
  --namespace-mappings portal:portal-restore

Database Data

Velero does not replace logical dump. For PostgreSQL:

• CNPG ScheduledBackup + WAL archiving.
• pg_dump/pg_dumpall for logical copy.
• Test restore in a disposable database.

The drill we did: 2.1 GB dump, restore in a disposable database, RTO of 85 seconds, and identical counts. That number is what matters in an emergency.

RPO and RTO

• RPO: how much data you accept losing.
• RTO: how long you accept being down.

Example:

Layer	RPO	RTO
Daily etcd snapshot	24h	1–2h
Daily Velero	24h	1–4h
Postgres WAL	minutes	30–60 min
Daily logical dump	24h	1h

If the business doesn't accept 24h of RPO, daily backup is not enough. The conversation is about product, not tooling.

Lab

1. Snapshot etcd and validate the file.
2. Velero backup of a namespace and restore to a new namespace.
3. Postgres dump/restore to a disposable database; measure the time.
4. Verify restored data (counts, a known record).
5. Document measured RPO/RTO.

Common Errors

Symptom	Cause
Velero backup PartiallyFailed	volume/plugin/credential
Restore doesn't create PVC	storageClass missing on destination
etcd restore doesn't come up	wrong data-dir/endpoint
Backup "ok" but restore is empty	namespace/resource exclude
RPO larger than expected	schedule/WAL stopped
Secret not restored	Velero without RBAC or excluded secret

Checklist

• [ ] etcd snapshot scheduled and tested.
• [ ] Velero schedule + healthy BSL.
• [ ] Database logical dump tested.
• [ ] Restore to disposable namespace/database.
• [ ] RPO/RTO measured and documented.
• [ ] Secrets included in the strategy.
• [ ] Layer-specific restore runbook.

Glossary

• RPO: acceptable maximum data loss.
• RTO: acceptable maximum recovery time.
• BSL: Velero BackupStorageLocation.
• Restic/Kopia: Velero volume backup.
• WAL: Postgres transaction log.
• CSI snapshot: volume snapshot by the driver.

Next

Now the last chapter of Season 2: safe node upgrades — drain, PDB, anti-affinity, and the lessons from our RAM upgrade.

Next: Kubernetes Course 20 — Safe Node Upgrades