Not every incident starts with a blackout. This one started with a "green" deploy in the build and a rollout that never finished.

Summary

The pipeline built the new image, imported it on the nodes and updated the Deployment. The new pods entered CrashLoopBackOff. The old pods continued serving, so the site didn't go down — but the deploy remained incomplete for 40 minutes, until the Jenkins timeout.

Cause: the cluster secret had APP_STORAGE_S3_ACCESSKEY/SECRETKEY = minioadmin. The new image included the guard ProdDefaultCredentialsGuard, which rejects default credentials in the prod profile. The old image didn't have the guard (or had been stripped before the change).

Worse: the new image had already applied the Flyway migration 3.5. Rolling back to the old image started failing in Flyway validation. The path was fix forward.

Timeline

Time	Event
T+0	Pipeline starts; build and image import on the 3 workers
T+40 min	Backend rollout reaches 900 s timeout
T+41 min	New pods in CrashLoopBackOff; old pods 2/2 serving
T+45 min	Logs show INF-SEC-01 (S3 default)
T+50 min	Rollback attempt fails: Flyway 3.5 applied
T+55 min	Dedicated MinIO user created; secret updated
T+60 min	New image promoted; rollout completes
T+65 min	Validation: readiness UP, /api/fornecedores 200, CNPG healthy

Root cause

1. Configuration drift. The cluster secret had MinIO default credentials (minioadmin), while the application already had a guard that rejects them in production.
2. Build does not validate runtime. The build passed; the guard only runs at startup.
3. Rollout without credentials preflight. The pipeline didn't check the secret before swapping the image.
4. Migration applied before the guard. Flyway ran, applied 3.5, and only then the guard killed the process. This made rollback unfeasible.

What worked

• Old pods continued serving during the rollout (RollingUpdate strategy).
• The guard did its job: refused default credential in production.
• The logs pointed to the cause in one line.
• The fix forward was possible without data loss.

What didn't work

• The pipeline didn't have credentials preflight.
• Rollback was attempted before realizing the migration was applied.
• The deploy alert didn't differentiate "build ok" from "rollout ok".
• The default credential was in the cluster for months, without rotation.

Fix

1. Create a dedicated MinIO user (non-default) with read/write policy.
2. Update the portal-secrets secret with the new access/secret.
3. Promote the new image (fix forward).
4. Validate readiness, critical endpoint and database.

The step-by-step is in the runbook docs/RUNBOOK-K8S-DEPLOY-INF-SEC-01.md of the repository.

Lessons

1. Default credential is technical debt with interest. Rotate before the guard forces you.
2. Preflight in the pipeline: fail before the build if the secret has default value.
3. Migration is forward-only. If the new image applied migration, image rollback is not safe.
4. Rollout needs timeout and alert. 40 minutes is too long to discover a CrashLoop.
5. Green build is not green deploy. Validate readiness and a business endpoint, not just /health.
6. Have a fix forward runbook. The way back may not exist.

Checklist for your pipeline

• [ ] The pipeline validates credentials/secrets before swapping the image.
• [ ] The rollout has timeout and alert.
• [ ] Post-deploy validation includes a business endpoint.
• [ ] I know if the image applies migrations.
• [ ] I have fix forward and data rollback plan.
• [ ] Default credentials are rotated.

Next

In chapter 09 we will expose the application with Services, Ingress, DNS and TLS — and understand why the real traffic path matters even in load testing.

Next: Kubernetes Course 09 — Services, Ingress, DNS and TLS (coming soon). Meanwhile, go back to the course index.