Deploy fails less because of Kubernetes and more because of the artifact: image with mutable tag, different build per environment, dependency without trace. This chapter is about making the artifact trustworthy and traceable.

Rule that solves most of it: build once, promote the same digest.

Course seal: tested with GitHub Actions, GHCR and Kubernetes 1.35.

Mutable tag is the problem

portal-backend:latest and even :v5 can point to different images tomorrow. The pod running is not the pod you tested.

Use:

• sha-<commit> as main tag.
• Digest (@sha256:...) in deploy.
• No latest in production.

docker build -t ghcr.io/example/portal-backend:sha-$GIT_SHA .
docker push ghcr.io/example/portal-backend:sha-$GIT_SHA
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/example/portal-backend:sha-$GIT_SHA | cut -d@ -f2)
echo "$DIGEST"

In the manifest:

image: ghcr.io/example/portal-backend@sha256:2f0c...c91

Digest is immutable by construction.

Separate build from deploy

Stage	What runs	Environment
build	test, lint, image, scan	runner
publish	push + SBOM + signature	runner
deploy homolog	updates digest in GitOps	ArgoCD
promote prod	PR with the same digest	ArgoCD

The production pipeline does not compile. It only changes the digest in the GitOps repository. This eliminates "worked in staging" due to build difference.

Minimum pipeline

name: build
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:sha-${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Build cache reduces minutes and, more importantly, makes the build reproducible.

Vulnerability scan

trivy image --severity HIGH,CRITICAL --exit-code 1 \
  ghcr.io/example/portal-backend:sha-$GIT_SHA

Structure the gate:

• CRITICAL with fix available: blocks.
• CRITICAL without fix: logs and tracks.
• HIGH: alerts, does not block initially.

Blocking everything on day one paralyzes the team. The goal is to reduce, not to halt.

SBOM

SBOM is the list of components in the image. Without it, you cannot answer "are we affected by Log4Shell?" in minutes.

syft ghcr.io/example/portal-backend:sha-$GIT_SHA -o spdx-json > sbom.spdx.json

With Trivy:

trivy image --format spdx-json -o sbom.spdx.json ghcr.io/example/portal-backend:sha-$GIT_SHA

Publish the SBOM as a pipeline artifact and reference it in the release. Common formats: SPDX and CycloneDX.

Signature and provenance

• cosign signs the image.
• SLSA provenance records how it was built.
• Admission controller (Kyverno/cosign policy) rejects unsigned images.

cosign sign --yes ghcr.io/example/portal-backend@sha256:2f0c...c91
cosign verify ghcr.io/example/portal-backend@sha256:2f0c...c91 \
  --certificate-identity-regexp '.*' --certificate-oidc-issuer https://token.actions.githubusercontent.com

This closes the chain: commit → build → signature → deploy.

The security error we have seen

A build passed because MinIO used minioadmin as default credential. The pipeline did not validate configuration — it only compiled. Result: broken deploy in the cluster with INF-SEC-01.

Lesson: the pipeline needs a configuration gate, not just unit testing:

grep -rE 'minioadmin|changeme|password.*=.*(admin|123)' deploy/ && exit 1

Simple, and it would have avoided the incident.

Traceability: from pod to commit

Standardize labels:

metadata:
  labels:
    app.kubernetes.io/name: portal-backend
    app.kubernetes.io/version: sha-3f9c1a
    org.opencontainers.image.revision: 3f9c1a...

With this:

kubectl -n portal get pod -l app.kubernetes.io/version=sha-3f9c1a

You answer "which pod runs commit X" in seconds — and the reverse as well.

Lab

1. Configure a pipeline that generates sha-<commit> and pushes.
2. Run Trivy and see the report.
3. Generate the SBOM in SPDX and inspect dependencies.
4. Change the deploy to use digest and confirm imageID in the pod:

kubectl -n portal get pod -o jsonpath='{.items[*].status.containerStatuses[*].imageID}'

5. Make a commit that introduces a fake credential and see the gate fail.
6. Promote the same digest from staging to production without rebuild.

Common errors

Symptom	Cause
Prod different from staging	build separated by environment
ImagePullBackOff	registry credential or nonexistent tag
Deploy "worked" and broke	reused mutable tag
SBOM not generated	step failed silently
Scan blocks everything	severity without exception policy
Image without signature accepted	missing admission policy
Traceability lost	image without revision label

Checklist

• [ ] Single build per commit.
• [ ] Tag sha-<commit> and deploy by digest.
• [ ] No latest in production.
• [ ] Scan with defined severity and policy.
• [ ] SBOM generated and archived.
• [ ] Signature and verification (cosign/Kyverno).
• [ ] Configuration gate (default credentials).
• [ ] Version/revision labels on pods.
• [ ] Promotion via PR in GitOps repo.

Glossary

• digest: immutable hash of the image.
• SBOM: component inventory.
• SLSA: build integrity framework.
• provenance: record of how the artifact was built.
• cosign: signing tool.
• gate: step that blocks the pipeline.

Next

Trustworthy and traceable artifact. Now harden the cluster: RBAC, Pod Security Admission, NetworkPolicy and secrets management.

Next: Kubernetes Course 25 — Security: RBAC, PSA, NetworkPolicy and secrets