This is the chapter you'll reread. Kubernetes doesn't hide the problem: it gives you states, events, and logs. The hard part is knowing what order to look.

Course seal: commands valid for Kubernetes 1.35. Replace names/namespaces with your own.

The method in 6 steps

1. What's happening? kubectl get pods -o wide
2. Why? kubectl describe pod <pod> (look at Events at the end)
3. Did the app complain? kubectl logs <pod> and --previous
4. Is the node healthy? kubectl describe node <node>, kubectl top
5. Is network/config correct? endpoints, DNS, secrets, probes
6. Do I need to get into the pod? kubectl exec or kubectl debug

The state tells the story

Status	First question	Command
Pending	did the scheduler manage to schedule?	describe pod → Events
ContainerCreating	image/volume/CNI?	describe pod
ImagePullBackOff	does the image exist/have access?	Events + imagePullSecrets
CrashLoopBackOff	why does the process die?	logs --previous
OOMKilled	did it run out of memory?	describe → Last State
RunContainerError	entrypoint/permission?	logs, describe
Terminating	finalizer/PDB/node?	describe, get pdb
Completed	Job finished?	normal for Jobs

Pending

The pod was created, but was not scheduled.

kubectl describe pod <pod> | tail -30

Common causes:

• Insufficient resources (Insufficient cpu/memory).
• Impossible nodeSelector/affinity.
• Taint on node without toleration.
• Unbound PVC (pod has unbound immediate PersistentVolumeClaims).
• Namespace quota exceeded.
• Required anti-affinity without available node.

Real case: during maintenance of a worker, Elasticsearch and RabbitMQ pods became Pending because the anti-affinity required one node per replica and there were only two schedulable nodes. It wasn't lack of resources; it was distribution rule.

ContainerCreating and ImagePullBackOff

kubectl describe pod <pod> | grep -A6 Events

• Nonexistent image/wrong tag: registry responds 404.
• Private registry: missing imagePullSecret.
• Rate limit: Docker Hub without authentication.
• Wrong architecture: amd64 image on arm64 node.
• Image only cached on some nodes: if the pod is rescheduled, it fails.

Real case: an image was cached on two nodes and not on the third. When the pod was rescheduled, it went into ErrImagePull. It "worked" until the node changed.

CrashLoopBackOff

The container starts and dies. The current log may be empty; use the previous one:

kubectl logs <pod> --previous
kubectl describe pod <pod> | grep -A5 'Last State'

Causes:

• Missing environment variable/secret.
• Unavailable dependency (database, queue, DNS).
• File/volume permission.
• OOMKilled (memory limit).
• Liveness killing the app before finishing boot.

Real case: the app started, Flyway applied the migration and the credential guard killed the process. The current log showed only the beginning; --previous showed the exception.

OOMKilled

kubectl describe pod <pod> | grep -i oom
kubectl top pod <pod>

Symptoms: Last State: Terminated, Reason: OOMKilled. Increase the limit based on real usage, not guesswork. JVM/Node memory needs margin for off-heap.

Misconfigured probes

• Failing readinessProbe: pod Running but 0/1, out of Service.
• Aggressive livenessProbe: loop restart during boot.
• Missing startupProbe in slow app: liveness kills before it comes up.

kubectl describe pod <pod> | grep -A4 -i probe

Network and DNS

kubectl -n <ns> get endpoints <service>
kubectl -n <ns> run dns --image=busybox:1.36 --rm -it --restart=Never -- nslookup <service>

If DNS fails, check CoreDNS and NetworkPolicy. If the Service has no endpoints, the selector is wrong or the pod is not Ready.

Enter the pod

kubectl exec -it <pod> -- sh
kubectl debug -it <pod> --image=busybox:1.36 --target=<container>

kubectl debug is useful when the image has no shell. For nodes, kubectl debug node/<node> creates a privileged pod — use with care.

Cluster events

kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl -n <ns> get events --sort-by=.lastTimestamp

Events expire (1h by default). If the problem was yesterday, you need logs/monitoring, not get events.

Permission and admission errors

• forbidden: ServiceAccount RBAC.
• admission webhook denied: policy/PSA.
• failed quota: ResourceQuota.
• LimitRange: request/limit out of allowed range.

kubectl auth can-i list pods --as=system:serviceaccount:<ns>:<sa>

Diagnostic checklist

• [ ] get pods -o wide (node, IP, restarts).
• [ ] describe pod with Events.
• [ ] logs and logs --previous.
• [ ] describe node and top.
• [ ] endpoints/DNS.
• [ ] secrets/configmaps/probes.
• [ ] namespace events.
• [ ] RBAC/admission.

Glossary

• Event: ephemeral record of cluster decision.
• BackOff: exponential wait before new attempt.
• OOMKilled: death by memory limit.
• Ephemeral container: diagnostic container attached to the pod.
• Admission webhook: validation/mutation at API entry.

Next

Season 1 completed. In Season 2 we will deep dive into storage, configuration, operators and backup — starting with PV, PVC, NFS and local-path.

Next: Kubernetes Course 11 — Storage: PV, PVC, NFS and local-path