You cannot fix what you cannot see. Kubernetes generates a huge amount of signals; the hard part is choosing which ones matter.

Observability has three pillars:

Pillar	Question	Tools
Metrics	“how much/what?”	Prometheus, Grafana
Logs	“why?”	Loki, Elasticsearch
Traces	“where?”	OpenTelemetry, Jaeger

Course seal: tested on Kubernetes 1.35 with kube-prometheus-stack.

metrics-server: the kubectl top

The metrics-server collects CPU/memory from pods and nodes via the metrics API. It is what powers kubectl top and HPA based on CPU/memory.

kubectl top nodes
kubectl top pods -A --sort-by=memory | head

If it returns error: Metrics API not available, the metrics-server is not installed or has a TLS/RBAC problem.

Important: metrics-server is not long-term storage. It serves recent metrics for the HPA. For history, Prometheus.

Prometheus: the de facto standard

Prometheus does scrape (pulls metrics) over HTTP at regular intervals, stores them in TSDB and allows queries in PromQL.

Common components:

• Prometheus: collects and stores.
• node-exporter: node metrics (CPU, memory, disk, network).
• kube-state-metrics: state of objects (pods, deployments, PDB).
• Alertmanager: groups, silences and routes alerts.
• Grafana: visualization.

Installation via Helm (lab):

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword='troque-isto'

Validate:

kubectl -n monitoring get pods
kubectl -n monitoring get servicemonitors

What to measure first

Don't try to measure everything. Start with what hurts:

Signal	Example metric
Healthy nodes	up{job="node-exporter"}
Ready pods	kube_pod_status_ready
Restarts	kube_pod_container_status_restarts_total
CPU/memory	container_cpu_usage_seconds_total, container_memory_working_set_bytes
Disk	node_filesystem_avail_bytes
API server	apiserver_request_duration_seconds
HTTP errors	http_server_requests_seconds_count{status=~"5.."}

Useful queries:

# nós down
up == 0

# pods que reiniciaram mais de 3 vezes na última hora
increase(kube_pod_container_status_restarts_total[1h]) > 3

# memória dos pods por namespace
sum by (namespace, pod) (container_memory_working_set_bytes{container!=""})

Grafana: dashboards that answer questions

A good dashboard answers:

• Is the cluster healthy?
• What changed in the last 24h?
• Which service is degraded?
• Are we within the SLO?

Avoid dashboards with 200 panels. Start with:

1. Overview: nodes, pods, CPU, memory, disk.
2. Namespace/application: replicas, restarts, latency, errors.
3. SLO: availability and error budget.

Alerts: what to wake someone up

A good alert is actionable. Examples:

- alert: NodeDown
  expr: up{job="node-exporter"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Nó {{ $labels.instance }} down"

- alert: PodCrashLooping
  expr: increase(kube_pod_container_status_restarts_total[10m]) > 3
  for: 5m
  labels:
    severity: warning

Rules:

• Every alert has a runbook.
• for avoids noise.
• Severity defines channel (critical wakes up; warning waits).
• Alertmanager groups and silences during maintenance.

Logs and traces

• Logs: structured (JSON), with traceId/requestId, defined retention. Loki is a lightweight option.
• Traces: OpenTelemetry in the app, collector and backend (Jaeger/Tempo). Without sampling, cost explodes; start with 10%.

Common errors

Symptom	Cause
kubectl top unavailable	metrics-server down/TLS
Target down in Prometheus	wrong ServiceMonitor/port/labels
Empty dashboard	wrong datasource or renamed metric
Prometheus out of disk	retention/TSDB without PVC
Alerts in loop	for too short or rule without hysteresis
Cardinality exploding	high cardinality labels (pod, request id)

Checklist

• [ ] metrics-server working (kubectl top).
• [ ] Prometheus with retention and PVC.
• [ ] node-exporter and kube-state-metrics.
• [ ] Cluster and application dashboards.
• [ ] Alerts with severity and runbook.
• [ ] Structured logs with correlation.
• [ ] Tracing with defined sampling.

Glossary

• scrape: periodic collection of metrics.
• TSDB: time series database.
• PromQL: query language.
• ServiceMonitor: Prometheus Operator CRD.
• SLO: service level objective.
• cardinality: number of label combinations.

Next

Now that you can see the cluster, let's understand operators — what they do, when to use them and why they change the way you operate data.

Next: Kubernetes Course 14 — Operators: what they are and why use them