Elasticsearch is not a transactional database. It is a distributed search engine: data is divided into shards, each shard has replicas, and the cluster needs quorum to elect the master.

Confusing it with a “Postgres that searches” is the fastest path to a red cluster.

Course badge: tested on Kubernetes 1.35 with ECK and Elasticsearch 8.x.

Concepts that matter

Concept	What it is
Index	collection of documents
Primary shard	piece of the index where data lives
Replica	copy of the shard for HA and reads
Node	Elasticsearch process
Cluster state	metadata and shard allocation

Health:

• green: all primary and replica shards allocated.
• yellow: primaries ok, replicas not allocated (common in single node).
• red: some primary not allocated — data is unavailable.

ECK: the operator

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: portal-es
spec:
  version: 8.19.0
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
      podTemplate:
        spec:
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchLabels:
                      elasticsearch.k8s.elastic.co/cluster-name: portal-es
                  topologyKey: kubernetes.io/hostname
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 1Gi
                  cpu: 250m
                limits:
                  memory: 1Gi
                  cpu: "1"
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes: ["ReadWriteOnce"]
            resources:
              requests:
                storage: 10Gi

ECK creates StatefulSets, Services, secrets, and configuration. Typical services:

• portal-es-es-http (9200).
• portal-es-es-transport (9300, headless).
• portal-es-es-internal-http.

Heap and memory

Rule of thumb: heap = 50% of container memory, capped at ~31 GB (compressed oops). The rest is filesystem cache.

If the container has 1 GiB, set -Xms512m -Xmx512m. Too large heap in a small container = OOMKilled.

Monitor:

• elasticsearch_jvm_memory_used_bytes
• elasticsearch_cluster_health_status
• elasticsearch_indices_search_query_time_seconds
• elasticsearch_indices_indexing_index_time_seconds

Shards: neither too few, nor too many

• Too small shards: metadata overhead.
• Too large shards: slow recovery.
• Replicas consume space and CPU.

For an index of a few GB, 1–3 primary shards are enough. Don't copy “5 shards, 1 replica” from an old tutorial without thinking.

Snapshots and restore

Snapshot is the Elasticsearch backup. Configure an S3/MinIO repository:

curl -X PUT "localhost:9200/_snapshot/portal-backup" -H 'Content-Type: application/json' -d '{
  "type": "s3",
  "settings": {
    "bucket": "portal-k8s-backup",
    "endpoint": "http://minio-s3:9000",
    "path_style_access": true
  }
}'

Manual snapshot:

curl -X PUT "localhost:9200/_snapshot/portal-backup/snap-2026-10-30?wait_for_completion=true"

Restore to a new index:

curl -X POST "localhost:9200/_snapshot/portal-backup/snap-2026-10-30/_restore" \
  -H 'Content-Type: application/json' -d '{
    "indices": "portal-tickets",
    "rename_pattern": "(.+)",
    "rename_replacement": "restored-$1"
  }'

Snapshots must be scheduled (SLM) and tested.

ILM: lifecycle

Indices grow. Use ILM to:

• hot → warm → cold → delete.
• rollover by size/age.
• force merge and shrink.

Without ILM, the disk fills up and the cluster goes red.

Lab

1. Inspect the cluster:

kubectl -n portal get elasticsearch
kubectl -n portal get pods -l elasticsearch.k8s.elastic.co/cluster-name=portal-es
kubectl -n portal get pvc | grep elasticsearch

2. Health and shards:

kubectl get --raw "/api/v1/namespaces/portal/services/portal-es-es-http:9200/proxy/_cluster/health?pretty"
kubectl get --raw "/api/v1/namespaces/portal/services/portal-es-es-http:9200/proxy/_cat/shards?v"

3. Snapshot and restore to a new index.

4. Simulate the loss of a pod and watch shard reallocation.

Common errors

Symptom	Cause
yellow	replicas not allocated (single node)
red	primary not allocated; data unavailable
disk full	watermark / no ILM
OOMKilled	heap too large in container
split brain	quorum lost (insufficient masters)
snapshot fails	credentials/endpoint/bucket
slow search	large shards, small heap, bad query

Checklist

• [ ] 3 nodes with anti-affinity and PVCs.
• [ ] Heap sized (50% of RAM, 31 GB cap).
• [ ] Health monitored (green/yellow/red).
• [ ] ILM configured.
• [ ] Snapshot repository + SLM.
• [ ] Restore tested to a new index.
• [ ] Disk and JVM alerts.

Glossary

• shard: division of the index.
• replica: copy of the shard.
• watermark: disk limit that blocks writes.
• ILM: index lifecycle management.
• SLM: snapshot lifecycle management.
• split brain: cluster with more than one master.

Next

Now we'll close Season 2 with real backup and restore: Velero, etcd, and the drill that proves backup works.

Next: Kubernetes Course 19 — Backup and restore: Velero, etcd, and drill