Queues and cache seem simple until the day the cluster loses quorum, the queue stops accepting messages, or Redis starts evicting keys. This chapter is about operating these two services with real HA.

Course seal: tested on Kubernetes 1.35 with RabbitMQ Cluster Operator and Spotahome Redis Operator.

RabbitMQ: quorum queue

Modern RabbitMQ uses quorum queues (Raft). They replace classic mirrored queues and require a majority to function.

A 3-node cluster tolerates 1 failure. With 2 nodes, it tolerates none.

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: "1"
      memory: 1Gi
  persistence:
    storageClassName: portal-nfs
    storage: 5Gi

The operator creates StatefulSet, Services, secrets, and configuration. Typical services:

• rabbitmq (AMQP 5672, management 15672, metrics 15692).
• rabbitmq-nodes (headless).

Quorum, PDB, and anti-affinity

• 3 replicas on different nodes (anti-affinity).
• PDB minAvailable: 2 to avoid losing quorum during drain.
• Resources defined: RabbitMQ suffers from low memory and full disk.

RabbitMQ has two classic alarms:

• Memory alarm: blocks publishers when it exceeds the limit.
• Disk alarm: blocks publishers when free disk space falls below the limit.

Monitor rabbitmq_queue_messages, rabbitmq_queue_consumers, memory, and disk.

Queues, DLQ, and policies

• Configure a dead-letter exchange for messages that fail.
• Set TTL and max-length to avoid infinite queues.
• Use policies to apply configurations to queues by default.
• Create separate vhosts and users per application.

Credentials: the operator generates a secret (rabbitmq-default-user). Do not use the default user in the application; create a user with minimal permissions.

Redis: cache and state

Redis is used as cache, session, rate limiting, and lock. There are two decisions:

1. Pure cache? Data can be lost; configure maxmemory and maxmemory-policy (e.g., allkeys-lru).
2. State? Needs persistence (AOF/RDB) and HA with sentinel.

The Spotahome Redis Operator creates RedisFailover with replicas and sentinels:

apiVersion: databases.spotahome.com/v1
kind: RedisFailover
metadata:
  name: portal-redis
spec:
  redis:
    replicas: 3
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi
  sentinel:
    replicas: 3
    resources:
      requests:
        cpu: 20m
        memory: 64Mi

Services:

• rfr-portal-redis (Redis master/replica).
• rfs-portal-redis (sentinel).

The application talks to the sentinel, which discovers the master. If the master goes down, the sentinel promotes a replica.

Redis: memory and eviction

Without maxmemory, Redis consumes up to the node's limit and is killed by the OOM killer.

maxmemory 400mb
maxmemory-policy allkeys-lru

For cache, allkeys-lru is reasonable. For queue/lock, eviction can be catastrophic — do not use the same instance for cache and state.

Monitor:

• redis_memory_used_bytes
• redis_evicted_keys_total
• redis_connected_clients
• latency and slow commands

Lab

1. RabbitMQ:

kubectl -n portal get rabbitmqcluster
kubectl -n portal get pods -l app.kubernetes.io/name=rabbitmq
kubectl -n portal get pdb rabbitmq-pdb

Access the management via port-forward and see queues/consumers.

2. Redis:

kubectl -n portal get redisfailover
kubectl -n portal get pods -l app.kubernetes.io/name=portal-redis

3. Knock down one pod of each and observe recovery:

kubectl -n portal delete pod rabbitmq-server-1
kubectl -n portal delete pod rfr-portal-redis-1
kubectl -n portal get pods -w

4. Test drain with PDB: see the drain respect minAvailable.

Common errors

Symptom	Cause
RabbitMQ does not accept publish	memory/disk alarm
Queue unavailable	loss of quorum (2 of 3)
Messages accumulating	stopped or slow consumer
Redis OOMKilled	without maxmemory/limit
Unexpected eviction	allkeys-* policy on critical data
Sentinel cannot find master	network/DNS or sentinel quorum
Drain blocked	PDB without sufficient replicas

Checklist

• [ ] RabbitMQ with 3 replicas and quorum queues.
• [ ] PDB and anti-affinity configured.
• [ ] Memory/disk alarms monitored.
• [ ] DLQ and TTL defined.
• [ ] Redis with maxmemory and appropriate policy.
• [ ] Sentinel with 3 replicas.
• [ ] Credentials per application, no default.
• [ ] Runbook for quorum loss.

Glossary

• quorum queue: RabbitMQ Raft queue.
• DLQ: dead-letter queue.
• sentinel: Redis discovery/failover service.
• eviction: key removal by memory policy.
• vhost: logical isolation in RabbitMQ.
• PDB: disruption budget.

Next

Now let's go to the search engine: Elasticsearch with ECK, shards, cluster health, and snapshots.

Next: Kubernetes Course 18 — Elasticsearch/ECK in practice