Alert rule: RedisMasterMissing
Overview
This Alert will be thrown if a Redis cluster has no master configured (no Redis Pod has role=master) for more than 5 minutes.
It will inform which Pod is affected, for example instance="11.32.186.10:9121".
Steps for Debugging
Check all the Redis Pods in the namespace of the affected Pod and see why no Pod is elected as master.
Get logs of Redis Sentinel
Display the logs of the Sentinel Pods. This should state which of the Pods is the master and which Pods are slaves.
user@computer:~$ kubectl logs redis-node-0 sentinel
1:X 13 Nov 2020 09:04:04.587 * Running mode=sentinel, port=26379.
1:X 13 Nov 2020 09:04:04.588 # Sentinel ID is ANID
1:X 13 Nov 2020 09:04:04.588 # +monitor master mymaster 10.42.0.117 6379 quorum 2
1:X 13 Nov 2020 09:04:04.588 * +slave slave 10.42.0.116:6379 10.42.0.116 6379 @ mymaster 10.42.0.117 6379
1:X 13 Nov 2020 09:04:04.592 * +slave slave 10.42.0.118:6379 10.42.0.118 6379 @ mymaster 10.42.0.117 6379
1:X 13 Nov 2020 09:04:04.750 * +sentinel sentinel ID1 10.42.0.117 26379 @ mymaster 10.42.0.117 6379
1:X 13 Nov 2020 09:04:04.760 # +new-epoch 1
1:X 13 Nov 2020 09:04:05.809 * +sentinel sentinel ID2 10.42.0.116 26379 @ mymaster 10.42.0.117 6379
1:X 13 Nov 2020 09:18:15.444 * +sentinel sentinel ID3 10.42.0.119 26379 @ mymaster 10.42.0.117 6379
1:X 13 Nov 2020 09:18:17.705 * +slave slave 10.42.0.119:6379 10.42.0.119 6379 @ mymaster 10.42.0.117 6379Query Redis Sentinel for Masters
Connect to the Sentinel container in one of the Redis node Pods, start the redis-cli providing the password used for Sentinel and query the Sentinel for the master.
user@computer:~$ kubectl exec redis-node-0 sentinel -it -- sh
$ redis-cli -p 26379 -a foobarbaz12
127.0.0.1:26379> SENTINEL get-master-addr-by-name mymaster
1) "10.42.0.171"
2) "6379"
127.0.0.1:26379> sentinel masters
1) 1) "name"
2) "mymaster"
3) "ip"
4) "10.42.0.171"
5) "port"
6) "6379"
[...]