How to Manually Perform a Redis Failover

In certain situations two of the three Redis nodes end up in CrashLoopBackOff because they receive a wrong (old) leader IP address from the running Sentinel.

In the logs this might look like:

sentinel Could not connect to Redis at 10.0.0.20:26379: Connection refused

To resolve this situation, get the IP address of the running leader, connect to a Redis Sentinel Pod, start a redis-cli and initiate the failover.

kubectl get pod -o wide

kubectl exec redis-node-0 -c sentinel -it -- bash
redis-cli -a $(< $REDIS_PASSWORD_FILE) -p 26379

SENTINEL REMOVE mymaster
SENTINEL MONITOR mymaster <NEW_IP_ADDRESS> 6379 1