Intro Hi there. I was unsure where to post this as it is somewhat a discussion / feature request.

It seems redis does not have a way to determine using the redis-cli if the current replica: - Is in sync with its master - Is syncing with its master

This means readiness/liveness checks implemented in helm charts often rely on the instance replying to PING, such as done here

This way of performing health checks is insufficient as we cannot tell if the instance is considered as a "good write replica" (min-replicas-to-write)

Causing the readiness check to pass when the instance isn't actually ready yet

Problem

We are running Redis-ha in Kubernetes. The replicas hold approx 80Gi of data (we have larger, but the issue happens at this size too)

We have min-replicas-to-write set to 1 (I can dump all our config here if it helps)

When we rollout any configuration change to these database, we get downtime (10+ minutes, scaling with memory).

Assuming we have a 3 pod setup (1 master, 2 replicas) Pod 0 - Replica Pod 1 - Replica Pod 2 - Master

Sequence of events


Config changes rolls out

Pod 0 restarts to pick up the new change

Pod 0 loads the current state from its RDB file into memory (this can take many minutes)
 - During this time, the readiness check fails as it doesn't respond to PING

 - Readiness probe failed: LOADING Redis is loading the dataset in memory

Pod 0 finishes loading state into memory

 - As this point readiness checks pass

Pod 1 restarts to pick up the new change

**At this point, two pods are down. Writes to the redis now fail as there isn't at least one good write replica**

Pod 0 realises it is out of sync and beings a full resync with the master (this can take many minutes)

Pod 1 starts to load its current state from its RDB file into memory (this can take many minutes)

Pod 1 finishes loading state into memory

 - As this point readiness checks pass

Pod 2 restarts to pick up the new change

Pod 0 replication with master fails as it just restarted

Pod 2 starts to load its current state from its RDB file into memory (this can take many minutes)

 **At this point, all three pods are trying to load state/establish a master. All 3 pods are not writable**

.
.
.

Now all 3 pods are on the new config. It just takes time for them to load in their state and reestablish a master, resync

 - This can cause many minutes of outage

However finally they are all back and healthy again

Log of Pod 0 above, until Pod 2 restarts and breaks the resync:

1:C 02 Nov 2022 11:01:06.216 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 02 Nov 2022 11:01:06.216 # Redis version=7.0.4, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 02 Nov 2022 11:01:06.216 # Configuration loaded
1:S 02 Nov 2022 11:01:06.216 * monotonic clock: POSIX clock_gettime
1:S 02 Nov 2022 11:01:06.218 * Running mode=standalone, port=6379.
1:S 02 Nov 2022 11:01:06.218 # Server initialized
1:S 02 Nov 2022 11:01:06.226 * Loading RDB produced by version 7.0.4
1:S 02 Nov 2022 11:01:06.226 * RDB age 662 seconds
1:S 02 Nov 2022 11:01:06.226 * RDB memory usage when created 67389.77 Mb
1:S 02 Nov 2022 11:07:00.253 * Done loading RDB, keys loaded: 19467, keys expired: 0.
1:S 02 Nov 2022 11:07:00.253 * DB loaded from disk: 354.034 seconds
1:S 02 Nov 2022 11:07:00.253 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 02 Nov 2022 11:07:00.253 * Ready to accept connections
**At this point the readiness check passes and Pod 1 restarts. However it isn't a good write replica yet and the outage begins
1:S 02 Nov 2022 11:07:01.159 * Connecting to MASTER 10.2.17.126:6379
1:S 02 Nov 2022 11:07:01.159 * MASTER <-> REPLICA sync started
1:S 02 Nov 2022 11:07:01.160 * Non blocking connect for SYNC fired the event.
1:S 02 Nov 2022 11:07:01.160 * Master replied to PING, replication can continue...
1:S 02 Nov 2022 11:07:01.161 * Trying a partial resynchronization (request b08f1602d502d3e53960d8a7d4953bf5013f4fec:63779542260).
1:S 02 Nov 2022 11:07:06.123 * Full resync from master: b08f1602d502d3e53960d8a7d4953bf5013f4fec:63818992099
1:S 02 Nov 2022 11:07:08.812 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk

**At this point Pod 2 (master) shutdown breaking replication**

Questions 1. Is there a way via redis-cli to determine if the current instance is a "good write replica" - i.e in sync with the master 2. If the answer to 1 is No, is there a suggestion of config change/ health check change that can be used to prevent down time on redis-ha config change rollouts?

Comment From: JamesMurkin

This can approximately be achieved using INFO REPLICATION to determine the health of the replication and using this in the health check. - Check connection to master is up - Check syncing isn't in progress