We have a high-availability Redis setup with a cluster of 3 nodes managed by Sentinel for automatic master failover, where write operations are directed to the master. Additionally, we have multiple read replicas that do not store data on disk but use swapdb for diskless replication. The goal is for these read replicas to continue serving read requests regardless of what happens to the master, especially during a failover event.

Expected Behavior

When the master is replaced by a new one (via failover), we expected one of two things: - A partial resync, where the replicas continue to serve requests without interruption and only synchronize the missing data. - A full resync, but the replicas should not be blocked from serving read queries.

In either case, replicas should remain available for read operations even during resynchronization, as their primary role is to handle read traffic.

Observed Behavior

During a master failover, instead of a seamless continuation of read operations, the replicas are temporarily blocked from serving read queries while they perform a full resync with the new master.

This blocking behavior contradicts the expectation that replicas should serve reads even during the resynchronization process, particularly with swapdb diskless replication, where the replicas should not be blocked by the master’s status.

This behavior results in complete service unavailability during failover and full resynchronization by replicas

I’ve noticed the following code in the diskless replication handling:

/* Async loading means we continue serving read commands during full resync, and
* "swap" the new db with the old db only when loading is done.
* It is enabled only on SWAPDB diskless replication when master replication ID hasn't changed,
* because in that state the old content of the db represents a different point in time of the same
* data set we're currently receiving from the master. */
if (memcmp(server.replid, server.master_replid, CONFIG_RUN_ID_SIZE) == 0) {
        asyncLoading = 1;
}

https://github.com/redis/redis/blob/unstable/src/replication.c#L2238-L2245

My question is: Why is replid2 not being checked here? Is there a specific reason why replid2 is not considered in this part of the code, given its role in handling more complex replication scenarios?

Looking forward to your insights.

Thank you!

Comment From: ShooterIT

When the master is replaced by a new one (via failover), we expected one of two things:
- A partial resync, where the replicas continue to serve requests without interruption and only synchronize the missing data.

yes, usually, the new master has more up-to-date data, i.e. the offset is bigger during last replication history, so replicas can just sync the diff data.

  • A full resync, but the replicas should not be blocked from serving read queries.

in this scenario, the reason why a full resync happens is the new master has more outdated data, i.e. its offset is smaller than other replicas, or even the new master has bigger offset but the replication backlog is much small not to contain the diff data.

when a replica is promoted to master, its replication id will be changed, so the reply of full resync will have the new replication id instead of old replication id (i.e. the second replication).

if (memcmp(server.replid, server.master_replid, CONFIG_RUN_ID_SIZE) == 0) {
        asyncLoading = 1;
}

for this check, i think it is necessary, it means the replica has the same replication history with master, this scenario happens on full resync because of disconnection rather than failover.

And i think checking replid2 makes no sense here. you should confirm why full resync happened when failover was performed, maybe the new voted master doesn't have biggest offset, or the replication backlog configuration is too small.