We are running a redis cluster (v6) with 7 shards. Each shard consist of 1 master and 1 replica. So 14 nodes in total. The cluster is a memory only cluster for storing cache objects.

When I'm updating one of the nodes (system/kernel updates), I need to reboot that node. This is what I do (for the first shard):

Say my master node is node A and the replica node is node B

  1. lookup the replica (= node B) and run system updates
  2. reboot the replica node in case of kernel updates
  3. the replication will come back up and redis shows: Clear FAIL state for node ####: replica is reachable again
  4. The master (= node A) creates a dump Starting BGSAVE for SYNC with target: disk
  5. The replica loads that dump file and states MASTER <-> REPLICA sync: Finished with success
  6. I promote the replica to master and I do everything from step 1 to 5.

What I see is when the original master node (= node A) comes back online, it is a master node(while it should be a replica node). It also loads data into memory (I think that dump.rdb file when it was a master).

After it has loaded the data into memory, it asks the current master (= node B this time) for a partial resync, which it can not because the backlog is gone (so far so good). The master (= node B) then starts a dump, syncs it to node A (which has now switched to replica). It flushes the old data and loads the data into memory.

So eventually the data and master/replica status is correct. But it loads some data after the reboot which in my opinion it shouldn't have to do.

Somehow, loading the dump.rdb file when it was a master is a step it actually shouldn't need to do. I know if the cluster was persistent, it would have had a backlog and the dump.rdb file might not be loaded.

I hope you can follow these steps. I'm also not sure if I'm doing something wrong here, since I couldn't find any documentation on what to do when you need to reboot nodes in a cluster.

[UPDATE - after some tests] When I remove the dump.rdb file before rebooting node A (now replica but originally master), then it starts directly by doing a full resync from the master.

Comment From: madolson

[UPDATE - after some tests] When I remove the dump.rdb file before rebooting node A (now replica but originally master), then it starts directly by doing a full resync from the master.

@leroy0211 Sorry for this issue getting ignored. This was basically going to be my recommendation to you. If you have saving configured, then on reboot it will attempt to save a snapshot and it will try to reload it if it's available in the directory. In your case the snapshot failed, but ideally if the partial sync succeeded it would have been the best case scenario since it would limit the load on the master.

Marking this as to-be-closed unless you have some more questions.

Comment From: mawlstace

had the same issue deleting dump.rdb solved the problem !

Comment From: leroy0211

Today I had the same issue again. And I remembered I asked it before.

I still wonder why redis doesn't remove the dump.rdb itself after it has completely imported the data from a full sync with its master. Lately I've been doing maintenance on these nodes again, but the dump.rdb file just stays there forever.

In our case we only keep the data in memory using 7 shards with one replica for each shard (14 instances in total). In case of a disaster of the cluster (or even a shard + its replica), I think redis would just load the dump.rdb file and continues. Even if that dump.rdb file is over a year old. I havent put this to the test, but I guess if both instances (master and slave) restart, and load the dump.rdb file, they share the same offset. And continue their work, on a very old dump. And serve way too old data.

Comment From: madolson

Yeah, it's just sort of how Redis works. Maybe we could add like a max stale argument, but I don't have much else to suggest.