I have a redis cluster of 3 master and each master have 2 slaves connected running in statefulset of kubernetes.
Problem : The problem is when i manually delete one of the master pod failover takes place and one of the slave of that master becomes the new master and when that deleted master pod resurrected it did not become the slave of this master it joins the other master in the cluster and that master now have 3 slaves.
So why this is happenning? according to redis failover it should join the same master
Comment From: madolson
@aadi0105 You are right this is unexpected behavior. How are you resurrecting the dead nodes? My guess is that some part is calling CLUSTER REPLICATE to a specific node.
Comment From: aadi0105
@madolson In Kubernetes pods automatically Resurrected if any pod stops working and then redis fail over takes place when pod stops working
Comment From: madolson
@aadi0105 Let me rephrase then, what image are you running and what happens on startup of the new pod. The pod still needs some way to join the cluster, and it sounds like it's not balancing after that join call.
Comment From: aadi0105
@madolson I am using Redis : latest image for the pods and cluster has 500 milliseconds of cluster-node-timeout. when i bring down a master form the multi master redis cluster, the default behaviour should be that one of the slave of this sub master should become the master and the master which is just brang down should become the slave of this new master but sometimes the slave of another sub master becomes the new master and the master which we brang down also becomes the slave .
For example : A redis multi master cluster consists of three sub masters A, B, C which has two slaves each. When i bring down B the desired behaviour is the failover happens nad one of the slave of B should become the master and when deleted B restarts it should join the sub cluster as slave but sometimes one of the slave of C bcomes the master B and now this cluster B has three slaves.
Comment From: madolson
@aadi0105 Which image, this one (https://hub.docker.com/_/redis) ? That one isn't vended with cluster mode support.
Comment From: aadi0105
I am using this redis:alpine (https://github.com/docker-library/redis/blob/35efd6cfd58a0d0ed7293ba593090e06e37e4d02/7.0/alpine/Dockerfile) for launching the redis pods. It has cluster mode enabled
Comment From: madolson
AFAIK, that docker image does not have built in support for creating a cluster, it just creates a bunch of freestanding nodes. So you have to be adding more logic somewhere to orchestrate the nodes coming together to form the cluster. In that case, there is probably some code causing that node to replicate from the the first master instead of the one corresponding to the correct shard.
@itamarhaber Do you know much about how people use the OSS docker image and if there is documentation for bootstrapping it with cluster mode? I see a bunch of really bad tutorials online, maybe we should support that mode better.