The problem/use-case that the feature addresses
I hope use domain create cluster instand of ip, I create redis cluster in kubenetes pod ,but pod ip alwaays change when pod restart.
Description of the feature
I can use domain create cluster success.
Comment From: uvletter
I guess StatefulSet is more suitable for deploying a redis cluster. BTW, redis cluster support hostname.
Comment From: zyy4676
yes, I ues sts.I create redis in 30 node,every node 2 redis-server. if use hostname, every node should write 30 node's hostsname in '/etc/hosts'. If I add on new node then need to change all "/etc/hosts" (add new node hostname). if use domain, no worry about ip change ,not neet to change '/etc/hosts'.
Comment From: uvletter
Looking into the code, I found the hostname is only used by clients, not for intra-cluster communication... If you use StatefulSet, every pod would be assigned with a fixed hostname, no need for changing the mapping between hostname and IP in /etc/hosts.
I think Pod IP changes when pod restarts doesn't matter, as Redis instance can find the cluster in the nodes.conf file, and rejoin the redis cluster. The other members in redis cluster would update the address of the restarted redis node automatically while the node name is unchanged.
Comment From: zyy4676
I think redis instance can find the cluster by nodes.conf(ID), communication use ip. I did some tests： 1) after pod restart , if pod ip not change, redis instance work normal. 2) after pod restart , if pod ip change, but the ip is a new ip, redis instance work normal. 3) after pod restart , if pod ip change and master change ip with salve (or exchaneg ip with other in instance), redis instance will work not normal.
ip always change,but domain is stable. if can use domain instand of ip create cluster in kubenetes is useful.
Comment From: uvletter
- after pod restart , if pod ip change and master change ip with salve (or exchaneg ip with other in instance), redis instance will work not normal.
I'm not sure if the cluster can recover after a slave and master restart at the same time and exchange their IPs. But the assumption for it is a little extreme: 1. It's not likely to happen in the actual production environment that master and slave crash at the same time. Given the cluster haven't turned on the persistency, master and slave restarting at the same time means data loss, so usually we would place the replicas to different AZ. 2. The range of LAN IP varies largely. The restarted pod is assigned exactly to the IP that belonged to another pod in the cluster sound like odd, that the behave can involves troubles in many system.
Comment From: uvletter
Besides, it's not recommended to restart redis in cluster mode without special process. If the master restart without persistency, it's data would be flushed, more severely, it will replicate the empty db to slaves, which means the data lose forever.
Comment From: masteroogway123
This situation may be encountered, for example, if the master node is in the same IP group, and when all master nodes are restarted, the IP addresses in this IP group may randomly correspond to different PODs.Of course, it is unlikely that all the masters will lose power, but if there are dozens of masters, 3-5 of them may lose power. They exchange IP addresses with each other when they restart.
Comment From: masteroogway123
I am not sure if this issue( https://github.com/redis/redis/issues/4289 ) will solve the problem @zyy4676
Comment From: zyy4676
@masteroogway123 thanks
Comment From: zyy4676
@uvletter Production environment may shutdown node, and all Pods will restart. The insufficient cpu/memory of the node may also cause the pod to restart. Pod restarts seem to be a very common scenario in the kubenetes cluster. Ip change alse seem to be a very common scenario in the kubenetes cluster.
Comment From: zyy4676
There have 2 way to evade this matter： 1）pod use host netwotk ,then pod ip will not change. 2) Modify the node.conf (IP) when the pod starts But both methods have their drawbacks， pod use host netwotk：if I start 2 redis-server in one node, the 2 redis-server must use 2 different port. Modify the node.conf (IP) when the pod starts : This file should be maintained by redis, we should not modify node.conf.
I think In kubenetes, a better way is to use domain.
hadoop/hbase/yarn/kubenetes seem to support domain。
Comment From: masteroogway123
I think redis instance can find the cluster by nodes.conf(ID), communication use ip. I did some tests：
- after pod restart , if pod ip not change, redis instance work normal.
- after pod restart , if pod ip change, but the ip is a new ip, redis instance work normal.
- after pod restart , if pod ip change and master change ip with salve (or exchaneg ip with other in instance), redis instance will work not normal.
ip always change,but domain is stable. if can use domain instand of ip create cluster in kubenetes is useful.
The focus of the question now is: why master change ip with other in instance(not new IP), redis instance will work not normal?
Comment From: zyy4676
I think it might be a bug. Redis uses gossip messaging。 If there are a lot of pod IPs changing, even the IP of the slave pod becomes the IP of the maste ,the slave may think that its master is itself, causing the process not mormal.
Comment From: zyy4676
At some point, the problem can be solved by restarting the process. So in a running instance, redis knows its IP from redis.conf or node.conf. If slave redis.conf ip already become master ip,but node.conf not chacge, so the slave messaging will wrong.
Comment From: uvletter
Redis cluster has some mechanism to deals with the IP drift for some common cases, but it seems it doesn't work well for the situation that two node exchanges their IPs. Maybe you can try to fix this issue, or wait for someone volunteering to pick it up.
Comment From: zyy4676
ok， I will try to fix and test.
Comment From: zyy4676
Redis transmits information through gossip. In k8s cluster, pods ip will change or exchang when pod restart.
If redis uses IP in the k8s cluster, then after most Pods restart and the IP changes or switches, there will be no way to transinformation through gossip because the IP recorded in redis-node.conf no longer exists and cannot be ping.
The domain name of the pod will not change 。If the domain name is used in the k8s cluster, even if the pod restart and IP change, redis can communicate through the domain. Redis will work properly。
Now, NET_IP_STR_LEN is 46 ( INET6_ADDRSTRLEN is 46 )。For support domain I suggest change it to 256, Because a full domain name is limited to 255 octets (including the separators). Here are articles with domain length limits ，RFC-2181 。
Comment From: kjoe
I wrote some observation/inspiration to https://github.com/redis/redis/issues/4289 if it helps.