My redis cluster situation
- One Master
- One Slave
- Three sentinel Runtime: k8s cluster
I use the statefulsets to deploy redis instances. when i create successfully, i deleted pod of redis master in order to simulate failover. After a period of time, i run sentinel master mymaster, the output tell me:
master address not changed
So, think that the failover is fail. The sentinel log is below:
The content of this log show me that: failover start well, but the fail happend when leader of sentinel want to elected a slave to promote a master. It can't to find any useable slave.
Furthermore, i check the situation of slave in sentinel: sentinel slaves mymaster。As expected, the infomation of slave is empty.
MyConclusion:
sentinel discover slave fail result in failover fail.
MyQuestion:
Why sentinel discovering fail? How to find the root case of that?
Comment From: fengzixu
My Sentinel Config：
sentinel monitor mymaster %master% 6379 2 sentinel auth-pass mymaster %password% sentinel down-after-milliseconds mymaster %down-failover-time% sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 logfile "" loglevel debug syslog-enabled yes syslog-ident native-redis bind 0.0.0.0
Comment From: fengzixu
My Master Config
sentinel monitor mymaster %master% 6379 2 sentinel auth-pass mymaster %password% sentinel down-after-milliseconds mymaster %down-failover-time% sentinel failover-timeout mymaster 180000 sentinel parallel-syncs mymaster 1 logfile "" loglevel debug syslog-enabled yes syslog-ident native-redis bind 0.0.0.0
Comment From: fengzixu
My Slave Config
daemonize no pidfile /var/run/redis.pid port 6379 tcp-backlog 511 bind 0.0.0.0 timeout 30 maxclients 10000 tcp-keepalive 60 loglevel debug logfile "" syslog-enabled yes syslog-ident native-redis databases 16 save 900 1 save 300 10 save 60 10000 stop-writes-on-bgsave-error yes rdbcompression yes rdbchecksum yes dbfilename dump.rdb dir "/redis-data" slaveof %master-ip% 6379 masterauth %redis-pass% slave-serve-stale-data yes rename-command FLUSHALL "" rename-command FLUSHDB "" slave-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-disable-tcp-nodelay no repl-timeout 600 repl-ping-slave-period 10 repl-backlog-size 16mb repl-backlog-ttl 3600 slave-priority 100 requirepass %redis-pass% appendonly yes appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite yes auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes lua-time-limit 5000 slowlog-log-slower-than 10000 slowlog-max-len 1024 latency-monitor-threshold 0 notify-keyspace-events "" hash-max-ziplist-entries 512 hash-max-ziplist-value 64 list-max-ziplist-entries 512 list-max-ziplist-value 64 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 aof-rewrite-incremental-fsync yes slave-announce-ip %slave_ip% slave-announce-port 6379
Comment From: jonahharris
You have TCP 26379 open, correct? Also, I've seen this with password-related issues in the past. Can you post a full sentinel log as well as server/slave logs?
Comment From: fengzixu
sentinel-2 whole log is here. download url @jonahharris
Comment From: fengzixu
@antirez
Comment From: alvinlee001
I have a similar architecture but without kubernetes, just 2 master-slave redis instances living in 2 VMs along side with 3 sentinels. From my logs somehow the slave is marked with “+sdown” after it has been started. @fengzixu Have u found a solution to this?
Comment From: alvinlee001
Turns out the issue is gone with bind 0.0.0.0, it will make weird issues go away for many. bind 0.0.0.0 is magical