Hello,
I have a Redis Master/Slave Replication Setup with one Master and two Slaves - all running Sentiel and pointing to the IP of the master. When I start Redis and Sentinal, it runs fine for a couple of minutes. After a while, the slaves switch to the Master with IP 127.0.0.1 for no reason - the Master is up and running just fine.
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
slave_repl_offset:725181
master_link_down_since_seconds:1553251429
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:f5c88c2be7d4eaced86f693c23e4758941aa0f7f
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:725181
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:245119
repl_backlog_histlen:480063
My guess is that it has something to do with the option "bind 0.0.0.0" that I use. I guess Sentinel discovers that the master IP is 127.0.0.1 and then switches the slaves to it.
Can someone comment on this and has an idea how to solve this? And yes, binding to 0.0.0.0 is what I want in this case. Redis is running in VMs that are encapsulated from the outside world. So there is no security problem here. The reason why I use this is, because I want to bind Redis to two interfaces (127.0.0.1 as well as the external Interface).
Thanks, Marc
Comment From: ghost
To proof my suggestion, I have configured each Redis server to bind to a single IP (external interface) and then it seems to work - no more switching to 127.0.0.1.
Comment From: ghost
Also, I get a lot of these message in the sentinel logfile
1299:X 22 Mar 10:49:50.588 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:52.691 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:53.320 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:54.748 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:54.755 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:56.811 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:57.476 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:58.915 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:49:58.946 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:00.979 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:00.991 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:02.999 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:03.034 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:05.019 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 127.0.0.1 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
1299:X 22 Mar 10:50:05.054 * +sentinel-address-switch master mymaster 192.168.100.11 6379 ip 192.168.100.11 port 26379 for 700b7f9616058b2ef985b4f4b8a5e4da54d24b31
Seems like it's switching back and fourth every second.
Comment From: vattezhang
What's the configuration of your scenario? These configurations in three redis will help to find the problem: bind in redis.conf & sentinel.conf sentinel monitor in sentinel.conf replicaofreplicaof in redis.conf
Comment From: ghost
@vattezhang the bind in the redis.conf is "bind 0.0.0.0" for all three Redis servers. I have specified no bind option in the sentinel.conf - this is how it looks like
port 26379
daemonize yes
sentinel monitor mymaster 192.168.100.11 6379 2
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1
Sentinel monitor is
sentinel monitor mymaster 192.168.100.11 6379 2
but it switches to 127.0.0.1 after a while.
Hope this helps.
Comment From: eliquinox
I have solved this problem of switching between different network interfaces by specifying sentinel announce-ip param in sentinel.conf.
Comment From: yossigo
@hirschnase Can you provide more details on your environment - OS and version, containers in use, etc? It seems like a local connection to the interface's public address results with a socket that is bound to the loopback address. I think we're seeing this problem manifested in other issues but it's limited only to specific environments.
As @eliquinox correctly indicates specifying an explicit announce-ip works around it.