I have a very simple Redis Sentinel Setup as
Redis 01 - 10.10.12.79 (master) Redis 02 - 10.10.12.80
Sentinel 01 - 10.10.12.79 Sentinel 02 - 10.10.12.80 Sentinel 03 - 10.10.12.77
HAProxy - 10.10.12.77
Both Redis hace same config except Redis 02 have "slaveof 10.10.12.80" setting, and listening on 6380
All sentinels have same following settings
port 26379 daemonize yes pidfile "/var/run/redis/sentinel.pid" loglevel notice logfile "/var/log/redis/redis-sentinel.log" sentinel monitor mymaster 10.10.12.79 6380 2 sentinel down-after-milliseconds mymaster 10000 sentinel failover-timeout mymaster 20000 sentinel parallel-syncs mymaster 1
Setup works fine until I "debug segfault" master. Ideally sentinel should elect 10.10.12.80 as a new master but it didnt. moreover logs on all sentinels show only
22460:X 15 Mar 14:55:15.998 # +sdown master mymaster 10.10.12.79 6380
Sentinel master results Sentinel 01 ##### redis-cli -p 26379 sentinel masters
1) 1) "name" 2) "mymaster" 3) "ip" 4) "10.10.12.79" 5) "port" 6) "6380" 7) "runid" 8) "d3c0bfd9046dfbb54d482ff0a9435adc7cccd538" 9) "flags" 10) "master" 11) "link-pending-commands" 12) "0" 13) "link-refcount" 14) "1" 15) "last-ping-sent" 16) "0" 17) "last-ok-ping-reply" 18) "481" 19) "last-ping-reply" 20) "481" 21) "down-after-milliseconds" 22) "10000" 23) "info-refresh" 24) "3095" 25) "role-reported" 26) "master" 27) "role-reported-time" 28) "3833705" 29) "config-epoch" 30) "0" 31) "num-slaves" 32) "1" 33) "num-other-sentinels" 34) "2" 35) "quorum" 36) "2" 37) "failover-timeout" 38) "20000" 39) "parallel-syncs" 40) "1"
Sentinel 02 ##### redis-cli -p 26379 sentinel masters
1) 1) "name" 2) "mymaster" 3) "ip" 4) "10.10.12.79" 5) "port" 6) "6380" 7) "runid" 8) "d3c0bfd9046dfbb54d482ff0a9435adc7cccd538" 9) "flags" 10) "master" 11) "link-pending-commands" 12) "0" 13) "link-refcount" 14) "1" 15) "last-ping-sent" 16) "0" 17) "last-ok-ping-reply" 18) "694" 19) "last-ping-reply" 20) "694" 21) "down-after-milliseconds" 22) "10000" 23) "info-refresh" 24) "8171" 25) "role-reported" 26) "master" 27) "role-reported-time" 28) "3962870" 29) "config-epoch" 30) "0" 31) "num-slaves" 32) "1" 33) "num-other-sentinels" 34) "2" 35) "quorum" 36) "2" 37) "failover-timeout" 38) "20000" 39) "parallel-syncs" 40) "1"
Sentinel 03 ##### redis-cli -p 26379 sentinel masters
1) 1) "name" 2) "mymaster" 3) "ip" 4) "10.10.12.79" 5) "port" 6) "6380" 7) "runid" 8) "d3c0bfd9046dfbb54d482ff0a9435adc7cccd538" 9) "flags" 10) "master" 11) "link-pending-commands" 12) "0" 13) "link-refcount" 14) "1" 15) "last-ping-sent" 16) "0" 17) "last-ok-ping-reply" 18) "433" 19) "last-ping-reply" 20) "433" 21) "down-after-milliseconds" 22) "10000" 23) "info-refresh" 24) "2242" 25) "role-reported" 26) "master" 27) "role-reported-time" 28) "3991134" 29) "config-epoch" 30) "0" 31) "num-slaves" 32) "1" 33) "num-other-sentinels" 34) "2" 35) "quorum" 36) "2" 37) "failover-timeout" 38) "20000" 39) "parallel-syncs" 40) "1"
Comment From: badboy
Did you check the logs of all Sentinels?
Comment From: arsalanniazi
Yes, it only showed following entry when disaster happened
22460:X 15 Mar 14:55:15.998 # +sdown master mymaster 10.10.12.79 6380
Comment From: schlitzered
i suspect you will have some messages like this in your sentinel logs "+sdown sentinel ...."
recent redis version introduced the "protected-mode" option, which defaults to yes.
with protected-mode set to yes, redis instances, without a password set will not allow remote clients to execute commands.
this also affects sentinels master election.
try it with setting "protected-mode no" in the sentinels. this will allow them to talk to each other.
setting a password for sentinels is currently not implemented. but maybe it will be with #1904
Comment From: jflebeau
Stumbled upon this today, thanks @schlitzered
Comment From: pcmanprogrammeur
Thank you very much @schlitzered
Comment From: amit777
so I have virtually the same setup. I checked the protected-mode and it's set to no for both the redis-server and redis-sentinel configurations. Any other ideas on why a new master would not get elected? the last line entry in all the sentinel logs is:
24057:X 25 Jun 2020 22:13:55.437 # +sdown master mymaster