We recently ran into an issue related to protected mode on one of our redis sentinel clusters.
In this cluster setup we have 1 master and 1 replica and 3 sentinels.
The master and replica could sync with the master reporting:
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.55.21.139,port=6379,state=online,offset=504617812,lag=0
master_replid:c92890756078c13baf6745a6ddcff0f5e5a28329
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:504618714
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:10485760
repl_backlog_first_byte_offset:494132955
repl_backlog_histlen:10485760
From the replica we'd also see everything as fine:
127.0.0.1:6379> info replication
# Replication
role:slave
master_host:10.55.11.139
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:505290184
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:c92890756078c13baf6745a6ddcff0f5e5a28329
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:505290184
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:10485760
repl_backlog_first_byte_offset:494804425
repl_backlog_histlen:10485760
However when trying to do a manual failover from one of the sentinels we'd receive:
127.0.0.1:26379> sentinel failover events-master
(error) NOGOODSLAVE No suitable replica to promote
Also when asking sentinel for info:
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=events-master,status=ok,address=10.55.11.139:6379,slaves=1,sentinels=3
So on the surface everything about the cluster the master and replica appear in good shape except we'd continue to receive "NOGOODSLAVE No suitable replica to promote"
trying to inspect sentinel slaves event-master - is where we started to get to the root of the issue but I think the amount of insight from this was very light and made it take much longer to figure out the root cause of our issue.
127.0.0.1:26379> sentinel slaves events-master
1) 1) "name"
2) "10.55.21.139:6379"
3) "ip"
4) "10.55.21.139"
5) "port"
6) "6379"
7) "runid"
8) ""
9) "flags"
10) "s_down,slave,disconnected"
11) "link-pending-commands"
12) "3"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "3439971"
17) "last-ok-ping-reply"
18) "3439971"
19) "last-ping-reply"
20) "3439971"
21) "s-down-time"
22) "3429958"
23) "down-after-milliseconds"
24) "10000"
25) "info-refresh"
26) "1569726095178"
27) "role-reported"
28) "slave"
29) "role-reported-time"
30) "3439971"
31) "master-link-down-time"
32) "0"
33) "master-link-status"
34) "err"
35) "master-host"
36) "?"
37) "master-port"
38) "0"
39) "slave-priority"
40) "100"
41) "slave-repl-offset"
42) "0"
It wasn't until from one of the sentinel servers that we tried to connect to the replica that the issue became super obvious:
[root@ip-10-55-11-138 ~]# redis-cli -h 10.55.21.139 -p 6379
10.55.21.139:6379> info
DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside.
So - I am submitting this issue because while google'ing for our issue it was not clear that the issue was our protected mode was preventing the sentinels access. We adjusted our replica configuration to allow the sentinels and immediately sentinel slaves is reporting ok link status and manual failover works. So hopefully this report helps someone else when faced with a similar issue. Also maybe there is some more error information that might be useful in the sentinel slaves command to assist admins in troubleshooting a replica issue like this... sentinel has been a life saver for our operations team it's fantastic and redis is like the swiss army knife for problem solving when it doubt redis can fix it.
Comment From: iamhritik
@taf2 If possible, can u also share the parameters that you updated in the redis replica node config file to allow the sentinels ?