Hi there,

I having a replication + sentinel which includes of 3 server. We had a weird problem yesterday when one guy from development team changed the password for redis authentication from client side (was a wrong password). It made sentinel do failover every about 3-5 minutes till now. Today after a quick correction, the password is back as it should and sentinel stop doing failover.

The question is why the wrong password from client side could make sentinel do failover many times like that? I tried to read sentinel config and redis config but i still could not figure out where config problem is related to this issue.

Can everybody advise me please?

Thanks in advance.

Son

Comment From: Linkerist

In redis.conf, you can see a section as following,

# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the slave to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the slave request.
#
# masterauth <master-password>
...
# Require clients to issue AUTH <PASSWORD> before processing any other
# commands.  This might be useful in environments in which you do not trust
# others with access to the host running redis-server.
#
# This should stay commented out for backward compatibility and because most
# people do not need auth (e.g. they run their own servers).
#
# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
# requirepass foobared

and in sentinel.conf,

# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and slaves.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for slaves, so it is not
# possible to set a different password in masters and slaves instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd

When you deploy replication & sentinel, you should set the same password for master and slave. And set it as auth-pass in sentinel. In raft routine, sentinel will perform scheduled operations for the specified Redis instance periodly. they may act as your description in the process of sending the AUTH command with the specified master password. Note that for slaves the password set for the master is used.
@sonchu

Comment From: sonchu

Hi @Linkerist ,

Thank you for quick response. Every steps are applied fine from my side (I am system administrator). The issue is from Client side when authentication password was incorrect and it caused replication failover. That's why i don't understand which caused the issue with failover.

Can you advise please?

Thanks in advance.

Son

Comment From: hwware

Hello @sonchu ,sorry for the late reply, the reason it is causing continuous fail over is because by default every 1 sec sentinel will send ping command to check the availability of monitoring masters, if the configured down-after-milliseconds passed and it doesn't successfully receive the correct reply, it will mark the master for S_DOWN state and that may cause the failover if most of sentinel "agree" this master is offline. Back to your case, since the client side modified the password, it caused Sentinel PING command get an invalid reply(in this case is a NOAUTH error). This makes the failover being triggered after down-after-milliseconds.