We have setup Redis with sentinel high availability using 3 nodes. Suppose fist node is master, when we reboot first node, failover happens and second node becomes master until this point every thing is OK. But when fist node comes back it cannot sync with master and we saw that in its config no "masterauth" is set. Here is the error log and Generated by CONFIG REWRITE config:
1182:S 29 May 2021 13:49:42.713 * Reconnecting to MASTER 192.168.1.2:6379 after failure 1182:S 29 May 2021 13:49:42.716 * MASTER <-> REPLICA sync started 1182:S 29 May 2021 13:49:42.716 * Non blocking connect for SYNC fired the event. 1182:S 29 May 2021 13:49:42.717 * Master replied to PING, replication can continue... 1182:S 29 May 2021 13:49:42.717 * (Non critical) Master does not understand REPLCONF listening-port: -NOAUTH Authentication required. 1182:S 29 May 2021 13:49:42.717 * (Non critical) Master does not understand REPLCONF capa: -NOAUTH Authentication required. 1182:S 29 May 2021 13:49:42.717 * Partial resynchronization not possible (no cached master) 1182:S 29 May 2021 13:49:42.718 # Unexpected reply to PSYNC from master: -NOAUTH Authentication required. 1182:S 29 May 2021 13:49:42.718 * Retrying with SYNC...
===========================
Generated by CONFIG REWRITE
save 3600 1 save 300 100 save 60 10000 user default on #eb5fbb922a75775721db681c49600c069cf686765eeebaa6e18fad195812140d ~ & +@all replicaof 192.168.1.2 6379
What is the problem?
Comment From: huangzhw
You should set masterauth <master-password> and masteruser <username> in all three instances.
Comment From: tabrizu891
We set masterauth in all nodes during setup. problem is after failover and config rewrite, masterauth line doesn't exists!
Comment From: tabrizu891
We tested this scenario without AUTH and there was no problem. Whitout AUTH everything was OK.
Comment From: huangzhw
So the problem is at first masterauth in your config files. After failover, the masterauth is missing.
What's you Redis version?
And can you give the config file before and after?
Comment From: tabrizu891
@huangzhw Yes that's exactly our problem, Redis version is 6.2.3 and I will send config file in couple of hours. Thanks.
Comment From: tabrizu891
@huangzhw Here is the config files before and after failover. Scenario: 3 nodes > 10.0.85.154:6379, 10.0.85.155:6379, 10.0.85.156:6379 At start 10.0.85.154 was master, then we rebooted this node and 10.0.85.156 became master. After 10.0.85.154 came back, it wasn't able to sync with master and i put error log picture.
redis_after.conf.txt
redis_before.conf.txt
Comment From: huangzhw
I don't see masterauth set in redis_before.conf. If you set masterauth after server start, you should execute CONFIG REWRITE yourself to save this config to config file.
Comment From: tabrizu891
Yes it was my mistake and i tested failure with second node (which i checked config file for masterauth before failover) and everything was fine. Sorry for silly mistake and thank you for time and support.