Describe the bug
Sentinel Autofailover doesnt work went adding password to default user or disable it
To reproduce
3 nodes,1 master, 2 slaves. Enable ACL in Sentinel and Redis
- redis: change default user to password "something"
- sentinel: change default user to password "something" (bug can be reproduced it by disabling default: ACL SETUSER default off= sentinel.conf:
Security
requirepass "sentineladmin"
sentinel monitor dev-master 10.20.192.105 6379 2 sentinel down-after-milliseconds dev-master 5000 sentinel failover-timeout dev-master 60000
sentinel auth-user dev-master replicauser sentinel auth-pass dev-master replicapass
Expected behavior
Replica and Sentinel works fine. If i disable master node (stop redis), any of the nodes get selected as new master node. If i manually do a failover in some of the nodes (sentinel failover dev-master), the failover and sync is successfull.
But no automatic failover happends (cant see nothing specific in the logs). If i change config to dont touch "default" user, automatic failover works
Additional information It looks like "default" user is used for autofailover. I have an user defined for replication I dont want to default user to be exposed on sentinel.
Comment From: hwware
Do you enable ACL in redis by redis-cli command, such as ACL SETUSER default >password or ACL SETUSER replicauser >replicapass OR you add the line: user replicauser on #d87764663a8fa4f9070761c10662fba88d93356e642d7e87ba0fee5608282fa8 resetchannels +@all in the config file?
If you run the ACL SETUSER to a new user, the new user has no any permission to run any command, you need to run: ACL SETUSER newuser +@all
If possbile, could you please post all master, replica, and sentinel node config file here, thanks
Comment From: andrescolodrero
HI, I generrate an entry in sentinel.conf. Working configuration working sentinel.conf: https://gist.github.com/andrescolodrero/3af7c993b72b450a9e300d5954088e51 redis.conf: https://gist.github.com/andrescolodrero/e7b69224e880296073a7dc0b96c087e3
Not working configuration, adding a password to default and git it full permissions: https://gist.github.com/andrescolodrero/6be794a7aafb14321374c9bb40f7b5ba
IF i check replication: 127.0.0.1:6379> auth default default OK 127.0.0.1:6379> info replication
Replication
role:master connected_slaves:2 slave0:ip=10.20.192.105,port=6379,state=online,offset=31190663,lag=0 slave1:ip=10.20.192.109,port=6379,state=online,offset=31190663,lag=0 master_failover_state:no-failover master_replid:c4b620a8a7ee44b1da12783fe567b0650f3989c9 master_replid2:40dd6562fee1ee6f4e9c2124472fcc1faca8db3f master_repl_offset:31190949 second_repl_offset:31125728 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:31125728 repl_backlog_histlen:65222
If i check sentinel:
root@tisrdsx53:/etc/redis# redis-cli -p 26379
127.0.0.1:26379> auth default default
OK
Sentinel
sentinel_masters:1 sentinel_tilt:0 sentinel_tilt_since_seconds:-1 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=dev-master,status=ok,address=10.20.192.110:6379,slaves=2,sentinels=3 127.0.0.1:26379> info snetinel 127.0.0.1:26379> info sentinel
Sentinel
sentinel_masters:1 sentinel_tilt:0 sentinel_tilt_since_seconds:-1 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=dev-master,status=ok,address=10.20.192.110:6379,slaves=2,sentinels=3
Now i stop redis master, those are he only events i can see on sentinel logs: 547041:X 30 Jul 2022 10:21:17.674 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 547041:X 30 Jul 2022 10:21:17.674 # Redis version=7.0.3, bits=64, commit=00000000, modified=0, pid=547041, just started 547041:X 30 Jul 2022 10:21:17.674 # Configuration loaded 547041:X 30 Jul 2022 10:21:17.675 * monotonic clock: POSIX clock_gettime 547041:X 30 Jul 2022 10:21:17.677 * Running mode=sentinel, port=26379. 547041:X 30 Jul 2022 10:21:17.694 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:21:17.694 # Sentinel ID is 7b89eeaef3191cc8ea26a1a2821d1062f95ab4d1 547041:X 30 Jul 2022 10:21:17.694 # +monitor master dev-master 10.20.192.110 6379 quorum 2 547041:X 30 Jul 2022 10:21:17.711 * +slave slave 10.20.192.105:6379 10.20.192.105 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:21:17.719 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:21:17.719 * +slave slave 10.20.192.109:6379 10.20.192.109 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:21:17.727 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:21:19.639 * +sentinel sentinel a7010d1159fe5e1b9e29c0ddc457644f901b08fb 10.20.192.110 26379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:21:19.643 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:21:19.841 * +sentinel sentinel 896a26e47b8f0e5f5aecd74c5b58e1c6c375164c 10.20.192.109 26379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:21:19.847 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:21:24.686 # +sdown sentinel a7010d1159fe5e1b9e29c0ddc457644f901b08fb 10.20.192.110 26379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:21:24.888 # +sdown sentinel 896a26e47b8f0e5f5aecd74c5b58e1c6c375164c 10.20.192.109 26379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:30:38.709 # +sdown master dev-master 10.20.192.110 6379
I can see the 2 slaves are communicating, but any of them being elected as master. After some minutes, Sign into some slave: 127.0.0.1:26379> auth default default OK 127.0.0.1:26379> sentinel failover dev-master OK
547041:X 30 Jul 2022 10:34:06.815 # Executing user requested FAILOVER of 'dev-master' 547041:X 30 Jul 2022 10:34:06.816 # +new-epoch 1 547041:X 30 Jul 2022 10:34:06.816 # +try-failover master dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:06.829 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:34:06.829 # +vote-for-leader 7b89eeaef3191cc8ea26a1a2821d1062f95ab4d1 1 547041:X 30 Jul 2022 10:34:06.829 # +elected-leader master dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:06.829 # +failover-state-select-slave master dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:06.930 # +selected-slave slave 10.20.192.105:6379 10.20.192.105 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:06.930 * +failover-state-send-slaveof-noone slave 10.20.192.105:6379 10.20.192.105 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:06.990 * +failover-state-wait-promotion slave 10.20.192.105:6379 10.20.192.105 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:07.250 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:34:07.250 # +promoted-slave slave 10.20.192.105:6379 10.20.192.105 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:07.250 # +failover-state-reconf-slaves master dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:07.303 * +slave-reconf-sent slave 10.20.192.109:6379 10.20.192.109 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:08.258 * +slave-reconf-inprog slave 10.20.192.109:6379 10.20.192.109 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:08.259 * +slave-reconf-done slave 10.20.192.109:6379 10.20.192.109 6379 @ dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:08.349 # +failover-end master dev-master 10.20.192.110 6379 547041:X 30 Jul 2022 10:34:08.349 # +switch-master dev-master 10.20.192.110 6379 10.20.192.105 6379 547041:X 30 Jul 2022 10:34:08.349 * +slave slave 10.20.192.109:6379 10.20.192.109 6379 @ dev-master 10.20.192.105 6379 547041:X 30 Jul 2022 10:34:08.349 * +slave slave 10.20.192.110:6379 10.20.192.110 6379 @ dev-master 10.20.192.105 6379 547041:X 30 Jul 2022 10:34:08.355 * Sentinel new configuration saved on disk 547041:X 30 Jul 2022 10:34:13.360 # +sdown slave 10.20.192.110:6379 10.20.192.110 6379 @ dev-master 10.20.192.105 6379 root@tisrdsc61:/home/andres#
Comment From: andrescolodrero
So this is how i made it work. i add authentication to sentinel, but didnt touch Default account. Somehow, redis reconfigure "default" user and setup ACL on sentinel.
Sentinel config:
# Generated by Ansible
# sentinel_26379.conf
supervised systemd
protected-mode no
dir /var/lib/redis/
pidfile /var/run/redis/sentinel_26379.pid
port 26379
# Security
requirepass sentineladmin
sentinel monitor dev-master 10.20.192.110 6379 2
sentinel down-after-milliseconds dev-master 5000
sentinel parallel-syncs dev-master 1
#elegible number of replicas for failover
sentinel parallel-syncs dev-master 1
sentinel failover-timeout dev-master 60000
sentinel auth-user dev-master replicauser
sentinel auth-pass dev-master #d87764663a8fa4f9070761c10662fba88d93356e642d7e87ba0fee5608282fa8
logfile /var/log/redis/redis-sentinel.log
# Keep Default as it is
user admin on >#713bfda78870bf9d1b261f565286f85e97ee614efe5f0faf7c34e7ca4f65baca ~* &* +@all
user replicauser on >#ba09c3a8ebd110305655903012d4e6e2312d687af1904a98072922002dda5e71 ~* &* +@all
After start sentinel, config rewrite the configuration file like this (check the entry for default user)
# sentinel_26379.conf
supervised systemd
protected-mode no
dir "/var/lib/redis"
pidfile "/var/run/redis/sentinel_26379.pid"
port 26379
# Security
requirepass "sentineladmin"
sentinel monitor dev-master 10.20.192.110 6379 2
sentinel down-after-milliseconds dev-master 5000
#elegible number of replicas for failover
sentinel failover-timeout dev-master 60000
sentinel auth-user dev-master replicauser
sentinel auth-pass dev-master #d87764663a8fa4f9070761c10662fba88d93356e642d7e87ba0fee5608282fa8
logfile "/var/log/redis/redis-sentinel.log"
# Keep Default as it is
user admin on #713bfda78870bf9d1b261f565286f85e97ee614efe5f0faf7c34e7ca4f65baca ~* &* +@all
user default on #13be1c500e163ddb686ed78dcfe3bfc117f12e6b4c0d6326b30895d0b12439f1 ~* &* +@all
# Generated by CONFIG REWRITE
latency-tracking-info-percentiles 50 99 99.9
user replicauser on #d87764663a8fa4f9070761c10662fba88d93356e642d7e87ba0fee5608282fa8 ~* &* +@all
sentinel myid 914b6e95860b93ea643693dac7bab7c10c677bef
sentinel config-epoch dev-master 0
sentinel leader-epoch dev-master 0
sentinel current-epoch 0
sentinel known-replica dev-master 10.20.192.105 6379
sentinel known-replica dev-master 10.20.192.109 6379
sentinel known-sentinel dev-master 10.20.192.105 26379 a94e43beee01a86bd84f1579076b56c126c20414
Now i can auth to sentinel wuth "admin" or "replicauser" account .
Comment From: hwware
Sorry, I have a little bit confused about your description here. Now after you auth to sentinel wuth "admin" or "replicauser" account, the auto failover works or not?
Please confirm here.
Thanks
Comment From: andrescolodrero
In the end it doesnt work when: 1. Disabling default user 2. Changing default user password.
In my case it works when: 1. Adding Auth (replicauser) and requirepass (forget about admin) 2. Dont touch "default" user in "sentinel.conf" 3. Again, i see this automatic entry once i start redis: user default on #13be1c500e163ddb686ed78dcfe3bfc117f12e6b4c0d6326b30895d0b12439f1 ~ & +@all
Documentation seems confusing, but it finally works
Comment From: hwware
Great, we will update the doc to avoid the confusion, Thanks for your confirmation
Comment From: AbhinavKoul
Hi @hwware , was the doc updated with the fix for this? I am facing a similar issue where redis seems to add ACL default user line automatically that overrides requirepass command.