Describe the bug
Sentinel takes a longer than expected time to choose a new master
To reproduce
have a configuration of : 1 redis-master 2 redis-slave 3 sentinel containers runing on each of the pods I am using helmchart version 5.0.7 by bitnami redis image: bitnami/redis:5.0.7-debian-10-r32 sentinel image: bitnami/redis-sentinel:5.0.7-debian-10-r27
changes to values.yaml : cluster: enabled: true slaveCount: 2 sentinel: enabled: true downAfterMilliseconds: 4000 failoverTimeout: 3000
delete master and wait for it to come back and check how much time passed untill first write succeded
during testing I run script that deleted the new master every couple of minutes This behavior doesn't always happen may need to repeat a couple times Expected behavior
For the write commands to resume whitin 5s actual behavior is the wait time can range between 5s to 60+s
Additional information `` Any additional information that is relevant to the problem. failover for following logs took 12s
logs from sentinel containers for each pod
redis-master-0 10.40.164.184
1:X 19 Dec 2022 16:29:02.035 # +try-failover master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:02.038 # +vote-for-leader b05d4e1c0365f592a4b2d38e6986d5a04403a6cd 45 1:X 19 Dec 2022 16:29:02.043 # 0f8395f8fa7d0d3a77105f0ccd7bdd60864de8df voted for b05d4e1c0365f592a4b2d38e6986d5a04403a6cd 45 1:X 19 Dec 2022 16:29:05.708 # -failover-abort-not-elected master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:05.761 # Next failover delay: I will not start a failover before Mon Dec 19 16:29:08 2022 1:X 19 Dec 2022 16:29:08.118 # +new-epoch 46 1:X 19 Dec 2022 16:29:08.120 # +vote-for-leader 0f8395f8fa7d0d3a77105f0ccd7bdd60864de8df 46 1:X 19 Dec 2022 16:29:08.163 # Next failover delay: I will not start a failover before Mon Dec 19 16:29:15 2022 1:X 19 Dec 2022 16:29:10.914 * +sentinel sentinel d2c8cecf746b59312ddeefb6d5228d36bd7d4350 10.40.66.219 26379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.013 # +new-epoch 47 1:X 19 Dec 2022 16:29:13.014 # +vote-for-leader d2c8cecf746b59312ddeefb6d5228d36bd7d4350 47 1:X 19 Dec 2022 16:29:13.058 # Next failover delay: I will not start a failover before Mon Dec 19 16:29:19 2022 1:X 19 Dec 2022 16:29:14.229 # +config-update-from sentinel d2c8cecf746b59312ddeefb6d5228d36bd7d4350 10.40.66.219 26379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.229 # +switch-master mymaster 10.40.73.241 6379 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.230 * +slave slave 10.40.37.36:6379 10.40.37.36 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.230 * +slave slave 10.40.171.85:6379 10.40.171.85 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.230 * +slave slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.252 * +slave slave 10.40.66.219:6379 10.40.66.219 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.240 # +sdown slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.240 # +sdown slave 10.40.171.85:6379 10.40.171.85 6379 @ mymaster 10.40.164.184 6379
redis-slave-0 10.40.37.36
1:X 19 Dec 2022 16:29:10.916 * +sentinel sentinel d2c8cecf746b59312ddeefb6d5228d36bd7d4350 10.40.66.219 26379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:11.605 # -failover-abort-not-elected master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:11.681 # Next failover delay: I will not start a failover before Mon Dec 19 16:29:14 2022 1:X 19 Dec 2022 16:29:13.017 # +new-epoch 47 1:X 19 Dec 2022 16:29:13.019 # +vote-for-leader d2c8cecf746b59312ddeefb6d5228d36bd7d4350 47 1:X 19 Dec 2022 16:29:13.020 # Next failover delay: I will not start a failover before Mon Dec 19 16:29:19 2022 1:X 19 Dec 2022 16:29:14.232 # +config-update-from sentinel d2c8cecf746b59312ddeefb6d5228d36bd7d4350 10.40.66.219 26379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.232 # +switch-master mymaster 10.40.73.241 6379 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.232 * +slave slave 10.40.37.36:6379 10.40.37.36 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.232 * +slave slave 10.40.76.17:6379 10.40.76.17 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.232 * +slave slave 10.40.171.85:6379 10.40.171.85 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.232 * +slave slave 10.40.173.94:6379 10.40.173.94 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.232 * +slave slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.299 * +slave slave 10.40.66.219:6379 10.40.66.219 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:17.380 * +fix-slave-config slave 10.40.37.36:6379 10.40.37.36 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.278 # +sdown slave 10.40.76.17:6379 10.40.76.17 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.278 # +sdown slave 10.40.171.85:6379 10.40.171.85 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.278 # +sdown slave 10.40.173.94:6379 10.40.173.94 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.278 # +sdown slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:24.411 * +fix-slave-config slave 10.40.66.219:6379 10.40.66.219 6379 @ mymaster 10.40.164.184 6379
redis-slave-1 10.40.66.219
1:X 19 Dec 2022 16:29:12.953 # +sdown master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.007 # +odown master mymaster 10.40.73.241 6379 #quorum 3/2 1:X 19 Dec 2022 16:29:13.007 # +new-epoch 47 1:X 19 Dec 2022 16:29:13.007 # +try-failover master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.011 # +vote-for-leader d2c8cecf746b59312ddeefb6d5228d36bd7d4350 47 1:X 19 Dec 2022 16:29:13.015 # b05d4e1c0365f592a4b2d38e6986d5a04403a6cd voted for d2c8cecf746b59312ddeefb6d5228d36bd7d4350 47 1:X 19 Dec 2022 16:29:13.017 # 0f8395f8fa7d0d3a77105f0ccd7bdd60864de8df voted for d2c8cecf746b59312ddeefb6d5228d36bd7d4350 47 1:X 19 Dec 2022 16:29:13.101 # +elected-leader master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.101 # +failover-state-select-slave master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.156 # +selected-slave slave 10.40.164.184:6379 10.40.164.184 6379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.156 * +failover-state-send-slaveof-noone slave 10.40.164.184:6379 10.40.164.184 6379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:13.227 * +failover-state-wait-promotion slave 10.40.164.184:6379 10.40.164.184 6379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.148 # +promoted-slave slave 10.40.164.184:6379 10.40.164.184 6379 @ mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.148 # +failover-state-reconf-slaves master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.227 # +failover-end master mymaster 10.40.73.241 6379 1:X 19 Dec 2022 16:29:14.227 # +switch-master mymaster 10.40.73.241 6379 10.40.164.184 6379 1:X 19 Dec 2022 16:29:14.227 * +slave slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.277 # +sdown slave 10.40.73.241:6379 10.40.73.241 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.712 * +slave slave 10.40.66.219:6379 10.40.66.219 6379 @ mymaster 10.40.164.184 6379 1:X 19 Dec 2022 16:29:18.716 * +slave slave 10.40.37.36:6379 10.40.37.36 6379 @ mymaster 10.40.164.184 6379
Comment From: moticless
Hi @victorgitmain, If I am not wrong Bitnami 5.0.7 is aligned with Redis version. Redis 5.0.7 was released on November 2019. Please make the effort to test it with unstable and verify if it got resolved.
Comment From: nasromm
Hello @moticless , Thank you for the response, can you please clarify what do you mean by unstable ?
Comment From: igxlin
Hello @moticless , Thank you for the response, can you please clarify what do you mean by unstable ?
@victorgitmain unstable is the default branch of redis. i.e. the latest version of redis.
Comment From: nasromm
looks like this issue was fixed in more recent version tried testing with bitnami/redis chart version: 17.7.3 app version: 7.0.8 didn't encounter this issue or issue was minimalized during testing Thanks for the sugestion