Hello, I am using Redis 6.2.6. I have 3 node of Redis, setted up 1 master and 2 slave nodes. On them, redis-sentinel configured. All redis servers are using SSL certificates with TLS options. redis-sentinel is using SSL certificates with TLS options as well.

Sentinels are identifying redis master and slaves, it can read all states of them. But when I stop redis service on master, i can see +sdown record at sentinel log but it stucked at that step and failover not happening.

When I triger failover via sentinel cls (sentinel failover mymaster), failover occurs without any problem. Can somebody have any idea?

Redis Master config

bind 0.0.0.0 protected-mode yes port 6379 tcp-backlog 511 timeout 0 tcp-keepalive 300 tls-port 6479 tls-cert-file /opt/ssl/smpaliveli.crt tls-key-file /opt/ssl/smpaliveli.key tls-ca-cert-file /opt/ssl/smpaliveli.crt tls-auth-clients optional tls-replication yes tls-protocols "TLSv1.2" tls-ciphersuites TLS_CHACHA20_POLY1305_SHA256 tls-session-cache-timeout 60 daemonize no pidfile /var/run/redis_6379.pid loglevel notice logfile /var/log/redis/redis.log databases 16 always-show-logo no set-proc-title yes proc-title-template "{title} {listen-addr} {server-mode}" stop-writes-on-bgsave-error no rdbcompression yes rdbchecksum yes dbfilename dump.rdb rdb-del-sync-files no dir /var/lib/redis replica-serve-stale-data yes replica-read-only yes repl-diskless-sync no repl-diskless-sync-delay 5 repl-diskless-load disabled repl-disable-tcp-nodelay no replica-priority 100 requirepass ali maxmemory 10000 maxmemory-policy noeviction lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no lazyfree-lazy-user-del no lazyfree-lazy-user-flush no oom-score-adj no oom-score-adj-values 0 200 800 disable-thp yes appendonly no appendfilename "appendonly.aof" appendfsync everysec no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes aof-use-rdb-preamble yes lua-time-limit 5000 slowlog-max-len 128 latency-monitor-threshold 0 notify-keyspace-events "" list-max-ziplist-size -2 list-compress-depth 0 set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 hll-sparse-max-bytes 3000 stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes client-output-buffer-limit normal 0 0 0 client-output-buffer-limit replica 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes rdb-save-incremental-fsync yes jemalloc-bg-thread yes

Redis Slave config

Redis Master Sentinel Config

bind 0.0.0.0

port 26379

daemonize no

pidfile "/var/run/redis-sentinel.pid"

logfile "/var/log/redis/sentinel.log"

dir "/tmp"

sentinel monitor redis-cluster 172.21.26.1 6479 2

sentinel auth-pass redis-cluster ali

sentinel down-after-milliseconds redis-cluster 1000

acllog-max-len 128

sentinel parallel-syncs redis-cluster 2

sentinel failover-timeout redis-cluster 60000

sentinel deny-scripts-reconfig yes

sentinel resolve-hostnames no

sentinel announce-hostnames no

REDIS_SENTINEL_TLS_ENABLED=yes

tls-replication yes tls-cert-file "/opt/ssl/smpaliveli.crt" tls-key-file "/opt/ssl/smpaliveli.key" tls-ca-cert-file "/opt/ssl/smpaliveli.crt"

Generated by CONFIG REWRITE

protected-mode no supervised systemd user default on nopass ~ & +@all sentinel myid 8f03bdeaa3f7e76d23f69b954f9afbd61ad0dcd0 sentinel config-epoch redis-cluster 0 sentinel leader-epoch redis-cluster 0 sentinel current-epoch 0 sentinel known-replica redis-cluster 172.21.26.3 6479 sentinel known-replica redis-cluster 172.21.26.2 6479 sentinel known-sentinel redis-cluster 172.21.26.2 26379 7b8515d46a5691e5d399ee23c27cf3e52b56f5a5 sentinel known-sentinel redis-cluster 172.21.26.3 26379 cd8455ab0a1fe7c569663af8c8625323a74be367

Comment From: Alfacapital1

Comment From: yossigo

@cihantunali Did you try to reproduce this on the same environment, without TLS?

Comment From: cihantunali

@yossigo Yes I did, It is working without TLS.

Comment From: moticless

@cihantunali, Sentinel is configured to connect to redis with TLS, while at the same time, it attempts to expose its own port without TLS. Now, enabling tls-replication affects both connections and therefore sentinel port must be aligned with redis connection and expose tls-port rather than a plain port in the configuration.

Please try to replace port in sentinel configuration with tls-port and see if it makes a difference. Thanks

Comment From: cihantunali

Hello @moticless , When I changed to port to tls-port, sentinel could not run because of port binding. It says, "Already binded" but it is not. If I changed port number different than 26379, it starts to work but still shows and "26379" when i checked from systemctl. I am researching it.

Comment From: cihantunali

update: with tls-port definition, I used 36379, It worked, it successfully failover automatically. But rollback is not working for a reason which I could not find. I stopped redis01 as master and it failed over without any problem. But when I re-open redis01, sentinel sensed it but could not turn others to slave. Thus there are 2 masters :(

Redis Redis-sentinel TLS support problem

Sentinel can not convert existing master to slave when original master comes back online and this problem won't shown up when not using TLS.

Comment From: moticless

Please verify that: - On restart, instances preserve same IP and port. - Configuration files are persisted over instance restarts and writable by instance (search in config file for # Generated by CONFIG REWRITE)

Comment From: moticless

Here is my attempt to reproduce your issue. Thanks.

Comment From: moticless

@cihantunali, did you have any progress with this issue?

Comment From: cihantunali

Hello Moti, Sorry, I could not try it out because of health issues. But this week I will try and update the case. Thanks.

Moti Cohen @.***>, 30 Oca 2022 Paz, 11:05 tarihinde şunu yazdı:

@cihantunali https://github.com/cihantunali, did you have any progress with this issue?

— Reply to this email directly, view it on GitHub https://github.com/redis/redis/issues/9952#issuecomment-1025091883, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMNYYO47DS64LKQTP6SIHJDUYTWMNANCNFSM5KGEMTNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>