Redis [QUESTION] Can sentinel mode Redis cluster support both TLS port and plain port at the same time?

Hi Team, I want to extend this question further !

We have sentinel mode Redis cluster running on plain port , we would like to enable TLS port for existing both Redis and sentinel , such that we can migrate the client applications from existing plain port connections to TLS without down time, or migrating the data new cluster .

I found its not working , the moment we have tls-replicationenabled on redis , all the internal node communication b/w redis <- ->redis happens over TLS, and the replica wont be able to connect with its leader!

17806:S 25 Apr 2023 15:44:53.871 # Timeout connecting to the MASTER...
17805:S 25 Apr 2023 15:44:53.872 * Reconnecting to MASTER 127.0.0.1:6379 after failure
17806:S 25 Apr 2023 15:44:53.872 * MASTER <-> REPLICA sync started

here is the set up i have used.

#!/bin/sh

# kill running instances
ps aux | grep ./redis-server | grep -v grep | awk '{print $2}' | xargs kill -9
ps aux | grep ./redis-sentinel | grep -v grep | awk '{print $2}' | xargs kill -9

# start redis
echo "masterauth Redis123 " >> redis-6379.conf
echo "requirepass Redis123 " >> redis-6379.conf
echo "masterauth Redis123 " >> redis-6380.conf
echo "requirepass Redis123 " >> redis-6380.conf
echo "masterauth Redis123 " >> redis-6381.conf
echo "requirepass Redis123 " >> redis-6381.conf

echo "protected-mode no" >> redis-6379.conf
echo "protected-mode no" >> redis-6380.conf
echo "protected-mode no" >> redis-6381.conf

echo "tls-replication yes" >> redis-6379.conf
echo "tls-replication yes" >> redis-6380.conf
echo "tls-replication yes" >> redis-6381.conf

echo "tls-auth-clients yes" >> redis-6379.conf
echo "tls-auth-clients yes" >> redis-6380.conf
echo "tls-auth-clients yes" >> redis-6381.conf

echo "loglevel debug" >> redis-6379.conf
echo "loglevel debug" >> redis-6380.conf
echo "loglevel debug" >> redis-6381.conf


nohup ./redis-server redis-6379.conf --port 6379  --tls-port 6479  --dbfilename 6379.rdb --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile redis-6379.log &
nohup ./redis-server redis-6380.conf --port 6380  --tls-port 6480  --dbfilename 6380.rdb --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile redis-6380.log &
nohup ./redis-server redis-6381.conf --port 6381  --tls-port 6481  --dbfilename 6381.rdb --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile redis-6381.log &

# wait for redis
sleep 10;

#Initialize replicas
#./redis-cli -p 6380  -a Redis123 slaveof 127.0.0.1 6379 
#./redis-cli -p 6381  -a Redis123 slaveof 127.0.0.1 6379
./redis-cli -p 6480  -a Redis123  --tls  --cert ./tls/redis.crt --key ./tls/redis.key --cacert ./tls/ca.crt slaveof 127.0.0.1 6479 
./redis-cli -p 6481  -a Redis123  --tls  --cert ./tls/redis.crt --key ./tls/redis.key --cacert ./tls/ca.crt slaveof 127.0.0.1 6479


# start nonTLS Sentinel cluster
echo "masterauth Sentinel123 " > sentinel-16379.conf
echo "requirepass Sentinel123 " >> sentinel-16379.conf
echo "masterauth Sentinel123 " > sentinel-16380.conf
echo "requirepass Sentinel123 " >> sentinel-16380.conf
echo "masterauth Sentinel123 " > sentinel-16381.conf
echo "requirepass Sentinel123 " >> sentinel-16381.conf

echo "protected-mode no" >> sentinel-16379.conf
echo "protected-mode no" >> sentinel-16380.conf
echo "protected-mode no" >> sentinel-16381.conf

echo "loglevel debug" >> sentinel-16379.conf
echo "loglevel debug" >> sentinel-16380.conf
echo "loglevel debug" >> sentinel-16381.conf

echo "sentinel monitor shard_1 127.0.0.1 6379 2" >> sentinel-16379.conf 
echo "sentinel auth-pass shard_1 Redis123" >> sentinel-16379.conf
echo "sentinel monitor shard_1 127.0.0.1 6379 2" >> sentinel-16380.conf
echo "sentinel auth-pass shard_1 Redis123" >> sentinel-16380.conf
echo "sentinel monitor shard_1 127.0.0.1 6379 2" >> sentinel-16381.conf
echo "sentinel auth-pass shard_1 Redis123" >> sentinel-16381.conf


nohup ./redis-sentinel sentinel-16379.conf --port 16379  --logfile sentinel-16379.log &
nohup ./redis-sentinel sentinel-16380.conf --port 16380  --logfile sentinel-16380.log &
nohup ./redis-sentinel sentinel-16381.conf --port 16381  --logfile sentinel-16381.log &


# start TLS Sentinel cluster
echo "masterauth Sentinel123 " > sentinel-16479.conf
echo "requirepass Sentinel123 " >> sentinel-16479.conf
echo "masterauth Sentinel123 " > sentinel-16480.conf
echo "requirepass Sentinel123 " >> sentinel-16480.conf
echo "masterauth Sentinel123 " > sentinel-16481.conf
echo "requirepass Sentinel123 " >> sentinel-16481.conf

echo "protected-mode no" >> sentinel-16479.conf
echo "protected-mode no" >> sentinel-16480.conf
echo "protected-mode no" >> sentinel-16481.conf

echo "tls-replication yes" >> sentinel-16479.conf
echo "tls-replication yes" >> sentinel-16480.conf
echo "tls-replication yes" >> sentinel-16481.conf

echo "tls-auth-clients yes" >> sentinel-16479.conf
echo "tls-auth-clients yes" >> sentinel-16480.conf
echo "tls-auth-clients yes" >> sentinel-16481.conf

echo "loglevel debug" >> sentinel-16479.conf
echo "loglevel debug" >> sentinel-16480.conf
echo "loglevel debug" >> sentinel-16481.conf

echo "sentinel monitor shard_tls_1 127.0.0.1 6479 2" >> sentinel-16479.conf
echo "sentinel auth-pass shard_tls_1 Redis123" >> sentinel-16479.conf
echo "sentinel monitor shard_tls_1 127.0.0.1 6479 2" >> sentinel-16480.conf
echo "sentinel auth-pass shard_tls_1 Redis123" >> sentinel-16480.conf
echo "sentinel monitor shard_tls_1 127.0.0.1 6479 2" >> sentinel-16481.conf
echo "sentinel auth-pass shard_tls_1 Redis123" >> sentinel-16481.conf


nohup ./redis-sentinel sentinel-16479.conf --port 0  --tls-port 16479  --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile sentinel-16479.log &
nohup ./redis-sentinel sentinel-16480.conf --port 0  --tls-port 16480  --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile sentinel-16480.log &
nohup ./redis-sentinel sentinel-16481.conf --port 0  --tls-port 16481  --tls-cert-file ./tls/redis.crt  --tls-key-file ./tls/redis.key  --tls-ca-cert-file ./tls/ca.crt --logfile sentinel-16481.log &

am i missing something , or is that even possible ? please suggest, thanks in advance!

Comment From: satheeshaGowda

Hello @itamarhaber @oranagra , hope all is well!

would you mind if i request you to share some insight here ?

Comment From: moticless

Hi @satheeshaGowda , the part that sentinel without TLS gets failed is because sentinels discover the replicas via the master and the master reply to them with list of replicas and corresponding list of their tls-port which obviously won't let sentinel communicate successfully with replicas. As it turned out, sentinel doesn’t do a good job in defining the support of TLS along with non-TLS communication. I intend to investigate the part that sentinel with TLS gets failed. I would have expect it to work. I will let you know what are my findings. Thanks.

Comment From: moticless

ok. I managed to use your script successfully (and then simulate failover) with two modifications: * Based on my last comment, I tried to use only sentinel with TLS (The one with comment start TLS Sentinel cluster) * Took care to cleanup at start the files redis-6379.conf, redis-6380.conf and redis-6381.conf. Maybe it is your problem as you tried to re-run the script but those files had leftovers that might impact your test.

Comment From: moticless

Hi @satheeshaGowda , any updates?