Hi, I have problem connecting redis instances into cluster over TLS ports.

No matter what i try I’m getting stuck on „Waiting for the cluster to join……….” This happens when trying to set cluster on TLS enabled ports. When using the same ports as non-TLS everything works fine.

SETUP:

There are 3 VMs with 2 instances of redis 6.0.9 on each, on ports 6379 and 6380.
Each VM is on Oracle linux Server version 8.4 and is created with use of chef-automation and terraform. Redis is installed from Oracle Linux 8 Application Stream repository.
There is route between hosts, telnet opens connection on both ports: app access and „higher” node communication ports eg 16380 and 16379
TLS configuration contains: (full config and the end) tls-port 6379 tls-auth-clients no tls-replication yes tls-cluster yes tls-protocols TLSv1.2 also normal port is set to 0: port 0 bind is set to NIC and loopback: bind 10.128.185.206 127.0.0.1

ERRORS:

There are errors in redis log. They can be seen right from the beginning, and they occur every minute:

error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=12)

which is connected to local loopback bind, when 127.0.0.1 is removed from bind directive, Error stops showing every minute, but its still: „Waiting for the cluster to join………” when trying to set cluster.

When set loglevel to debug: On other node in logs i can see that connection was trying to establish: Oct 5 08:59:18 l000d00redb01 redis-6379[87848]: Connection with Node ee89d9868e41282b3587340f5b6270f23ba06b89 at 10.128.185.208:16379 failed: error:14094413:SSL routines:ssl3_read_bytes:sslv3 alert unsupported certificate And info about one connected client: Oct 5 08:59:19 l000d00redb01 redis-server[87848]: 87848:M 05 Oct 2021 08:59:19.234 . 1 clients connected (0 replicas), 1482728 bytes in use

But no explanation what is wrong with certificate This looks like some error due to wrong certificates.

What i checked:

openssl verify- shows certificate is signed with CA
certificate is a wildcard type for my organization, and is used with success in other solutions.
for my own sake i compiled and run redis 6.2 from github and tried set up cluster with no luck. Same error, still „Waiting for the cluster to join………”
i also tried this with generated sample self-signed certificate from redis utils script. On both 6.0.9 and 6.2 Same result.
tried with SElinux turned off.

I use this command to set up cluster: redis-cli --tls -h 10.128.185.207 --cluster create 10.128.185.208:6379 10.128.185.208:6380 10.128.185.207:6379 10.128.185.207:6380 10.128.185.206:6379 10.128.185.206:6380 --cluster-replicas 1

which gives me this message:

Performing hash slots allocation on 6 nodes... Master[0] -> Slots 0 - 5460 Master[1] -> Slots 5461 - 10922 Master[2] -> Slots 10923 - 16383 Adding replica 10.128.185.207:6380 to 10.128.185.208:6379 Adding replica 10.128.185.206:6380 to 10.128.185.207:6379 Adding replica 10.128.185.208:6380 to 10.128.185.206:6379 M: 9240da721c64de8d42cf9fcff89ab1614eb573a5 10.128.185.208:6379 slots:[0-5460] (5461 slots) master S: e334e449ef79f7651ae63afb4f20d8daff5fa598 10.128.185.208:6380 replicates b663747733a16ea068c3213bdca1e3449dae1985 M: 597f7b38162e9e64f62b7d9617b334cfea016b9b 10.128.185.207:6379 slots:[5461-10922] (5462 slots) master S: 889c6c8bdaf5e5760f38071b38e5adeec8588967 10.128.185.207:6380 replicates 9240da721c64de8d42cf9fcff89ab1614eb573a5 M: b663747733a16ea068c3213bdca1e3449dae1985 10.128.185.206:6379 slots:[10923-16383] (5461 slots) master S: 06869b485de4ed8570145e1b1fe040754f3925e7 10.128.185.206:6380 replicates 597f7b38162e9e64f62b7d9617b334cfea016b9b Can I set the above configuration? (type 'yes' to accept):

After accepting i can only see: „Waiting for the cluster to join………”

Here is my certificate:

Certificate: Data: Version: 3 (0x2) Serial Number: 4d:6c:d2:8d:00:05:00:02:1b:c0 Signature Algorithm: sha256WithRSAEncryption Issuer: DC = pl, DC = company, CN = SubCA-COMPANY Validity Not Before: Jun 30 12:36:58 2021 GMT Not After : Jun 30 12:36:58 2023 GMT Subject: C = PL, ST = Mazowieckie, L = Warszawa, O =COMPANY., OU = IT, CN = company.com Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public-Key: (2048 bit) Modulus: 00:b6:71:ca:de:6f:12:31:d4:d8:a3:12:b1:1d:90: cc:58:17:4c:6e:e5:7f:17:68:e1:05:aa:6e:66:15: 19:52:20:d3:08:53:80:f4:06:17:f8:3f:90:c1:34: d4:b2:dc:29:60:3b:aa:27:ce:99:95:b4:db:42:13: e8:78:cc:a3:7e:89:b8:04:7b:d9:4c:71:14:da:27: b3:da:ea:98:cf:ff:14:b3:aa:21:07:aa:e2:8c:f6: 4e:1f:08:2b:39:c2:da:2a:44:f6:22:d1:e9:8f:e7: 54:3b:19:93:21:0e:4b:75:85:4c:b8:19:de:00:1c: 04:c0:b3:b0:51:68:cc:3d:31:80:18:11:ef:ab:50: fd:ed:64:2f:4c:26:ba:7d:2a:99:f7:de:e4:b6:90: d7:f0:5d:97:9b:10:33:52:bb:4e:f1:67:82:58:4c: 92:8a:9b:f2:2a:cf:fe:ad:a5:d6:ec:1c:ba:0f:32: f6:30:52:87:f5:19:6c:79:27:01:28:c9:9e:49:1e: 6f:6c:03:4a:54:d1:14:c7:b2:39:6b:24:ff:f2:cf: 36:31:49:14:46:a2:14:e8:e6:14:6e:7d:09:6b:83: 45:f8:f0:81:68:21:2a:4e:2f:e3:de:4f:88:1d:bf: 30:eb:cd:10:eb:1f:e3:04:18:f1:bf:f7:ba:e5:12: d4:93 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 Subject Alternative Name: DNS:*.company.com X509v3 Subject Key Identifier: 4A:0B:1A:95:8F:DB:1F:A8:AC:34:05:A2:94:D8:4E:90:C4:26:5F:05 X509v3 Authority Key Identifier: keyid:70:11:7C:50:03:6B:9F:CC:9D:CA:4C:C3:20:62:8A:1A:B0:62:B7:97

        X509v3 CRL Distribution Points:

            Full Name:
              URI:ldap:///CN=SubCA-COMPANY(5),CN=crt002,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=company,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint
              URI:http:// crt002.company.com/CertEnroll/SubCA-COMPANY(5).crl

        Authority Information Access: 
            CA Issuers - URI:ldap:///CN=SubCA-COMPANY,CN=AIA,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=company,DC=com?cACertificate?base?objectClass=certificationAuthority
            OCSP - URI:http://crt002.company.com/ocsp

        1.3.6.1.4.1.311.21.7: 
            0/.'+.....7.........\...;...6...'.N.......k..i...
        1.3.6.1.4.1.311.21.10: 
            0.0

..+....... Signature Algorithm: sha256WithRSAEncryption 99:fe:f5:17:67:74:bd:6d:2b:de:f0:f7:48:49:0c:d5:31:1b: 36:b5:52:a6:0d:06:fd:96:79:99:b6:ba:06:88:fc:1e:e6:a1: b0:62:13:bc:22:14:e6:33:b0:0b:82:1e:a6:62:f1:af:17:ad: f4:43:bc:25:de:d7:33:65:6d:52:cd:25:79:db:7a:90:a4:68: ff:cc:26:22:f8:fb:1b:0d:f9:1e:35:e2:8e:23:ff:a5:9b:02: 3c:4c:49:74:12:25:88:ff:e9:e6:d4:44:f3:c4:e5:36:1d:b7: 48:32:1e:5f:6a:02:1b:9e:93:01:0d:c4:94:45:00:f4:c5:57: 7b:cc:98:e7:e7:13:2e:89:a7:d7:96:ae:fc:00:f2:ec:9f:23: 22:87:4e:3d:49:f7:b6:7c:73:af:59:ed:32:aa:5c:56:ac:4d: d2:28:59:96:6b:0a:e8:4c:6a:17:92:ec:88:5c:29:df:db:26: 01:09:87:06:a9:8e:e9:42:f3:63:cc:10:8b:72:95:ce:a1:f9: 21:25:dc:af:2a:43:a1:c3:d5:95:e1:7a:98:f6:65:1f:2f:d2: af:e4:b3:25:e9:26:ed:44:e7:8a:35:a2:d5:9d:a6:a4:4c:30: 6e:48:e0:61:be:ba:06:1b:3d:40:5e:cf:fa:5e:0b:ca:eb:54: ff:7f:e2:91

And config file for redis.

Redis.conf:

bind 10.128.185.206 127.0.0.1 port 0 timeout 0 tcp-keepalive 0 tls-port 6379 tls-cert-file /etc/pki/tls/certs/company.com.crt tls-key-file /etc/pki/tls/private/company.com.key tls-ca-cert-file /etc/pki/tls/certs/CA-company.com.crt tls-auth-clients no tls-replication yes tls-cluster yes tls-protocols TLSv1.2 supervised no pidfile /var/run/redis/6379/redis_6379.pid loglevel notice syslog-enabled yes syslog-ident redis-6379 syslog-facility local0 databases 16 always-show-logo yes dbfilename dump-6379.rdb
rdb-del-sync-files no dir /var/lib/redis replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no repl-diskless-sync-delay 5
repl-diskless-load disabled

repl-timeout 60

repl-backlog-size 1mb

repl-backlog-ttl 3600
replica-priority 100
acllog-max-len 128 maxclients 10000 lazyfree-lazy-eviction no lazyfree-lazy-expire no lazyfree-lazy-server-del no replica-lazy-flush no lazyfree-lazy-user-del no oom-score-adj no oom-score-adj-values 0 200 800

appendonly no

appendfilename appendonly-6379.aof appendfsync everysec
no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb aof-load-truncated yes
aof-use-rdb-preamble yes lua-time-limit 5000 cluster-enabled yes cluster-config-file nodes-6379.conf cluster-node-timeout 5000 slowlog-log-slower-than 10000 slowlog-max-len 1024 latency-monitor-threshold 0 hash-max-ziplist-entries 512 hash-max-ziplist-value 64
list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64
stream-node-max-bytes 4096 stream-node-max-entries 100 activerehashing yes
client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60 hz 10 dynamic-hz yes aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
jemalloc-bg-thread yes

Any help would be much appreciated.

Comment From: yossigo

@grykaPL The certificate validation fails, you could confirm this by enabling tls-auth-clients and trying to connect to the server using this certificate and redis-cli. You can also double-check if this works with self-signed certs or tests certs.

Note that your certificate has a server-side key usage attribute, which may prevent it from being used as a client certificate. This depends on specific versions / configurations of OpenSSL.

Comment From: huangcuiyang

this also happend to me, can not create cluster with tls, hunging in "Waiting for the cluster to join………" * Redis v6.2.5

Redis [BUG] Can't setup cluster on TLS enabled ports