I've deployed redis 6.0.5 in Kubernetes and enable TLS.

The config file : 

 cluster-enabled yes
 cluster-require-full-coverage no
 cluster-node-timeout 15000
 cluster-config-file /data/nodes.conf
 cluster-migration-barrier 1
 appendonly yes
 protected-mode no
 tls-ca-cert-file /etc/tls/redis/ca.pem
 tls-key-file /etc/tls/redis/redis-key.pem
 tls-cert-file /etc/tls/redis/redis.pem
 tls-replication yes
 tls-cluster yes
 port 0
 tls-port 6379
 tls-auth-clients no
 tls-protocols "TLSv1.2 TLSv1.3"

The healthcheck and readiness are as follow (They are not failing)

readinessProbe:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 15
  timeoutSeconds: 5
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - "redis-cli -h $(hostname) ping"
  initialDelaySeconds: 20

Right after the servers start I get the following error : 

11:S 01 Jul 2020 19:11:42.086 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=8)
11:S 01 Jul 2020 19:11:42.087 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=8)
11:S 01 Jul 2020 19:11:43.084 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=8)

Is there anything I miss in the config ? Is there anyway to skip certificate check ?

Comment From: hasan4791

in the liveness & readiness probe, you need to provide the --tls & --cert arguments too. Check here

Comment From: yakhyadabo

I've set --tls, --cacert, --cert and --key arguments. But I'm still having the same error message.

Comment From: yossigo

@yakhyadabo the errors suggest that Redis accepts a clear-text (non TLS) connection, not a certificate check issue.

Comment From: yakhyadabo

@yossigo, Here's what my setup looks like : redis.conf

cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
appendonly yes
protected-mode no
tls-ca-cert-file /etc/tls/redis/ca.pem
tls-key-file /etc/tls/redis/redis-key.pem
tls-cert-file /etc/tls/redis/redis.pem
tls-replication yes
tls-cluster yes
port 0
tls-port 6379
tls-auth-clients no
tls-protocols "TLSv1.2 TLSv1.3"

Starting the server 

redis-server /etc/redis/redis.conf

Creating the cluster 

redis-cli --cluster create --cluster-replicas 1 host1:6379 host2:6379 host3:6379 host4:6379 host5:6379 host6:6379

Do you know what can be wrong here ?

Comment From: yossigo

@yakhyadabo Looks good to me. If your cluster forms properly and the Kubernetes probes are successful, those messages probably indicate you have Redis clients that are not configured to use TLS. You should either configure your clients to use TLS or set up a non-TLS port for them to connect to.

Comment From: scottc-git

@yakhyadabo I had the same error but the error went away after configuring my helm chart to use tls-auth-clients = true

$ kubectl exec -it bcr-redis-master-0 -n bcr-admin -- bash
Defaulting container name to redis.
Use 'kubectl describe pod/bcr-redis-master-0 -n bcr-admin' to see all of the containers in this pod.
I have no name!@bcr-redis-master-0:/$ env | grep -i tls
REDIS_TLS_CERT_FILE=/opt/bitnami/redis/certs/tls.crt
REDIS_TLS_AUTH_CLIENTS=yes
REDIS_TLS_ENABLED=yes
REDIS_TLS_CA_FILE=/opt/bitnami/redis/certs/ca.crt
REDIS_TLS_KEY_FILE=/opt/bitnami/redis/certs/tls.key
REDIS_TLS_PORT=6379

Comment From: yakhyadabo

@scottc-git I set tls-auth-clients value to yes in the config file. I still receive the same message in the logs.

Comment From: haljin

I have the same problem with, what I assume, is the same helmchart (bitnami/redis). Everything seems to be working fine, until a bit more load is put on Redis. I have disabled those healthchecks, logged onto the pod and tried to manually do

redis-cli -a $REDIS_PASSWORD $no_auth_warning -h localhost -p $REDIS_TLS_PORT --tls --cacert /opt/bitnami/redis/certs/root-ca --cert /opt/bitnami/redis/certs/redis-crt --key /opt/bitnami/redis/certs/redis-key ping

The command freezes. Redis logs 1:M 27 Sep 2020 21:03:27.689 # Error accepting a client connection: error:1408F10B:SSL routines:ssl3_get_record:wrong version number (conn: fd=13)

What's more I think my replicas lost sync to master temporarily when that happened too:

1:S 27 Sep 2020 21:05:08.418 # Connection with master lost.
1:S 27 Sep 2020 21:05:08.419 * Caching the disconnected master state.
1:S 27 Sep 2020 21:05:08.419 * REPLICAOF 192.168.39.4:6379 enabled (user request from 'id=8 addr=192.168.5.34:42355 fd=9 name=sentinel-df3c5b0a-cmd age=638 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=353 qbuf-free=32415 obl=45 oll=0 omem=0 events=r cmd=exec user=default')
1:S 27 Sep 2020 21:05:08.420 # CONFIG REWRITE executed with success.
1:S 27 Sep 2020 21:05:08.431 * Connecting to MASTER 192.168.39.4:6379
1:S 27 Sep 2020 21:05:08.432 * MASTER <-> REPLICA sync started
1:S 27 Sep 2020 21:05:08.438 * Non blocking connect for SYNC fired the event.
1:S 27 Sep 2020 21:05:08.439 * Master replied to PING, replication can continue...
1:S 27 Sep 2020 21:05:08.440 * Trying a partial resynchronization (request 90fc716f451f0e6970d7659602558398b8626f49:20539703).
1:S 27 Sep 2020 21:05:08.440 * Successful partial resynchronization with master.
1:S 27 Sep 2020 21:05:08.440 # Master replication ID changed to 673f6cd782dfea87b3b1b6e7510d6a1ae2235d0e

The chart is 

name: redis
    repository: https://charts.bitnami.com/bitnami
    version: 11.0.2
    enabled: true

configured with

  tls:
    enabled: true
    authClients: true
    certFilename: redis-crt
    certKeyFilename: redis-key
    certCAFilename: root-ca

And Redis is bitnami/redis:6.0.8

Comment From: yossigo

@haljin Thanks for your report. I'm not familiar with the details of the bitnami chart so it's hard for me to tell if that could possibly be a configuration issue with it, or a problem in Redis itself which gets to reproduce only in those specific conditions.

If I interpret your report correctly, your redis-server basically got into some hung state where you're unable to establish a TLS connection with it. Later the replica drops and manages to reconnect successfully. Are you also able to connect with redis-cli at this point?

It will be useful to get more information about the state of redis-server right before that happens -- INFO and CLIENT LIST in particular.

Comment From: haljin

@yossigo it seems the real problem has been https://github.com/RedisGraph/RedisGraph/issues/1357 - it had a side effect on locking up client connections in a very strange way.

Comment From: yossigo

@haljin Thanks for updating. I'm closing this issue as it doesn't seem to relate to a problem in redis-server itself.