Redis [NEW] Add functionality accept ACME certificates for TLS

The problem/use-case that the feature addresses

Redis TLS cannot currently be used with ACME certificates

Description of the feature

A flag to pass in with TLS config to indicate a ACME certificate will be passed - in this scenario, the ca.crt is not needed as these certificates pass the full cert chain within the tls.crt. Scripts can then handle obtaining the necessary certificates from the cert chain to enable TLS.

Alternatives you've considered

Could be change downstream (at cert-manager for our install) to separate the ca.crt and include it with the certificate secret - cert-manager has made it clear they do not plan on implementing this.

We could manually break out this somewhere on our cluster & place within a new secret prior to passing to redis (rough logic). However, we believe that redis should follow suit with other applications to natively accept these ACME based certificates.

Comment From: zuiderkwast

Hello.

Scripts can then handle obtaining the necessary certificates from the cert chain to enable TLS.

What exactly would Redis need to do? Can we just allow CA certs to be absent and will then OpenSSL take them automatically from the cert file?

If you have a patch and it doesn't add a lot of complexity to redis, I think it's easier to get it in.

Comment From: slavak

@JDKnobloch Do you mean Redis would obtain the tls-ca-cert-file by extracting the intermediate certificate from a provided tls-cert-file?

Comment From: JDKnobloch

Sorry - forgot to return to this issue

@slavak precisely - our certs are provided to us fully within a single tls-cert-file via a cert chain. So we have the CA cert available, just not as cleanly as a standalone tls-ca-cert-file & it introduces an issue of extraction that we would prefer not to handle manually for this one application.

I would think could be a somewhat common issue for users implementing TLS as our ACME / lets-encrypt / cert-manager stack is fairly standard (at least not completely outlandish), so would be nice to have & could potentially mitigate future users winding up in a similar situation.

Comment From: slavak

I'm willing to give implementing this a shot, but I'd need your help, since I don't have any real certificates of this type to test with and couldn't find any spec that describes exactly how this works.

IIUC, it's basically a PEM file that contains a certificate bundle, from the leaf certificate up to a signing CA, and the algorithm would be to load this bundle and look for the first certificate in it that's a CA?

I wrote a small snippet of code that does this. Anyone more familiar with the OpenSSL API and/or ACME certificates that can tell me if this looks right?

````c X509 *ca_cert = NULL;

X509_STORE *ctx = X509_STORE_new(); assert(X509_STORE_load_locations(ctx, "/tmp/acme.pem", NULL));

STACK_OF(X509_OBJECT) sk = X509_STORE_get0_objects(ctx); for (int i = 0; i < sk_X509_OBJECT_num(sk); i++) { X509 c = X509_OBJECT_get0_X509(sk_X509_OBJECT_value(sk, i));; if (!c) continue; if (X509_check_ca(c)) { ca_cert = X509_dup(c); break; } }

X509_STORE_free(ctx);

Comment From: yossigo

@slavak @JDKnobloch Wouldn't we get the same effect if we point both tls-cert-file and tls-ca-cert-file to the same file that contains the whole chain?

This configuration might have undesired side effects, such as being overly permissive and accepting client certificates issued by any intermediate cert - but that would be the case also if we follow the original ask above.

Comment From: slavak

That's... a good question.

Comment From: JDKnobloch

@yossigo double checked git history to confirm and I indeed attempted this without success. Cannot remember the exact outcome but suffice to say it was unsuccessful - I believe pods failed to start.

@slavak that is correct as far as I understand. I can't find great documentation on the finer details on ACME certificate contents, but per cert-manager docs this is our current position:

When a certificate is issued by an intermediate CA and the Issuer can provide the issued certificate's chain, the contents of tls.crt will be the requested certificate followed by the certificate chain. Additionally, if the Certificate Authority is known, the corresponding CA certificate will be stored in the secret with key ca.crt. For example, with the ACME issuer, the CA is not known and ca.crt will not exist in acme-crt-secret.

We have the tls.crt containing the full chain as a PEM signed cert chain, but CA is not known

Comment From: yossigo

@JDKnobloch Can you provide sample PEM files to demonstrate / test this?

Comment From: JDKnobloch

@yossigo Sure - are you just looking for the general layout? Obviously I don't want to go pasting a cert chain that is in use around but I can provide that with dummy values if that suffices

Comment From: slavak

@JDKnobloch I think any kind of sample files that would allow us to reproduce the issue would be helpful. A (set of) certificate/private key file(s) that was generated for this purpose alone would work fine.

Comment From: nebed

@yossigo @slavak here is a dummy cert generated from the letsencrypt staging endpoint

there are 3 certificate fields and according to them the middle certificate is the intermediate CA certificate

Comment From: garry-t

this ticket will solve big pain in ass and will give ability to use letsencrypt out of the box. isn't it?

Comment From: zuiderkwast

The certificate chain we get with the ACME protocol is described in the RFC under 7.4.2. Downloading the Certificate. What you get is of MIME-type application/pem-certificate-chain, which is describe like this:

A file of this type contains one or more certificates encoded with the PEM textual encoding

(...)

In order to provide easy interoperation with TLS, the first certificate MUST be an end-entity certificate. Each following certificate SHOULD directly certify the one preceding it. Because certificate validation requires that trust anchors be distributed independently, a certificate that represents a trust anchor MAY be omitted from the chain, provided that supported peers are known to possess any omitted certificates.

In tls.c, we use SSL_CTX_use_certificate_chain_file() to load the cert file, so I would suppose we already accept a cert chain in the cert file. The OpenSSL manual describes the function like this:

SSL_CTX_use_certificate_chain_file() loads a certificate chain from file into ctx. The certificates must be in PEM format and must be sorted starting with the subject's certificate (actual client or server certificate), followed by intermediate CA certificates if applicable, and ending at the highest level (root) CA. SSL_use_certificate_chain_file() is similar except it loads the certificate chain into ssl.

@nebed's certificate chain (posted above) contains these certificates:

Common Name: test.nebed.io \ Issuer: (STAGING) Artificial Apricot R3, (STAGING) Let's Encrypt
Common Name: (STAGING) Artificial Apricot R3 \ Issuer: (STAGING) Pretend Pear X1, (STAGING) Internet Security Research Group
Common Name: (STAGING) Pretend Pear X1 \ Issuer: (STAGING) Doctored Durian Root CA X3, (STAGING) Internet Security Research Group

Apart from the "STAGING" marks, it looks like Let's Encrypt's certificate hierarchy which is

DST Root CA X3 ==> ISRG X1 ==> R3 ==> Subscriber certificates

The root certificate is not included in the cert chain, so I guess you need to provide it in your CA certificates. In my system (Debian) the trusted CA certificates are installed under /usr/share/ca-certificates/ and I found the DST Root CA X3 in the file /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt.

So my conclusion is that you should be able to put the cert chain in the tls-cert-file and put your systems's ca certificates directory in tls-ca-cert-dir.

Has anyone tried that?

Comment From: nebed

you're a genius @zuiderkwast ! that works!

$ redis-cli -h redis-qa-master.k8s-dev.xxxxxx.xxxxxxx.com -p 6380 --tls -a "xxxxxxxxx"
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
redis-qa-master.k8s-dev.xxxxx.xxxxxx:6380> HELLO
 1) "server"
 2) "redis"
 3) "version"
 4) "7.2.3"
 5) "proto"
 6) (integer) 2
 7) "id"
 8) (integer) 382
 9) "mode"
10) "standalone"
11) "role"
12) "master"
13) "modules"
14) (empty array)

let me see if I can create a PR to the bitnami/redis chart to support this

Comment From: nebed

created an issue on the bitnami/redis container to add a new environment variable for this config in redis.conf - https://github.com/bitnami/containers/issues/56361 would make the chart update easier.

Comment From: zuiderkwast

It's great that it works! So we don't need any new implementation in redis then, right?

If you can't set CA dir, maybe you can set CA file to the file containing "DST Root CA X3"?

Comment From: nebed

Yes, it works perfectly, I don't think there's anything to be done from Redis' side. I can currently set the CA dir in redis.conf, it would just be convenient from the bitnami/redis container to have it exposed as an environment variable. Thanks a lot for the help @zuiderkwast.

Comment From: zuiderkwast

Great. Maybe we could just improve documentation about this.

Comment From: garry-t

@zuiderkwast Dosent work for me . I use certificates signed by sectigo. tls-ca-cert-dir /usr/share/ca-certificates/mozilla/

Error condition on socket for SYNC: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

Comment From: zuiderkwast

@garry-t have you checked which certificates you have in the cert chain and that the bottom of those is signed by one of the certs in your ca dir? I pasted the pem contents in an online form like https://www.sslchecker.com/certdecoder or https://www.sslshopper.com/certificate-decoder.html

Comment From: garry-t

@zuiderkwast I found what CA certs used for my certificates here. After that, I found another directory with ssl certs and use it in config, it started work. Changed from: tls-ca-cert-dir=/usr/share/ca-certificates/mozilla/ To tls-ca-cert-dir=/etc/ssl/certs/ After that certificates was able to be validated. For now not sure what is the difference between two folders, but for Ubuntu 20.04 this path works : /etc/ssl/certs/. -> This is openssl compatible directory. see In both folders I have same certificate COMODO_ECC_Certification_Authority, but with different extensions

openssl x509 -enddate -noout -in /usr/share/ca-certificates/mozilla/COMODO_RSA_Certification_Authority.crt 
notAfter=Jan 18 23:59:59 2038 GMT

openssl x509 -enddate -noout -in /etc/ssl/certs/COMODO_RSA_Certification_Authority.pem 
notAfter=Jan 18 23:59:59 2038 GMT

Comment From: zuiderkwast

Interesting. Yes, the CA cert store location seems to be different on every OS.

I found a list using the source code of an application that can find and use system's CA certs automatically. This is where it would look for a trusted CA cert bundle:

Linux: * "/etc/ssl/certs/ca-certificates.crt" (Debian, Ubuntu, Gentoo) * "/etc/pki/tls/certs/ca-bundle.crt" (Fedora, RHEL 6, Amazon Linux) * "/etc/ssl/ca-bundle.pem" (OpenSUSE) * "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem" (CentOS, RHEL 7) * "/etc/ssl/cert.pem" (Alpine Linux)

FreeBSD/OpenBSD/NetBSD:

"/etc/ssl/cert.pem"
"/etc/openssl/certs/cacert.pem" (netbsd (if installed))
"/etc/openssl/certs/ca-certificates.crt"
"/usr/local/share/certs/ca-root-nss.crt"

SunOS: * "/etc/certs/CA/" (Oracle Solaris, some older illumos distros) * "/etc/ssl/cacert.pem" (OmniOS)

MacOS doesn't seem to have the CA certs in the file system, but IIUC, OpenSSL can use MacOS key store API to get them, if it's compiled with that.

Comment From: zuiderkwast

If you have curl installed, curl-config --ca can tell you where the system CA bundle is located. I believe it's usually a good idea to use the system's CA certs (except if you're using your own self-signed CA certificates for clients and servers within your own network).