Describe the bug
OpenSSL 3 clients are more sensitive to how connections are closed by servers.
As far as I understand it, if there is an unexpected EOF from the server, because the connection was closed without SSL_shutdown being called to perform a bidirectional close (which theoretically allows a truncation attack), clients compiled against OpenSSL 3 will now throw an error.
There are details at https://github.com/openssl/openssl/issues/11378 and https://github.com/openssl/openssl/issues/11381
This mainly affects Ruby's redis Gem (https://github.com/redis/redis-rb/issues/1106, https://github.com/mperham/sidekiq/issues/5402, etc), as Python and PHP have a corresponding "keep allowing unexpected EOFs" flags set in their native OpenSSL functionalities (e.g. https://github.com/php/php-src/issues/8369)
This happens when e.g. the idle timeout in Redis is set to a non-zero value; once connections are terminated due to the idle timeout, a reconnect doesn't work, as OpenSSL 3 on the client now handles the associated unexpected EOF as an error.
To reproduce
The following Dockerfile reproduces the issue. Reproduction instructions are inline; basically, after the build, in a docker run, redis-server is started, and a Ruby client is connected in an irb session; then, in a second terminal window, redis-cli in a docker exec is used to kill that connection (a CLIENT KILL has the same effect as letting the connection hit the idle timeout), and back in the docker run session, another get() call doesn't gracefully reconnect like with OpenSSL 1, but throw the "unexpected EOF" error.
# syntax=docker/dockerfile:1.3-labs
FROM ubuntu:22.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get install -y lsb-release
# The default sources list minus backports, restricted and multiverse.
RUN cat <<EOF >/etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs) main universe
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-security main universe
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-updates main universe
EOF
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y --no-install-recommends \
apt-transport-https \
apt-utils \
autoconf \
automake \
build-essential \
bzip2 \
cmake \
coreutils \
curl \
gcc \
git \
gnupg \
less \
libc6-dev \
libjemalloc-dev \
libssl-dev \
lsb-release \
make \
patch \
pkg-config \
python-is-python3 \
python3 \
rename \
ruby \
ruby-dev \
tar \
unzip \
wget \
xz-utils \
zip
RUN curl https://download.redis.io/redis-stable.tar.gz | tar xz
RUN cd redis-stable && make BUILD_TLS=yes && make install && ./utils/gen-test-certs.sh
# 4.7.1 contains a workaround, so we install 4.7.0 to demonstrate the problem
RUN gem install redis --version 4.7.0
# terminal 1 % docker build --tag redis-openssl3-repro .
# terminal 1 % docker run --rm -ti --name redis-openssl3-repro-run redis-openssl3-repro bash
# terminal 1 [in docker run] root@dbfebf2025eb:/# redis-server --tls-port 6379 --port 0 --tls-cert-file redis-stable/tests/tls/redis.crt --tls-key-file redis-stable/tests/tls/redis.key --tls-ca-cert-file redis-stable/tests/tls/ca.crt --tls-auth-clients no &
# terminal 1 [in docker run] root@dbfebf2025eb:/# irb
# terminal 1 [in docker run] irb(main):001:0> require "redis"
# terminal 1 [in docker run] => true
# terminal 1 [in docker run] irb(main):002:0> r = Redis.new(url: "rediss://127.0.0.1:6379", ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })
# terminal 1 [in docker run] => #<Redis client v4.7.0 for rediss://127.0.0.1:6379/0>
# terminal 1 [in docker run] irb(main):003:0> r.get("foo")
# terminal 1 [in docker run] => nil
# terminal 2 % docker exec -ti redis-openssl3-repro-run redis-cli --tls --cert redis-stable/tests/tls/redis.crt --key redis-stable/tests/tls/redis.key --cacert redis-stable/tests/tls/ca.crt
# terminal 2 [in docker exec] 127.0.0.1:6379> CLIENT LIST
# terminal 2 [in docker exec] id=3 addr=127.0.0.1:57978 laddr=127.0.0.1:6379 fd=8 name= age=47 idle=47 …
# terminal 2 [in docker exec] id=5 addr=127.0.0.1:57982 laddr=127.0.0.1:6379 fd=9 name= age=15 idle=0 …
# terminal 2 [in docker exec] 127.0.0.1:6379> CLIENT KILL ID 3
# terminal 2 [in docker exec] (integer) 1
# terminal 1 [in docker run] irb(main):004:0> r.get("foo")
# terminal 1 [in docker run] /usr/lib/ruby/3.0.0/openssl/buffering.rb:214:in `sysread_nonblock': SSL_read: unexpected eof while reading (OpenSSL::SSL::SSLError)
# terminal 1 [in docker run] from /usr/lib/ruby/3.0.0/openssl/buffering.rb:214:in `read_nonblock'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/connection/ruby.rb:55:in `block in _read_from_socket'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/connection/ruby.rb:54:in `loop'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/connection/ruby.rb:54:in `_read_from_socket'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/connection/ruby.rb:47:in `gets'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/connection/ruby.rb:382:in `read'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:311:in `block in read'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:299:in `io'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:310:in `read'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:161:in `block in call'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:279:in `block (2 levels) in process'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:420:in `ensure_connected'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:269:in `block in process'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:356:in `logging'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:268:in `process'
# terminal 1 [in docker run] from /var/lib/gems/3.0.0/gems/redis-4.7.0/lib/redis/client.rb:161:in `call'
# terminal 1 [in docker run] ... 8 levels...
You can use FROM ubuntu:20.04, which will use a Ruby built against OpenSSL 1.1.1, to see how older clients are less strict and this error does not occur.
Comment From: yossigo
@dzuelke Thanks for reporting this, I've been looking at different test behavior due to OpenSSL 3.0 but I realize now the implication might be greater than just that. I'll take a look - I'm not sure we'll always be able to afford waiting for SSL_shutdown() to complete successfully, but even a best-effort approach is better than what we have now.
Comment From: yossigo
@dzuelke Can you please confirm that #10931 solves this problem in your case?
Comment From: dzuelke
@yossigo Just built with that patch applied and can confirm it fixes the issue! Nice work!
Comment From: shashwatatw
I'm facing a similar issue while using python 3.10 from a linux device, what is the possible issue. unable fire a single get request. while it works fine with python 2.7