Describe the bug
Since Redis got upgraded to Redis 6.2.5, it stopped allowing existing connections using any of the previous configurations that either used AUTH or not. Supposedly the new ACLs are backwards compatible according to the documentation to prevent downtime until new ACL accounts are working correctly but that is not the case for my CentOS 7 VPS running Virtuozzo .
I have now set all default users to nopass, and I can log successfully into each of my 8 instances (1 for each separate Vhost's Wordpress or Community Forum) via SSH, but none of my existing websites or forums or Netdata can connect to my Redis instances anymore, they always get connection denied from the respective Unix sockets.
To reproduce
Using CentOS 7, Virtuozzo container, PHP 7.4, Apache 2.4, latest CPanel WHM. MariaDB 10.3 PHP Redis Extension v5.3.4.
Upgrade to Redis 6.2.5 and all existing Unix Socket connections reject existing connections, whether using a previously configured AUTH pw or not. Only command line seems to work.
Via SSH command line:
```` redis-cli -s /var/lib/redis/redis.sock redis /var/lib/redis/redis.sock> 1) "user default on nopass sanitize-payload ~ & +@all" 2) "user test on sanitize-payload #my pass hash ~ & +@all" 3) "user netdata on sanitize-payload #my pass hash ~ & +@all" redis /var/lib/redis/redis.sock> acl getuser default 1) "flags" 2) 1) "on" 2) "allkeys" 3) "allchannels" 4) "allcommands" 5) "nopass" 6) "sanitize-payload" 3) "passwords" 4) (empty array) 5) "commands" 6) "+@all" 7) "keys" 8) 1) "" 9) "channels" 10) 1) "" redis /var/lib/redis/redis.sock>
root@vps123456 [/var/lib/redis]# ll total 259608 drwxr-x--- 2 redis redis 4096 Jul 25 21:13 ./ drwxr-xr-x 36 root root 4096 Jul 17 07:49 ../ -rw-r--r-- 1 redis redis 14422385 Jul 25 04:26 appendonly2.aof -rw-r--r-- 1 redis redis 52278753 Jul 24 09:18 appendonly3.aof -rw-r--r-- 1 redis redis 40298789 Jul 25 21:39 appendonly4.aof -rw-r--r-- 1 redis redis 64918089 Jul 25 06:58 appendonly5.aof -rw-r--r-- 1 redis redis 34668449 Jul 24 20:12 appendonly6.aof -rw-r--r-- 1 redis redis 50239425 Jul 24 20:47 appendonly7.aof -rw-r--r-- 1 redis redis 0 Jun 27 21:29 appendonly8.aof -rw-r--r-- 1 redis redis 705 Jul 19 04:16 appendonly.aof -rw-r--r-- 1 redis redis 6559162 Jul 25 21:09 dump2.rdb -rw-r--r-- 1 redis redis 307829 Jul 25 21:09 dump3.rdb -rw-r--r-- 1 redis redis 971139 Jul 25 21:09 dump4.rdb -rw-r--r-- 1 redis redis 143723 Jul 25 21:09 dump5.rdb -rw-r--r-- 1 redis redis 581821 Jul 25 21:13 dump6.rdb -rw-r--r-- 1 redis redis 374226 Jul 25 21:13 dump7.rdb -rw-r--r-- 1 redis redis 92 Jul 25 21:07 dump8.rdb -rw-r--r-- 1 redis redis 165 Jul 25 18:14 dump.rdb srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis2.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis3.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis4.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis5.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis6.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis7.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis8.sock= srwxrwxrwx 1 redis redis 0 Jul 25 21:08 redis.sock= root@vps123456 [/var/lib/redis]#
root@vps123456 [/etc/redis]# ll total 780 drwxr-x--- 2 redis root 4096 Jul 25 20:58 ./ drwxr-xr-x 102 root root 20480 Jul 25 21:51 ../ -rw-r--r-- 1 redis redis 93950 Jul 25 20:57 redis2.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:42 redis3.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:41 redis4.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:41 redis5.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:40 redis6.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:38 redis7.conf -rw-r--r-- 1 redis redis 93950 Jul 25 20:39 redis8.conf -rw-r--r-- 1 redis redis 93947 Jul 25 20:57 redis.conf -rw-r----- 1 redis redis 13793 Jul 22 06:10 sentinel.conf -rw-r--r-- 1 redis redis 273 Jul 25 21:07 users.acl root@vps123456 [/etc/redis]#
less /var/log/redis/redis.log (Set to verbose mode)
32323:C 25 Jul 2021 20:56:02.364 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 32323:C 25 Jul 2021 20:56:02.364 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=32323, just started 32323:C 25 Jul 2021 20:56:02.364 # Configuration loaded 32323:M 25 Jul 2021 20:56:02.365 * monotonic clock: POSIX clock_gettime 32323:M 25 Jul 2021 20:56:02.365 * Running mode=standalone, port=6379. 32323:M 25 Jul 2021 20:56:02.365 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 32323:M 25 Jul 2021 20:56:02.365 # Server initialized 32323:M 25 Jul 2021 20:56:02.366 * DB loaded from append only file: 0.000 seconds 32323:M 25 Jul 2021 20:56:02.366 * Ready to accept connections 32323:M 25 Jul 2021 20:56:02.366 * The server is now ready to accept connections at /var/lib/redis/redis.sock 32323:M 25 Jul 2021 20:57:07.903 # CONFIG REWRITE executed with success. 32323:M 25 Jul 2021 21:07:41.185 # User requested shutdown... 32323:M 25 Jul 2021 21:07:41.185 * Calling fsync() on the AOF file. 32323:M 25 Jul 2021 21:07:41.186 * Saving the final RDB snapshot before exiting. 32323:M 25 Jul 2021 21:07:41.966 * DB saved on disk 32323:M 25 Jul 2021 21:07:41.966 * Removing the pid file. 32323:M 25 Jul 2021 21:07:41.966 * Removing the unix socket file. 32323:M 25 Jul 2021 21:07:41.966 # Redis is now ready to exit, bye bye... 1931:C 25 Jul 2021 21:08:07.299 * Supervised by systemd. Please make sure you set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit. 1931:C 25 Jul 2021 21:08:07.299 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 1931:C 25 Jul 2021 21:08:07.299 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1931, just started 1931:C 25 Jul 2021 21:08:07.299 # Configuration loaded 1931:M 25 Jul 2021 21:08:07.300 * monotonic clock: POSIX clock_gettime 1931:M 25 Jul 2021 21:08:07.301 * Running mode=standalone, port=6379. 1931:M 25 Jul 2021 21:08:07.301 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 1931:M 25 Jul 2021 21:08:07.301 # Server initialized 1931:M 25 Jul 2021 21:08:07.302 * DB loaded from append only file: 0.000 seconds 1931:M 25 Jul 2021 21:08:07.302 * Ready to accept connections 1931:M 25 Jul 2021 21:08:07.302 * The server is now ready to accept connections at /var/lib/redis/redis.sock
PHP Info:
redis
Redis Support => enabled Redis Version => 5.3.4 Redis Sentinel Version => 0.1 Available serializers => php, json
Directive => Local Value => Master Value redis.arrays.algorithm => no value => no value redis.arrays.auth => no value => no value redis.arrays.autorehash => 0 => 0 redis.arrays.connecttimeout => 0 => 0 redis.arrays.consistent => 0 => 0 redis.arrays.distributor => no value => no value redis.arrays.functions => no value => no value redis.arrays.hosts => no value => no value redis.arrays.index => 0 => 0 redis.arrays.lazyconnect => 0 => 0 redis.arrays.names => no value => no value redis.arrays.pconnect => 0 => 0 redis.arrays.previous => no value => no value redis.arrays.readtimeout => 0 => 0 redis.arrays.retryinterval => 0 => 0 redis.clusters.auth => no value => no value redis.clusters.cache_slots => 0 => 0 redis.clusters.persistent => 0 => 0 redis.clusters.read_timeout => 0 => 0 redis.clusters.seeds => no value => no value redis.clusters.timeout => 0 => 0 redis.pconnect.connection_limit => 0 => 0 redis.pconnect.echo_check_liveness => 1 => 1 redis.pconnect.pool_pattern => no value => no value redis.pconnect.pooling_enabled => 1 => 1 redis.session.lock_expire => 0 => 0 redis.session.lock_retries => 10 => 10 redis.session.lock_wait_time => 2000 => 2000 redis.session.locking_enabled => 0 => 0
**Expected behavior**
Redis v6.2 to still work upon upgrade as per documentation without any immediate code adjustments or downtime. Failing that it should work (allow Unix socket connections) after ensuring no password is set in redis.conf and confirmed via ACL GETUSER.
**Additional information**
Basic setup, running under SystemD.
Many thanks.
**Comment From: TheOldMan2000**
Managed to resolve this, it's not a bug. Somehow the CHMOD folder permissions for the folder containing the Unix sockets had changed following the upgrade to Redis 6.2.5, which may be a bug in itself.
Solution was to CHMOD the folder to read drwxr-x--x using
`CHMOD 0751 /var/lib/redis`
I wish I'd spotted this days ago, but hopefully it may help someone in the future.
**Comment From: oranagra**
@TheOldMan2000 I think i'm missing the information of which redis version you were upgrading from? i.e. is it 6.2.4 or an earlier version?
Also, please specify (maybe you already did), where did you get the redis binaries (some specific package / repo, or built it yourself)? maybe we can report the permission issues to the package maintainer.
**Comment From: TheOldMan2000**
Thanks, I wasn't 100% sure tbh, but just found this:
yum history pkg-info redis
Loaded plugins: fastestmirror, tsflags, universal-hooks
Transaction ID : 336
Begin time : Fri Jul 23 20:57:41 2021
Package : redis-6.2.4-1.el7.remi.x86_64
State : Updated
Size : 4,471,207
Build host : builder.remirepo.net
Build time : Wed Jun 2 07:17:19 2021
Packager : Remi Collet
Vendor : Remi's RPM repository https://rpms.remirepo.net/
License : BSD
URL : http://redis.io
Source RPM : redis-6.2.4-1.el7.remi.src.rpm
Commit Time : Wed Jun 2 13:00:00 2021
Committer : Remi Collet remi@remirepo.net
Reason : user
Command Line : --enablerepo=remi install redis
From repo : remi
Installed by : root
Transaction ID : 336
Begin time : Fri Jul 23 20:57:41 2021
Package : redis-6.2.5-1.el7.remi.x86_64
State : Update
Size : 4,475,191
Build host : builder.remirepo.net
Build time : Thu Jul 22 06:10:49 2021
Packager : Remi Collet
Vendor : Remi's RPM repository https://rpms.remirepo.net/
License : BSD
URL : http://redis.io
Source RPM : redis-6.2.5-1.el7.remi.src.rpm
Commit Time : Thu Jul 22 13:00:00 2021
Committer : Remi Collet remi@remirepo.net
Reason : user
Command Line : update --skip-broken
From repo : remi
Installed by : root
Transaction ID : 308
Begin time : Sat Jun 26 11:41:00 2021
Package : redis-6.2.4-1.el7.remi.x86_64
State : Install
Size : 4,471,207
Build host : builder.remirepo.net
Build time : Wed Jun 2 07:17:19 2021
Packager : Remi Collet
Vendor : Remi's RPM repository https://rpms.remirepo.net/
License : BSD
URL : http://redis.io
Source RPM : redis-6.2.4-1.el7.remi.src.rpm
Commit Time : Wed Jun 2 13:00:00 2021
Committer : Remi Collet remi@remirepo.net
Reason : user
Command Line : --enablerepo=remi install redis
From repo : remi
Installed by : root
````
Comment From: oranagra
@remicollet FYI, please see if there's any issue with socket dir permission on upgrade. i'm closing this one for now, since it's not related to this repo.
Comment From: remicollet
Sockets are used to be stored in /var/run/redis which has the proper permissions (755) and SELinux context (redis_var_run_t), not in /var/lib/redis which have other permissions (750) and other SElinux context (redis_var_lib_t).
And yes, on package update, permissions are restored to their default value, this is the expected bahavior.
Comment From: TheOldMan2000
Thanks both for the reply and info.
Problem is that /var/run/ has never worked reliably in terms of permissions on my Inmotion Hosting VPS (CENTOS 7.9 virtuozzo WHM v96.0.13), so I use /var/lib/ instead for Redis, Netdata etc because that is where the MySQL Unix Socket resides.
PS. Thanks @remicollet for your RPMs!