I have tried this on a system that had Redis 5, and on a fresh install. Both fail when we install or update to Redis 6.0.3. The redis-sentinel service times out and fails to start.

redis-sentinel.service - Redis Sentinel Loaded: loaded (/usr/lib/systemd/system/redis-sentinel.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/redis-sentinel.service.d └─limit.conf Active: failed (Result: timeout) since Wed 2020-05-20 00:07:58 +03; 10s ago Process: 10126 ExecStart=/usr/bin/redis-sentinel /etc/redis-sentinel.conf --daemonize no --supervised systemd (code=exited, status=0/SUCCESS) Main PID: 8313 (code=killed, signal=KILL)

May 20 00:07:47 redis-vm-1 systemd[1]: Starting Redis Sentinel... May 20 00:07:58 redis-vm-1 systemd[1]: redis-sentinel.service start operation timed out. Terminating. May 20 00:07:58 redis-vm-1 systemd[1]: Failed to start Redis Sentinel. May 20 00:07:58 redis-vm-1 systemd[1]: Unit redis-sentinel.service entered failed state. May 20 00:07:58 redis-vm-1 systemd[1]: redis-sentinel.service failed.

It was working perfectly fine with redis-sentinel 5. We tried turning off the firewall, turning off selinux. Even removing all nodes, but still the same error.

Here's our config:

bind 10.10.10.1 port 26379 daemonize no logfile "/var/log/redis/sentinel.log" dir "/"

Generated by CONFIG REWRITE

protected-mode no supervised systemd sentinel myid b5d79022cfc1cd3788311692655f8daa45bd15f5 sentinel current-epoch 0

This is the log:

10251:X 20 May 2020 00:15:22.399 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 10251:X 20 May 2020 00:15:22.399 # Redis version=6.0.3, bits=64, commit=00000000, modified=0, pid=10251, just started 10251:X 20 May 2020 00:15:22.399 # Configuration loaded 10251:X 20 May 2020 00:15:22.399 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit. 10251:X 20 May 2020 00:15:22.402 * Running mode=sentinel, port=26379. 10251:X 20 May 2020 00:15:22.404 # Sentinel ID is b5d79022cfc1cd3788311692655f8daa45bd15f5 10251:signal-handler (1589922932) Received SIGTERM scheduling shutdown... 10251:X 20 May 2020 00:15:32.416 # User requested shutdown... 10251:X 20 May 2020 00:15:32.416 # Sentinel is now ready to exit, bye bye...

And this is the service:

[Unit] Description=Redis Sentinel After=network.target

[Service] ExecStart=/usr/bin/redis-sentinel /etc/redis-sentinel.conf --daemonize no --supervised systemd ExecStop=/usr/libexec/redis-shutdown redis-sentinel Type=notify User=redis Group=redis RuntimeDirectory=redis RuntimeDirectoryMode=0755 TimeoutStartSec=100 TimeoutStopSec=100

[Install] WantedBy=multi-user.target

Running sentinel from the command line instead of as a service works normally with no errors. Also the redis service is running normally, only the redis-sentinel service is causing this issue.

Comment From: dekonnection

Try to install libsystemd-dev before re-building Redis: it seems that since 6.0, Redis implements systemd notifies with this library. If not detected at build time, Redis is built without systemd support. I encountered the same issue on Ubuntu and Debian, installing libsystemd-dev and rebuilding redis fixed it.

See: https://github.com/antirez/redis/commit/129d14e1431e913426485526663e1a9aac67838c

Comment From: jkimalane

I can confirm that this issue exists on CentoOS 7 with Redis 6.x.x and explicitly manifest itself while trying to start redis-sentinel with systemctl. It seems that READY=1 signal is never sent to systemd and systemd just times out while waiting for it.

An ugly hack to work around this issue is to override the service type for redis-sentinel :

[Service]
Type=
Type=simple

At least after those changes I was able to control sentinel with systemctl, but for sure, it's just a workaround

Comment From: PatrickGhosn

Try to install libsystemd-dev before re-building Redis: it seems that since 6.0, Redis implements systemd notifies with this library. If not detected at build time, Redis is built without systemd support. I encountered the same issue on Ubuntu and Debian, installing libsystemd-dev and rebuilding redis fixed it.

See: 129d14e I'm installing it from the Remi repo, and not building it myself.

I can confirm that this issue exists on CentoOS 7 with Redis 6.x.x and explicitly manifest itself while trying to start redis-sentinel with systemctl. It seems that READY=1 signal is never sent to systemd and systemd just times out while waiting for it.

An ugly hack to work around this issue is to override the service type for redis-sentinel :

[Service] Type= Type=simple

At least after those changes I was able to control sentinel with systemctl, but for sure, it's just a workaround

I'll try this out. Personally I've run it as a background process for the moment.

Comment From: dirkd

The sentinel branch for starting up in server.c is missing systemd communication at

https://github.com/antirez/redis/blob/c38fd1f66152109a2d2801b1d181cdc1901d6b3b/src/server.c#L5164-L5167

This was working in 5.0 since the readiness signal was sent unconditionally before the branch by the calling redisSupervisedSystemd() (via redisIsSupervised()). This regression was introduced in 641c64ada10404356fc76c0b56a69b32c76f253c

So even when redis is built with libsystemd support redis-sentinel will currently never send "READY=1". #7168 should fix that when it's merged.

Comment From: vaibhaw-

Please target this to the next release (6.0.4?). This is pretty serious for systems that are managed through sentinels - sentinel processes are killed by systemd while waiting for notifications so there's no one around to manage the HA setup.

I have been trying to figure what went wrong for a few hours before realizing that it isn't the systemd config file that is at fault but that the sd_notify(..., "READY=1\n") is no longer being sent to systemd.

If this is the intended behavior going forward, at least document this departure from previous behavior somewhere.

Comment From: PatrickGhosn

This is fixed in 6.0.4. Thank you.

Comment From: dirkd

Please note that this hasn't been fixed in 6.0.4 upstream. @remicollet just cherry-picked the related patch into his builds (https://git.remirepo.net/cgit/rpms/redis.git/commit/?id=d0ecd1252c3703bcdcae4453f648c935b62ed957).

So this issue should probably be reopened since the regression is still in place.

Comment From: dlosku

I can also confirm that this issue is not fixed in the 6.0.5 upstream either.

Comment From: yoonghm

I have similar timeout issue with redis-server but I got the solution from https://github.com/redis/redis/issues/7217. I compiled redis with the following command:

make USE_SYSTEMD=yes MALLOC=jemalloc V=1 PREFIX=/home/s BUILD_TLS=yes install

I am using redis 6.0.5.

Comment From: oranagra

https://github.com/redis/redis/pull/7168 merged. let me know if there are any other issues in that area. thanks

Comment From: willzhang

same problems at redis 6.0.9 on centos 7.8

this filx :https://serverfault.com/questions/1031891/redis-service-hangs-with-systemctl-and-type-notify

yum install systemd-devel
make BUILD_WITH_SYSTEMD=yes USE_SYSTEMD=yes

and redis@.service

cat > /etc/systemd/system/redis@.service << EOF
[Unit]
Description=Redis persistent key-value database
After=network.target

[Service]
User=redis
Group=redis
Type=notify
ExecStart=/usr/local/bin/redis-server /etc/redis/%i.conf --daemonize no --supervised systemd
ExecStop=/usr/local/bin/redis-cli -p %i shutdown
LimitNOFILE=10240
TimeoutStartSec=90s
TimeoutStopSec=90s
Restart=always

[Install]
WantedBy=multi-user.target
EOF

service status

[root@redis01 lib]# systemctl status redis@7000.service
● redis@7000.service - Redis persistent key-value database
   Loaded: loaded (/etc/systemd/system/redis@.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-11-08 15:04:28 CST; 22min ago
 Main PID: 23767 (redis-server)
   Status: "Ready to accept connections"
   CGroup: /system.slice/system-redis.slice/redis@7000.service
           └─23767 /usr/local/bin/redis-server 127.0.0.1:7000 [cluster]

Nov 08 15:04:28 redis01 systemd[1]: Starting Redis persistent key-value database...
Nov 08 15:04:28 redis01 systemd[1]: Started Redis persistent key-value database.
[root@redis01 lib]# 

Comment From: yossigo

@willzhang I'm not sure I understand your comment, are you reporting a successful fix or a reproduction of the problem?