Redis Redis sentinel service couldn’t be started at Centos7.1

Hello,

I install a redis cluster service and version is 3.0.7. It has three masters, three slaves and one sentinel node. I can start the 3 masters and 3 slaves services, but I start sentinel service failure. If I run sentinel process by command, it can be run. I make same installation in centos7.2. I don’t meet the issue and all of services are running.

I get some error messages from log file, but I don’t understand its mean. Who could help me point the reason of failure?

Below is error mesages from /var/log/messages

Sep  5 07:35:35 localhost systemd: Starting Advanced key-value store...
Sep  5 07:35:35 localhost sentinel_26379[8331]: Redis 3.0.7 (00000000/0) 64 bit, sentinel mode, port 26379, pid 8331 ready to start.
Sep  5 07:35:35 localhost sentinel_26379[8331]: WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Sep  5 07:35:35 localhost sentinel_26379[8331]: Sentinel runid is 01e6398bfe2c8dfa56aad2eb2a074c467a3b8652
Sep  5 07:35:35 localhost sentinel_26379[8331]: Sentinel config file /etc/redis/sentinel_26379.conf is not writable: Read-only file system. Exiting...
Sep  5 07:35:35 localhost systemd: PID 8299 read from file /var/run/redis/sentinel_26379.pid does not exist.
Sep  5 07:35:35 localhost systemd: sentinel_26379.service never wrote its PID file. Failing.
Sep  5 07:35:35 localhost systemd: Failed to start Advanced key-value store.
Sep  5 07:35:35 localhost systemd: Unit sentinel_26379.service entered failed state.
Sep  5 07:35:35 localhost systemd: sentinel_26379.service holdoff time over, scheduling restart.

Below is sentinel service status

[root@localhost ~]# systemctl status sentinel_26379.service
sentinel_26379.service - Advanced key-value store
   Loaded: loaded (/etc/systemd/system/sentinel_26379.service; enabled)
   Active: failed (Result: start-limit) since Thu 2019-09-05 07:35:36 EDT; 18min ago
     Docs: http://redis.io/documentation,
           man:redis-sentinel(1)
  Process: 8358 ExecStart=/opt/redis/bin/redis-server /etc/redis/sentinel_26379.conf --sentinel (code=exited, status=0/SUCCESS)
 Main PID: 8359 (code=exited, status=1/FAILURE)

Sep 05 07:35:36 localhost.localdomain systemd[1]: Unit sentinel_26379.service entered failed state.
Sep 05 07:35:36 localhost.localdomain systemd[1]: sentinel_26379.service holdoff time over, scheduling restart.
Sep 05 07:35:36 localhost.localdomain systemd[1]: Stopping Advanced key-value store...
Sep 05 07:35:36 localhost.localdomain systemd[1]: Starting Advanced key-value store...
Sep 05 07:35:36 localhost.localdomain systemd[1]: sentinel_26379.service start request repeated too quickly, refusing to start.
Sep 05 07:35:36 localhost.localdomain systemd[1]: Failed to start Advanced key-value store.
Sep 05 07:35:36 localhost.localdomain systemd[1]: Unit sentinel_26379.service entered failed state.

If you need the more information, please let me know. Thanks you very much.

Best regards Ray

Comment From: tanshaolong

Who could please give me some update for the issue? Thank you

Comment From: laixintao

Your log is pretty clear:

Sep  5 07:35:35 localhost sentinel_26379[8331]: Sentinel config file /etc/redis/sentinel_26379.conf is not writable: Read-only file system. Exiting...

Comment From: tanshaolong

@laixintao Thanks for your response,

Actually, I change mode of the /etc/redis/sentinel_26379.conf to 777, I have same issue yet. I don't meet same issue for centos 7.2 with same installation steps.

Comment From: laixintao

Have you tried to install this in another centos 7.1?

Is selinux active?

Comment From: tanshaolong

I ensure I have disabled selinux and firewall.

I install my redis by ansible playbook(https://github.com/DavidWittman/ansible-redis). I try centos7.1 to 7.5 with same playbook. They are successful except centos 7.1.

I have installed redis more than 3 times for centos7.1, but all of tests are same error.

Also, the issue whether or not is related with the redis bug(https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799696)? Thank you.

Comment From: laixintao

Can you try this command in redis ( not sentinel) and see what happens?

>  config rewrite

And can you paste this:

$ systemctl --version

And your redis.service file for systemd

Comment From: tanshaolong

Sorry for lately response.

I had destroyed the redis environment for linux Centos 7.1. I rebuild some vm machines for reproduce the issue at yesterday, but I am failure. I just be failure to create redis master service and it is difference with my formerly test that be failure to create sentinel node. Please see below error message for redis master.

Jan  2 14:18:42 localhost redis-server: 8627:C 02 Jan 14:18:42.840 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Jan  2 14:18:42 localhost redis-server: 8627:C 02 Jan 14:18:42.840 # Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=8627, just started
Jan  2 14:18:42 localhost redis-server: 8627:C 02 Jan 14:18:42.840 # Configuration loaded
Jan  2 14:18:42 localhost redis-server: oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Jan  2 14:18:42 localhost redis-server: Redis version=4.0.6, bits=64, commit=00000000, modified=0, pid=8627, just started
Jan  2 14:18:42 localhost redis-server: Configuration loaded
Jan  2 14:18:42 localhost redis_6379[8628]: Can't open nodes.conf in order to acquire a lock: Read-only file system
Jan  2 14:18:42 localhost systemd: PID file /var/run/redis/6379.pid not readable (yet?) after start.
Jan  2 14:18:42 localhost systemd: redis_6379.service never wrote its PID file. Failing.
Jan  2 14:18:42 localhost systemd: Failed to start Advanced key-value store.
Jan  2 14:18:42 localhost systemd: Unit redis_6379.service entered failed state.
Jan  2 14:18:42 localhost systemd: redis_6379.service holdoff time over, scheduling restart.
Jan  2 14:18:42 localhost systemd: Stopping Advanced key-value store...
Jan  2 14:18:42 localhost systemd: Starting Advanced key-value store...

I check a redis master node, I don’t find “nodes.conf” and “/var/run/redis/6379.pid” files.

Also, I use same playbook to test for centos 7.4. It is successful. Currently, I retain the failure redis environment. If you need the more information. Please let me know. Thank you very much.

Comment From: laixintao

I think it's the same reason, please show me your $ systemctl --version and redis.service file, and can you show me you ansible playbook?

Comment From: tanshaolong

Please see below for detail.

# systemctl --version
systemd 208
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ

redis.service

# cat /etc/systemd/system/redis_6379.service
[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)

[Service]
Type=forking
ExecStart=/opt/redis/bin/redis-server /etc/redis/6379.conf
EnvironmentFile=-/etc/default/redis_6379
PIDFile=/var/run/redis/6379.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis


UMask=007
PrivateTmp=yes
LimitNOFILE=16384
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/redis/6379
ReadWriteDirectories=-/var/run/redis
CapabilityBoundingSet=~CAP_SYS_PTRACE

# redis-server writes its own config file when in cluster mode so we allow
# writing there (NB. ProtectSystem=true over ProtectSystem=full)
ProtectSystem=true
ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target

Also, I can directly start redis progress by running command "/opt/redis/bin/redis-server /etc/redis/6379.conf"

My ansible role reference https://github.com/DavidWittman/ansible-redis. Below is my playbook.

- name: configure redis server
  hosts: redis_master,redis_slave
  roles:
    - role: ansible-redis
      redis_version: 4.0.6
      yum_baseurl: "http://192.168.2.99/base_url/"
      cluster_install: True
      redis_assigned_slots: True
      redis_password: *******
      redis_tarball: /root/ansible/packages/redis/redis-4.0.6.tar.gz
      redis_syslog_facility: "USER"
      redis_stop_writes_on_bgsave_error: "yes"
      redis_min_slaves_max_lag: 10
      redis_sentinel_failover_timeout: 180000
      redis_sentinel_port: 26379
      redis_rdbcompression: "yes"
      redis_notify_keyspace_events: '""'
      redis_sentinel_quorum: 2
      redis_sentinel_down_after_milliseconds: 30000
      redis_timeout: 0
      redis_tcp_keepalive: 0
      redis_appendonly: "no"
      redis_tcp_backlog: 511
      redis_dir: "/var/lib/redis/6379"
      redis_daemonize: "yes"
      redis_slave_read_only: "yes"
      redis_slowlog_log_slower_than: 10000
      redis_loglevel: "notice"
      redis_auto_aof_rewrite_percentage: 100
      redis_slave_priority: 100
      redis_maxclients: 1000
      redis_no_appendfsync_on_rewrite: "no"
      redis_sentinel_notification_script: "false"
      redis_logfile: '""'
      redis_slowlog_max_len: 128
      redis_db_filename: "dump.rdb"
      redis_sentinel_bind: "0.0.0.0"
      redis_appendfsync: "everysec"
      redis_sentinel_logfile: '""'
      redis_bind: "0.0.0.0"
      redis_appendfilename: "appendonly.aof"
      redis_syslog_enabled: "yes"
      redis_make_32bit: "false"
      redis_maxmemory: "noeviction"
      redis_auto_aof_rewrite_min_size: "64mb"
      redis_sentinel_client_reconfig_script: "false"
      redis_sentinel_parallel_syncs: 1
      redis_port: 6379
      redis_rdbchecksum: "yes"
      redis_sentinel_dir: "/var/lib/redis/sentinel_26379"
      redis_databases: 16


- name: configure redis sentinel
  hosts: redis_sentinel
  vars:
    - redis_sentinel_settings:
        port: 6379
        quorum: 2
        auth_pass: *******
        down_after_milliseconds: 30000
        parallel_syncs: 1
        failover_timeout: 180000
        notification_script: false
        client_reconfig_script: false
  roles:
    - role: ansible-redis
      redis_version: 4.0.6
      # redis_sentinel: true
      redis_password: ********
      redis_tarball: /root/ansible/packages/redis/redis-4.0.6.tar.gz
      redis_syslog_facility: "USER"
      redis_stop_writes_on_bgsave_error: "yes"
      redis_min_slaves_max_lag: 10
      redis_sentinel_failover_timeout: 180000
      redis_sentinel_port: 26379
      redis_rdbcompression: "yes"
      redis_notify_keyspace_events: '""'
      redis_sentinel_quorum: 2
      redis_sentinel_down_after_milliseconds: 30000
      redis_timeout: 0
      redis_tcp_keepalive: 0
      redis_appendonly: "no"
      redis_tcp_backlog: 511
      redis_dir: "/var/lib/redis/6379"
      redis_daemonize: "yes"
      redis_slave_read_only: "yes"
      redis_slowlog_log_slower_than: 10000
      redis_loglevel: "notice"
      redis_auto_aof_rewrite_percentage: 100
      redis_slave_priority: 100
      redis_maxclients: 1000
      redis_no_appendfsync_on_rewrite: "no"
      redis_sentinel_notification_script: "false"
      redis_logfile: '""'
      redis_slowlog_max_len: 128
      redis_db_filename: "dump.rdb"
      redis_sentinel_bind: "0.0.0.0"
      redis_appendfsync: "everysec"
      redis_sentinel_logfile: '""'
      redis_bind: "0.0.0.0"
      redis_appendfilename: "appendonly.aof"
      redis_syslog_enabled: "yes"
      redis_make_32bit: "false"
      redis_maxmemory: "noeviction"
      redis_auto_aof_rewrite_min_size: "64mb"
      redis_sentinel_client_reconfig_script: "false"
      redis_sentinel_parallel_syncs: 1
      redis_port: 6379
      redis_rdbchecksum: "yes"
      redis_sentinel_dir: "/var/lib/redis/sentinel_26379"
      redis_databases: 16

Comment From: laixintao

Please also paste your /etc/redis/6379.conf and /etc/default/redis_6379

Omit your password

Comment From: laixintao

Note you already leaked your auth_pass on previous discuss, remember to change this.

Comment From: tanshaolong

The redis node don’t exist “/etc/default/redis_6379” file.

Below is for “/etc/redis/6379.conf”

[root@localhost ~]# cat /etc/redis/6379.conf
# Ansible managed

# General
daemonize yes
pidfile /var/run/redis/6379.pid
dir /var/lib/redis/6379
port 6379
bind 0.0.0.0
timeout 0
tcp-keepalive 0
tcp-backlog 511
loglevel notice
logfile ""
syslog-enabled yes
syslog-ident redis_6379
syslog-facility USER
databases 16

# Snapshotting
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb

# Replication
slave-serve-stale-data yes
slave-read-only yes
repl-disable-tcp-nodelay no
slave-priority 100
min-slaves-max-lag 10
masterauth *******
# Security
requirepass ******
# Limits
maxclients 1000
maxmemory noeviction
maxmemory-policy noeviction

# Append Only Mode
appendonly no
appendfilename appendonly.aof
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# Lua
lua-time-limit 5000

# Slow Log
slowlog-log-slower-than 10000
slowlog-max-len 128

# Event Notification
notify-keyspace-events ""
# Advanced
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
aof-rewrite-incremental-fsync yes

# Cluster
cluster-enabled yes

Comment From: tanshaolong

Note you already leaked your auth_pass on previous discuss, remember to change this.

Thank for your remind! :)

Comment From: laixintao

I think there are problems with this ansible playbook, please just use the default redis.conf until you know that are these parameters meaning.

cluster-enabled yes

if you don't need this, change it to no

daemonize yes

This shoule be set to not

Can you try with the basic and official systemd service file?

https://github.com/antirez/redis/blob/f4b81970601ee138af13f66d62ad10eabe56f41c/utils/systemd-redis_server.service

Comment From: tanshaolong

Ok, I will try your suggestion.

Comment From: tanshaolong

If I remove “cluster-enabled yes” and change “daemonize” to “no”, I can success to install a redis node and not cluster mode.

If I want to install a redis cluster, How should I configure the redis.conf for Linux7.1? Or, could you give me some doc for reference? Also, why doesn’t this issue happen at other Centos version with same configuration? Thank you very much.

Comment From: laixintao

The problem isn't your centos version I think, it's about systemd and ansible playbook.

Redis-cluser needs to write a node.conf file and systemd didn't let it.

Can you change to cluster-enabled yes and try with this systemd servie file?

# example systemd service unit file for redis-server
#
# In order to use this as a template for providing a redis service in your
# environment, _at the very least_ make sure to adapt the redis configuration
# file you intend to use as needed (make sure to set "supervised systemd"), and
# to set sane TimeoutStartSec and TimeoutStopSec property values in the unit's
# "[Service]" section to fit your needs.
#
# Some properties, such as User= and Group=, are highly desirable for virtually
# all deployments of redis, but cannot be provided in a manner that fits all
# expectable environments. Some of these properties have been commented out in
# this example service unit file, but you are highly encouraged to set them to
# fit your needs.
#
# Please refer to systemd.unit(5), systemd.service(5), and systemd.exec(5) for
# more information.

[Unit]
Description=Redis data structure server
Documentation=https://redis.io/documentation
#Before=your_application.service another_example_application.service
#AssertPathExists=/var/lib/redis

[Service]
ExecStart=/usr/local/bin/redis-server --supervised systemd --daemonize no
## Alternatively, have redis-server load a configuration file:
#ExecStart=/usr/local/bin/redis-server /path/to/your/redis.conf
LimitNOFILE=10032
NoNewPrivileges=yes
#OOMScoreAdjust=-900
#PrivateTmp=yes
Type=notify
TimeoutStartSec=infinity
TimeoutStopSec=infinity
UMask=0077
#User=redis
#Group=redis
#WorkingDirectory=/var/lib/redis

[Install]
WantedBy=multi-user.target

It's a simple one

Comment From: tanshaolong

Thank for your help! I can start the redis master node now when changed the systemd server file by your suggestion. I will update my ansible playbook. Thank you very much. :)

Comment From: laixintao

Glad to help, then the problem is about systemd files' readwritedirectory I think.

Can you close this issue 😃

On Mon, Jan 6, 2020 at 11:11 Ray Tan notifications@github.com wrote:

Thank for your help! I can start the redis master node now when changed the systemd server file by your suggestion. I will update my ansible playbook. Thank you very much. :)

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/6370?email_source=notifications&email_token=ACJ2JI2GLN5YX2GYSGMTDSDQ4KOMZA5CNFSM4IUGXMA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIEJDRY#issuecomment-570986951, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACJ2JI2LQPKU4AINYQ27DSDQ4KOMZANCNFSM4IUGXMAQ .

Comment From: tanshaolong

As the testing, the redis server is running after I just remove "ReadOnlyDirectories" from the systemd file. I close the issue as it is resolved. :) Thank for your help!

Comment From: streamnsight

I was having trouble starting redis-sentinel as well, but it was because I was trying to set a reconfig script with sentinel client-reconfig-script mymaster /var/run/redis/reconfig.sh

and I was getting:

Apr 17 16:17:15 haproxy redis-sentinel[15992]: *** FATAL CONFIG FILE ERROR ***
Apr 17 16:17:15 haproxy redis-sentinel[15992]: Reading the configuration file, at line 146
Apr 17 16:17:15 haproxy redis-sentinel[15992]: >>> 'sentinel client-reconfig-script mymaster /var/run/redis/reconfig.sh'
Apr 17 16:17:15 haproxy redis-sentinel[15992]: Client reconfiguration script seems non existing or non executable.

even though my script was executable and owned by the user redis

It happened to be a selinux issue

sudo setsebool -P redis_enable_notify 1 did not help, and there is no equivalent redis_enable_reconfig flag that I could find.

and I had to set sudo semanage permissive -a redis_t

which I am not sure is ideal. I didn't find any other way, so if someone knows of a way to activate recofnig script without making redis_t permissive, that's be useful to know.

@antirez might have some insight?

Comment From: fengtan

@streamnsight I ran into the same issue and solved it by setting the selinux context redis_exec_t on the notification script:

sudo chcon -t redis_exec_t /var/run/redis/notify.sh

Reference: https://man.linuxreviews.org/man8/redis_selinux.8.html