I was performing some tests with Redis Sentinel and I prepared this configuration file:
# Sentinel binding port
port 26379
# daemonize
daemonize yes
pidfile "/var/run/redis_sentinel.pid"
# logging
logfile "/var/log/redis_sentinel.log"
# monitoring set
sentinel monitor setmaster 10.0.0.1 6379 2
# assume master is down after being unresponsive for 20s
sentinel down-after-milliseconds setmaster 20000
# reconfigure one slave at a time
sentinel parallel-syncs setmaster 1
# wait for 3m before assuming failover went wrong
sentinel failover-timeout setmaster 180000
# Sentinel working directory
dir "/tmp"
just as expected, Sentinel rewrote it. However, I noticed that it made some mess with the comments:
# Sentinel binding port
port 26379
# daemonize
daemonize yes
pidfile "/var/run/redis_sentinel.pid"
# logging
logfile "/var/log/redis_sentinel.log"
# monitoring set
sentinel myid 4aa48f86aea39c400e71d90d7d98db0f7b923640
# assume master is down after being unresponsive for 20s
sentinel deny-scripts-reconfig yes
# reconfigure one slave at a time
sentinel monitor setmaster 10.0.0.1 6379 2
# wait for 3m before assuming failover went wrong
sentinel down-after-milliseconds setmaster 20000
# Sentinel working directory
dir "/tmp"
# Generated by CONFIG REWRITE
sentinel config-epoch setmaster 0
sentinel leader-epoch setmaster 0
sentinel current-epoch 0
In fact, it kept comments position unchanged, while modifying some other lines.
The test was performed using Redis 5.0 RC5.
Cheers
Marco
Comment From: OrangeDog
A fresh install on Ubuntu Bionic, which starts sentinel after installation, results in this config.
Note that myid, monitor, config-epoch and leader-epoch have replaced the wrong properties instead of being appended.
# Example sentinel.conf
daemonize yes
pidfile "/var/run/sentinel/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1
#
# protected-mode no
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379
# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4
# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir "/var/lib/redis"
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Slaves are auto-discovered, so you don't need to specify slaves in
# any way. Sentinel itself will rewrite this configuration file adding
# the slaves using additional configuration options.
# Also note that the configuration file is rewritten when a
# slave is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel myid 1a7f950c9d03b14f632da3cd1fcdb86bf9b1fe12
# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and slaves.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for slaves, so it is not
# possible to set a different password in masters and slaves instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached slave or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel monitor mymaster 127.0.0.1 6379 2
# sentinel parallel-syncs <master-name> <numslaves>
#
# How many slaves we can reconfigure to point to the new slave simultaneously
# during the failover. Use a low number if you use the slaves to serve query
# to avoid that all the slaves will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel config-epoch mymaster 0
# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
# already tried against the same master by a given Sentinel, is two
# times the failover timeout.
#
# - The time needed for a slave replicating to a wrong master according
# to a Sentinel current configuration, to be forced to replicate
# with the right master, is exactly the failover timeout (counting since
# the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
# did not produced any configuration change (SLAVEOF NO ONE yet not
# acknowledged by the promoted slave).
#
# - The maximum time a failover in progress waits for all the slaves to be
# reconfigured as slaves of the new master. However even after this time
# the slaves will be reconfigured by the Sentinels anyway, but not with
# the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
sentinel leader-epoch mymaster 0
# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.
# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh
# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
#
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
#
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected slave
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
# Generated by CONFIG REWRITE
sentinel current-epoch 0
Comment From: OrangeDog
Even without comments, the rewrite breaks the config by replacing those properties.
For down-after-milliseconds, parallel-syncs, and failover-timeout it appears to be because they're set to the defaults. However, it will still move other properties. This is a big headache when you're trying to manage your configuration.
sentinel monitor mymaster 192.0.2.1 6379 2
# -- start managed zone --
daemonize yes
pidfile "/var/run/sentinel/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
syslog-enabled yes
protected-mode no
port 26379
dir "/var/lib/redis"
sentinel auth-pass mymaster secret
# -- end managed zone --
becomes
sentinel myid b08f665788c86f86346af473c6251e74c42a4186
# -- start managed zone --
daemonize yes
pidfile "/var/run/sentinel/redis-sentinel.pid"
logfile "/var/log/redis/redis-sentinel.log"
syslog-enabled yes
protected-mode no
port 26379
dir "/var/lib/redis"
sentinel monitor mymaster 192.0.2.1 6379 2
# -- end managed zone --
# Generated by CONFIG REWRITE
sentinel auth-pass mymaster secret
sentinel config-epoch mymaster 0
sentinel leader-epoch mymaster 0
sentinel known-slave mymaster 192.0.2.2 6379
sentinel known-slave mymaster 192.0.2.3 6379
sentinel known-sentinel mymaster 192.0.2.2 26379 d6677e79f358bd20eb5fc2e55b23451aa26f026d
sentinel known-sentinel mymaster 192.0.2.3 26379 0690ef28aa86cee7d7cdb80cf2700666d5a73216
sentinel current-epoch 0
Comment From: OrangeDog
Working on my upgrade to 5 and it started adding sentinel deny-scripts-reconfig yes, destroying the idempotent balance I'd managed to achieve.
Please stop reordering config files on disk. For existing properties, replace the value. For new properties, append them to the end.
The documentation claims that CONFIG REWRITE does this, but evidently SENTINEL flushconfig doesn't.
Comment From: OrangeDog
@antirez nobody has looked at this issue for nearly two years
Comment From: marcobellaccini
Really glad to see this closed! 🎉
Comment From: hwware
thanks @marcobellaccini and really sorry for the long wait, there were lots of issues pending in Redis that need to be fixed. But we will keep moving forward and fix these eventually. Please let us know if you see other issues in the future. Thanks
Comment From: OrangeDog
This was fixed in 6.2 right? So still a problem in Ubuntu's jammy packages.
Comment From: oranagra
yes, #8271 is part of 6.2.0 GA