We provisioned 6 instances of Redis with 6 sentinels.
When the master comes down, a "new master" is elected. So far so good. If we restart the "old master", he is changed in slave mode, but a synchronization process causes the "new master" to restore the state from the "old master" during few seconds.
Is the behavior expected ?
Hereunder logs. Thanks.
Old master logs
1:M 07 Oct 2020 11:47:29.545 # Redis is now ready to exit, bye bye...
1:C 07 Oct 2020 11:49:07.910 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 07 Oct 2020 11:49:07.910 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 07 Oct 2020 11:49:07.910 # Configuration loaded
1:M 07 Oct 2020 11:49:07.911 * Running mode=standalone, port=6379.
1:M 07 Oct 2020 11:49:07.911 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 07 Oct 2020 11:49:07.911 # Server initialized
1:M 07 Oct 2020 11:49:07.911 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 07 Oct 2020 11:49:07.965 * Reading RDB preamble from AOF file...
1:M 07 Oct 2020 11:49:07.965 * Loading RDB produced by version 6.0.8
1:M 07 Oct 2020 11:49:07.965 * RDB age 5293 seconds
1:M 07 Oct 2020 11:49:07.965 * RDB memory usage when created 1.46 Mb
1:M 07 Oct 2020 11:49:07.965 * RDB has an AOF tail
1:M 07 Oct 2020 11:49:07.965 * Reading the remaining AOF tail...
1:M 07 Oct 2020 11:49:08.055 * DB loaded from append only file: 0.142 seconds
1:M 07 Oct 2020 11:49:08.055 * Ready to accept connections
1:S 07 Oct 2020 11:49:18.425 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 07 Oct 2020 11:49:18.426 * REPLICAOF 35.189.116.150:6381 enabled (user request from 'id=7 addr=34.65.181.108:36687 fd=11 name=sentinel-bc88aba5-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=202 qbuf-free=32566 obl=45 oll=0 omem=0 events=r cmd=exec user=default')
1:S 07 Oct 2020 11:49:18.426 # CONFIG REWRITE executed with success.
1:S 07 Oct 2020 11:49:19.084 * Connecting to MASTER 35.189.116.150:6381
1:S 07 Oct 2020 11:49:19.084 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:49:19.084 * Non blocking connect for SYNC fired the event.
1:S 07 Oct 2020 11:49:19.085 * Master replied to PING, replication can continue...
1:S 07 Oct 2020 11:49:19.086 * Trying a partial resynchronization (request de9bcb529ee1bfe1364f7bcf0d7136ad500b08a0:1).
1:S 07 Oct 2020 11:49:19.089 * Full resync from master: 399a31f3d5e148997280b4722f022a33e201a50f:7765877
1:S 07 Oct 2020 11:49:19.089 * Discarding previously cached master state.
1:S 07 Oct 2020 11:49:19.165 * MASTER <-> REPLICA sync: receiving 194 bytes from master to disk
1:S 07 Oct 2020 11:49:19.165 * MASTER <-> REPLICA sync: Flushing old data
1:S 07 Oct 2020 11:49:19.165 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 07 Oct 2020 11:49:19.165 * Loading RDB produced by version 6.0.8
1:S 07 Oct 2020 11:49:19.165 * RDB age 0 seconds
1:S 07 Oct 2020 11:49:19.165 * RDB memory usage when created 1.68 Mb
1:S 07 Oct 2020 11:49:19.166 * MASTER <-> REPLICA sync: Finished with success
1:S 07 Oct 2020 11:49:19.167 * Background append only file rewriting started by pid 18
1:S 07 Oct 2020 11:49:19.195 * AOF rewrite child asks to stop sending diffs.
18:C 07 Oct 2020 11:49:19.195 * Parent agreed to stop sending diffs. Finalizing AOF...
18:C 07 Oct 2020 11:49:19.195 * Concatenating 0.00 MB of AOF diff received from parent.
18:C 07 Oct 2020 11:49:19.195 * SYNC append only file rewrite performed
18:C 07 Oct 2020 11:49:19.196 * AOF rewrite: 6 MB of memory used by copy-on-write
1:S 07 Oct 2020 11:49:19.284 * Background AOF rewrite terminated with success
1:S 07 Oct 2020 11:49:19.284 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
1:S 07 Oct 2020 11:49:19.284 * Background AOF rewrite finished successfully
New master logs
``1:S 07 Oct 2020 11:47:19.875 * Background AOF rewrite finished successfully
1:S 07 Oct 2020 11:47:29.579 # Connection with master lost.
1:S 07 Oct 2020 11:47:29.579 * Caching the disconnected master state.
1:S 07 Oct 2020 11:47:29.798 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:29.799 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:29.799 # Error condition on socket for SYNC: Operation now in progress
1:S 07 Oct 2020 11:47:30.803 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:30.803 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:30.804 # Error condition on socket for SYNC: Operation now in progress
1:S 07 Oct 2020 11:47:31.805 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:31.805 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:31.810 # Error condition on socket for SYNC: Operation now in progress
1:S 07 Oct 2020 11:47:32.810 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:32.810 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:32.811 # Error condition on socket for SYNC: Operation now in progress
1:S 07 Oct 2020 11:47:33.814 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:33.815 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:33.815 # Error condition on socket for SYNC: Operation now in progress
1:S 07 Oct 2020 11:47:34.817 * Connecting to MASTER 35.189.116.150:6380
1:S 07 Oct 2020 11:47:34.817 * MASTER <-> REPLICA sync started
1:S 07 Oct 2020 11:47:34.818 # Error condition on socket for SYNC: Operation now in progress
1:M 07 Oct 2020 11:47:34.983 * Discarding previously cached master state.
1:M 07 Oct 2020 11:47:34.984 # Setting secondary replication ID to 927cea31aa75fe6af74421ccdaab201e7483d262, valid up to offset: 7679838. New replication ID is 399a31f3d5e148997280b4722f022a33e201a50f
1:M 07 Oct 2020 11:47:34.984 * MASTER MODE enabled (user request from 'id=10 addr=34.65.181.108:37089 fd=12 name=sentinel-886589d3-cmd age=15 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=188 qbuf-free=32580 obl=45 oll=0 omem=0 events=r cmd=exec user=default')
1:M 07 Oct 2020 11:47:34.984 # CONFIG REWRITE executed with success.
1:M 07 Oct 2020 11:47:36.730 * Replica 34.65.181.108:6379 asks for synchronization
1:M 07 Oct 2020 11:47:36.730 * Partial resynchronization request from 34.65.181.108:6379 accepted. Sending 608 bytes of backlog starting from offset 7679838.
1:M 07 Oct 2020 11:47:36.814 * Replica 34.65.181.108:6381 asks for synchronization
1:M 07 Oct 2020 11:47:36.814 * Partial resynchronization request from 34.65.181.108:6381 accepted. Sending 643 bytes of backlog starting from offset 7679838.
1:M 07 Oct 2020 11:47:38.018 * Replica 34.65.181.108:6380 asks for synchronization
1:M 07 Oct 2020 11:47:38.018 * Partial resynchronization request from 34.65.181.108:6380 accepted. Sending 1963 bytes of backlog starting from offset 7679838.
1:M 07 Oct 2020 11:48:19.066 * 10000 changes in 60 seconds. Saving...
1:M 07 Oct 2020 11:48:19.069 * Background saving started by pid 20
20:C 07 Oct 2020 11:48:19.074 * DB saved on disk
20:C 07 Oct 2020 11:48:19.075 * RDB: 10 MB of memory used by copy-on-write
1:M 07 Oct 2020 11:48:19.169 * Background saving terminated with success
1:M 07 Oct 2020 11:49:19.086 * Replica 35.189.116.150:6380 asks for synchronization
1:M 07 Oct 2020 11:49:19.086 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'de9bcb529ee1bfe1364f7bcf0d7136ad500b08a0', my replication IDs are '399a31f3d5e148997280b4722f022a33e201a50f' and '927cea31aa75fe6af74421ccdaab201e7483d262')
1:M 07 Oct 2020 11:49:19.086 * Starting BGSAVE for SYNC with target: disk
1:M 07 Oct 2020 11:49:19.089 * Background saving started by pid 21
21:C 07 Oct 2020 11:49:19.093 * DB saved on disk
21:C 07 Oct 2020 11:49:19.093 * RDB: 10 MB of memory used by copy-on-write
1:M 07 Oct 2020 11:49:19.164 * Background saving terminated with success
1:M 07 Oct 2020 11:49:19.165 * Synchronization with replica 35.189.116.150:6380 succeeded
Comment From: b0l0k
OK. Found.
The issue came from StackExchange.Redis c# sentinel mgmt.