Hi, I am currently doing some local testing on a redis cluster. The redis cluster node consists of 3 masters and 2 slaves:
8af8e6e6f85d8b67a4426600266de6f4c5298eaa 127.0.0.1:7006@17006 slave 35236135e3efec74b3f7061c67244c90e37bd285 0 1674575529000 5 connected
35236135e3efec74b3f7061c67244c90e37bd285 127.0.0.1:7001@17001 master - 0 1674575528593 5 connected 5461-10922
e6b6b13d1c41e0cd932c27389ceb87eaeb426892 127.0.0.1:7002@17002 master - 0 1674575529626 3 connected 10923-16383
5ebe54e0cab775be133c8333dcd08e1540919cbb 127.0.0.1:7000@17000 myself,master - 0 1674575529000 1 connected 0-5460
faf61f61a4bcbf47d262ddced2e1fd161f1cf8c5 127.0.0.1:7004@17004 slave 5ebe54e0cab775be133c8333dcd08e1540919cbb 0 1674575529800 1 connected
Not all the redis nodes are running on the same version. One of these nodes is running on an older redis version.
Node 127.0.0.1:7004 (faf61f61a4bcbf47d262ddced2e1fd161f1cf8c5) uses Redis 6.2.5 All other nodes use Redis 7.0.8
When i try to failover 127.0.0.1:7004 and make this node the master, the command responds "OK"
127.0.0.1:7004> cluster failover
OK
But in the logs, it says:
16525:S 24 Jan 2023 15:52:16.282 # Manual failover timed out.
and the cluster node distribution stays the same.
35236135e3efec74b3f7061c67244c90e37bd285 127.0.0.1:7001@17001 slave 8af8e6e6f85d8b67a4426600266de6f4c5298eaa 0 1674576738000 6 connected
e6b6b13d1c41e0cd932c27389ceb87eaeb426892 127.0.0.1:7002@17002 master - 0 1674576738595 3 connected 10923-16383
faf61f61a4bcbf47d262ddced2e1fd161f1cf8c5 127.0.0.1:7004@17004 myself,slave 5ebe54e0cab775be133c8333dcd08e1540919cbb 0 1674576739000 1 connected
8af8e6e6f85d8b67a4426600266de6f4c5298eaa 127.0.0.1:7006@17006 master - 0 1674576739000 6 connected 5461-10922
5ebe54e0cab775be133c8333dcd08e1540919cbb 127.0.0.1:7000@17000 master - 0 1674576738696 1 connected 0-5460
Is there a way to failover a slave which is running on an older version of redis?
Comment From: hwware
Could you please provide your 5 nodes detail config? exclude the confidential information So We could reproduce your issue. Thanks
Comment From: stefanklas
The nodes all have a very similar configuration:
cluster-enabled yes
cluster-node-timeout 5000
cluster-config-file /tmp/node4.conf
protected-mode no
maxmemory 2gb
requirepass foobar
masterauth foobar
port 7004
Comment From: stefanklas
127.0.0.1:7004> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:5
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:8571
cluster_stats_messages_pong_sent:9000
cluster_stats_messages_meet_sent:1
cluster_stats_messages_mfstart_sent:13
cluster_stats_messages_sent:17585
cluster_stats_messages_ping_received:9000
cluster_stats_messages_pong_received:8575
cluster_stats_messages_auth-req_received:3
cluster_stats_messages_received:17578
OS: Mac OS (Monterey), Darwin 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
Comment From: stefanklas
127.0.0.1:7004
26040:C 24 Jan 2023 16:28:11.807 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26040:C 24 Jan 2023 16:28:11.807 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=26040, just started
26040:C 24 Jan 2023 16:28:11.807 # Configuration loaded
26040:M 24 Jan 2023 16:28:11.807 * Increased maximum number of open files to 10032 (it was originally set to 256).
26040:M 24 Jan 2023 16:28:11.807 * monotonic clock: POSIX clock_gettime
26040:M 24 Jan 2023 16:28:11.808 * Node configuration loaded, I'm faf61f61a4bcbf47d262ddced2e1fd161f1cf8c5
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 7004
| `-._ `._ / _.-' | PID: 26040
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
26040:M 24 Jan 2023 16:28:11.833 # Server initialized
26040:M 24 Jan 2023 16:28:11.833 * Loading RDB produced by version 6.2.5
26040:M 24 Jan 2023 16:28:11.833 * RDB age 1 seconds
26040:M 24 Jan 2023 16:28:11.833 * RDB memory usage when created 1.53 Mb
26040:M 24 Jan 2023 16:28:11.833 * DB loaded from disk: 0.000 seconds
26040:M 24 Jan 2023 16:28:11.833 * Ready to accept connections
26040:S 24 Jan 2023 16:28:11.904 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
26040:S 24 Jan 2023 16:28:11.904 * Connecting to MASTER 127.0.0.1:7000
26040:S 24 Jan 2023 16:28:11.920 * MASTER <-> REPLICA sync started
26040:S 24 Jan 2023 16:28:11.920 # Cluster state changed: ok
26040:S 24 Jan 2023 16:28:11.982 * Non blocking connect for SYNC fired the event.
26040:S 24 Jan 2023 16:28:11.982 * Master replied to PING, replication can continue...
26040:S 24 Jan 2023 16:28:11.982 * Trying a partial resynchronization (request 28fa7145c8bd3230cfd8fbec87d9ccc24979ff6c:1).
26040:S 24 Jan 2023 16:28:16.278 * Full resync from master: f9d71fb41d4fd6ed56cd28720bc949392f78e784:10048
26040:S 24 Jan 2023 16:28:16.278 * Discarding previously cached master state.
26040:S 24 Jan 2023 16:28:16.280 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
26040:S 24 Jan 2023 16:28:16.281 * MASTER <-> REPLICA sync: Flushing old data
26040:S 24 Jan 2023 16:28:16.281 * MASTER <-> REPLICA sync: Loading DB in memory
26040:S 24 Jan 2023 16:28:16.282 # Can't handle RDB format version 10
26040:S 24 Jan 2023 16:28:16.282 # Failed trying to load the MASTER synchronization DB from disk
...
...
...
127.0.0.1:7000
26148:C 24 Jan 2023 16:29:49.406 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
26148:C 24 Jan 2023 16:29:49.406 # Redis version=7.0.8, bits=64, commit=00000000, modified=0, pid=26148, just started
26148:C 24 Jan 2023 16:29:49.406 # Configuration loaded
26148:M 24 Jan 2023 16:29:49.407 * Increased maximum number of open files to 10032 (it was originally set to 256).
26148:M 24 Jan 2023 16:29:49.407 * monotonic clock: POSIX clock_gettime
26148:M 24 Jan 2023 16:29:49.408 * Node configuration loaded, I'm 5ebe54e0cab775be133c8333dcd08e1540919cbb
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 7.0.8 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 7000
| `-._ `._ / _.-' | PID: 26148
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
26148:M 24 Jan 2023 16:29:49.409 # WARNING: The TCP backlog setting of 511 cannot be enforced because kern.ipc.somaxconn is set to the lower value of 128.
26148:M 24 Jan 2023 16:29:49.409 # Server initialized
26148:M 24 Jan 2023 16:29:49.410 * Loading RDB produced by version 7.0.8
26148:M 24 Jan 2023 16:29:49.410 * RDB age 2259 seconds
26148:M 24 Jan 2023 16:29:49.410 * RDB memory usage when created 2.18 Mb
26148:M 24 Jan 2023 16:29:49.410 * Done loading RDB, keys loaded: 0, keys expired: 0.
26148:M 24 Jan 2023 16:29:49.410 * DB loaded from disk: 0.000 seconds
26148:M 24 Jan 2023 16:29:49.410 * Ready to accept connections
26148:M 24 Jan 2023 16:29:49.594 * Replica 127.0.0.1:7004 asks for synchronization
26148:M 24 Jan 2023 16:29:49.594 * Full resync requested by replica 127.0.0.1:7004
26148:M 24 Jan 2023 16:29:49.594 * Delay next BGSAVE for diskless SYNC
26148:M 24 Jan 2023 16:29:51.477 # Cluster state changed: ok
26148:M 24 Jan 2023 16:29:54.496 * Starting BGSAVE for SYNC with target: replicas sockets
26148:M 24 Jan 2023 16:29:54.497 * Background RDB transfer started by pid 26160
26160:C 24 Jan 2023 16:29:54.498 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
26148:M 24 Jan 2023 16:29:54.499 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
26148:M 24 Jan 2023 16:29:54.499 # Connection with replica 127.0.0.1:7004 lost.
26148:M 24 Jan 2023 16:29:54.521 * Replica 127.0.0.1:7004 asks for synchronization
26148:M 24 Jan 2023 16:29:54.521 * Full resync requested by replica 127.0.0.1:7004
26148:M 24 Jan 2023 16:29:54.521 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC
26148:M 24 Jan 2023 16:29:54.598 * Background RDB transfer terminated with success
26148:M 24 Jan 2023 16:29:59.521 * Starting BGSAVE for SYNC with target: replicas sockets
26148:M 24 Jan 2023 16:29:59.521 * Background RDB transfer started by pid 26169
26169:C 24 Jan 2023 16:29:59.522 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
26148:M 24 Jan 2023 16:29:59.523 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
26148:M 24 Jan 2023 16:29:59.523 # Connection with replica 127.0.0.1:7004 lost.
...
...
Comment From: stefanklas
Have tried with version redis 6.2.9 which works. Maybe this is an issue just with 6.2.5?
Comment From: hwware
Have tried with version redis 6.2.9 which works. Maybe this is an issue just with 6.2.5?
If 6.2.9 works, please use it. new version always solve some bugs, better performance. note: The current latest version for Redis 6 is redis 6.2.10, please try it, Thanks