Describe the question：
I am deploying redis cluster, and I have individual slave nodes in site2 showing master_link_status:down in info replication. And the log keeps executing full sync repeatedly. (Seems to have failed, so keep trying again). Question: How do I optimize this configuration, and from what points do I consider how to solve this problem?
Keep repeating the following log in the log：
9:S 05 Sep 2024 01:03:53.698 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:03:55.725 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:03:55.725 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:03:55.725 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:03:55.726 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:03:55.726 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:00.125 * Full resync from master: 4988964b32fb00600a83ff42d3864eadcbd102f3:911714931441 9:S 05 Sep 2024 01:04:00.215 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk 9:S 05 Sep 2024 01:04:03.732 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:03.732 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:03.732 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:03.734 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:03.735 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:05.741 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:05.741 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:05.741 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:05.742 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:05.742 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:10.214 * Full resync from master: 4988964b32fb00600a83ff42d3864eadcbd102f3:911838678790 9:S 05 Sep 2024 01:04:10.303 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk 9:S 05 Sep 2024 01:04:13.777 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:13.778 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:13.778 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:13.780 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:13.782 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:15.833 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:15.834 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:15.834 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:15.835 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:15.835 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:20.296 * Full resync from master: 4988964b32fb00600a83ff42d3864eadcbd102f3:911995974370 9:S 05 Sep 2024 01:04:20.389 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk 9:S 05 Sep 2024 01:04:23.809 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:23.809 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:23.810 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:23.811 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:23.813 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287). 9:S 05 Sep 2024 01:04:25.934 * Connecting to MASTER xxxxxxx:6382 9:S 05 Sep 2024 01:04:25.934 * MASTER <-> REPLICA sync started 9:S 05 Sep 2024 01:04:25.935 * Non blocking connect for SYNC fired the event. 9:S 05 Sep 2024 01:04:25.935 * Master replied to PING, replication can continue... 9:S 05 Sep 2024 01:04:25.935 * Trying a partial resynchronization (request bc742bc367b1e899a0bc6d958817981d538e2ab2:137853614287).
This cluster has the following configurations: site1: Five Shards are deployed. Each shard has a leader node and two slave nodes
The configuration of each node is as follows: - timeout 0 - maxclients 100000 - lazyfree-lazy-eviction yes - lazyfree-lazy-expire yes - lazyfree-lazy-user-del yes - lazyfree-lazy-user-flush yes - lazyfree-lazy-server-del yes - replica-lazy-flush yes - latency-monitor-threshold 50 - client-output-buffer-limit pubsub 0 0 0
resources: limits: cpu: "2" memory: 8Gi requests: cpu: 600m memory: 2432Mi
site2: Three slave nodes are deployed for all Shards in site1.
The configuration of each node is as follows: - timeout 0 - maxclients 100000 - lazyfree-lazy-eviction yes - lazyfree-lazy-expire yes - lazyfree-lazy-user-del yes - lazyfree-lazy-user-flush yes - lazyfree-lazy-server-del yes - replica-lazy-flush yes - client-output-buffer-limit pubsub 0 0 0
resources: limits: cpu: "2" memory: 6Gi requests: cpu: 600m memory: 3Gi
Comment From: sundb
@AlenChenGitHub maybe you can try to increase config repl-backlog-size to a higher value, .e.g 10mb.
Comment From: AlenChenGitHub
@sundb hi，I have tried to configure repl-backlog-size 10mb in the configuration file. Currently,the log of the slave node is still displayed same question. Does it need to be redeployed to be useful?
Here's what I get out of the nodes: 1) "repl-backlog-size" 2) "10485760"
Comment From: sundb
@AlenChenGitHub this config is for master, did you make the wrong change?
Comment From: AlenChenGitHub
@sundb I have confirmed that on the master node, I executed config set repl-backlog-size 10mb, and the data obtained after that is still 10485760.
Comment From: AlenChenGitHub
@sundb The redis cluster is currently connected to the application in use, maybe I need to redeploy it to make it work?
Comment From: sundb
@AlenChenGitHub can you share the output of INFO ALL of the master and the replica.
i saw that reploff the replica requested is 137853614287, how long has your instances been running?
Comment From: AlenChenGitHub
@sundb This cluster has been running for 21 hours.
Comment From: AlenChenGitHub
master:
Defaulted container "redis" out of: redis, redis-exporter
Server
redis_version:7.0.13 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:ec0335a785106d85 redis_mode:cluster os:Linux 4.18.0-372.49.1.el8_6.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:12.2.1 process_id:8 process_supervised:no run_id:b52e33409926f81eded13acf6aadcf72a77682a3 tcp_port:6379 server_time_usec:1725503729457306 uptime_in_seconds:75983 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:14227697 executable:/data/redis-server config_file:/etc/config/redis.conf io_threads_active:0
Clients
connected_clients:755 cluster_connections:60 maxclients:100000 client_recent_max_input_buffer:20704 client_recent_max_output_buffer:25999072 blocked_clients:127 tracking_clients:0 clients_in_timeout_table:127
Memory
used_memory:2928326280 used_memory_human:2.73G used_memory_rss:3076980736 used_memory_rss_human:2.87G used_memory_peak:3073581376 used_memory_peak_human:2.86G used_memory_peak_perc:95.27% used_memory_overhead:83612145 used_memory_startup:2126624 used_memory_dataset:2844714135 used_memory_dataset_perc:97.22% allocator_allocated:2927100312 allocator_active:3024683008 allocator_resident:3089031168 total_system_memory:540453150720 total_system_memory_human:503.34G used_memory_lua:61440 used_memory_vm_eval:61440 used_memory_lua_human:60.00K used_memory_scripts_eval:4920 number_of_cached_scripts:9 number_of_functions:0 number_of_libraries:0 used_memory_vm_functions:32768 used_memory_vm_total:94208 used_memory_vm_total_human:92.00K used_memory_functions:184 used_memory_scripts:5104 used_memory_scripts_human:4.98K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.03 allocator_frag_bytes:97582696 allocator_rss_ratio:1.02 allocator_rss_bytes:64348160 rss_overhead_ratio:1.00 rss_overhead_bytes:-12050432 mem_fragmentation_ratio:1.05 mem_fragmentation_bytes:149997240 mem_not_counted_for_evict:32695648 mem_replication_backlog:10486308 mem_total_replication_buffers:43222432 mem_clients_slaves:32736672 mem_clients_normal:7487157 mem_cluster_links:63104 mem_aof_buffer:0 mem_allocator:jemalloc-5.2.1 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0
Persistence
loading:0 async_loading:0 current_cow_peak:9158656 current_cow_size:9158656 current_cow_size_age:2 current_fork_perc:12.57 current_save_keys_processed:56321 current_save_keys_total:447997 rdb_changes_since_last_save:1429800286 rdb_bgsave_in_progress:1 rdb_last_save_time:1725427746 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:5 rdb_current_bgsave_time_sec:3 rdb_saves:0 rdb_last_cow_size:7831552 rdb_last_load_keys_expired:0 rdb_last_load_keys_loaded:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_rewrites:0 aof_rewrites_consecutive_failures:0 aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:137841 total_commands_processed:1643631485 instantaneous_ops_per_sec:28305 total_net_input_bytes:1124493524202 total_net_output_bytes:7352247922816 total_net_repl_input_bytes:17876989190 total_net_repl_output_bytes:5091601766385 instantaneous_input_kbps:18783.78 instantaneous_output_kbps:180512.95 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:142480.55 rejected_connections:7624 sync_full:11807 sync_partial_ok:7 sync_partial_err:11805 expired_keys:341 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:3959 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:680127746 keyspace_misses:12265454 pubsub_channels:0 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:48582 total_forks:5712 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:252213 dump_payload_sanitizations:0 total_reads_processed:932412359 total_writes_processed:2741219386 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:1231501 reply_buffer_expands:3287233
Replication
role:master connected_slaves:5 slave0:ip=19.40.2.11,port=6384,state=online,offset=993220965315,lag=0 slave1:ip=19.40.2.11,port=6383,state=online,offset=993212289899,lag=1 slave2:ip=18.40.2.16,port=6383,state=online,offset=993208930866,lag=1 slave3:ip=18.40.2.16,port=6384,state=online,offset=993220655579,lag=0 slave4:ip=18.40.2.16,port=6382,state=wait_bgsave,offset=0,lag=0 master_failover_state:no-failover master_replid:4988964b32fb00600a83ff42d3864eadcbd102f3 master_replid2:b79f4ced1372529624c2380dd1a2ab51ecc19839 master_repl_offset:993223260968 second_repl_offset:137590869313 repl_backlog_active:1 repl_backlog_size:10485760 repl_backlog_first_byte_offset:993180192761 repl_backlog_histlen:43068208
CPU
used_cpu_sys:20954.965371 used_cpu_user:11366.824784 used_cpu_sys_children:1680.406237 used_cpu_user_children:11920.829606 used_cpu_sys_main_thread:20906.118326 used_cpu_user_main_thread:11352.971059
Modules
Commandstats
cmdstat_incrby:calls=107256,usec=433956,usec_per_call=4.05,rejected_calls=1,failed_calls=0 cmdstat_hello:calls=104320,usec=389877,usec_per_call=3.74,rejected_calls=0,failed_calls=0 cmdstat_replconf:calls=340195,usec=262927,usec_per_call=0.77,rejected_calls=0,failed_calls=0 cmdstat_ttl:calls=204454,usec=543329,usec_per_call=2.66,rejected_calls=0,failed_calls=0 cmdstat_psync:calls=11814,usec=350403,usec_per_call=29.66,rejected_calls=0,failed_calls=0 cmdstat_xack:calls=57794,usec=131046,usec_per_call=2.27,rejected_calls=0,failed_calls=0 cmdstat_role:calls=15373,usec=86692,usec_per_call=5.64,rejected_calls=0,failed_calls=0 cmdstat_zrem:calls=508133,usec=6918697,usec_per_call=13.62,rejected_calls=1,failed_calls=0 cmdstat_type:calls=176,usec=348,usec_per_call=1.98,rejected_calls=0,failed_calls=0 cmdstat_evalsha:calls=777759,usec=13424344,usec_per_call=17.26,rejected_calls=2,failed_calls=1422 cmdstat_select:calls=3,usec=1,usec_per_call=0.33,rejected_calls=0,failed_calls=0 cmdstat_srem:calls=302976,usec=2711344,usec_per_call=8.95,rejected_calls=0,failed_calls=0 cmdstat_xadd:calls=169281,usec=1618198,usec_per_call=9.56,rejected_calls=0,failed_calls=0 cmdstat_sismember:calls=140,usec=304,usec_per_call=2.17,rejected_calls=0,failed_calls=0 cmdstat_hkeys:calls=24,usec=36,usec_per_call=1.50,rejected_calls=0,failed_calls=0 cmdstat_exec:calls=58,usec=1882,usec_per_call=32.45,rejected_calls=0,failed_calls=0 cmdstat_cluster|replicate:calls=351,usec=112608,usec_per_call=320.82,rejected_calls=0,failed_calls=0 cmdstat_cluster|addslots:calls=1,usec=118,usec_per_call=118.00,rejected_calls=0,failed_calls=0 cmdstat_cluster|nodes:calls=7829,usec=1162172,usec_per_call=148.44,rejected_calls=0,failed_calls=0 cmdstat_cluster|meet:calls=23,usec=230,usec_per_call=10.00,rejected_calls=0,failed_calls=0 cmdstat_cluster|failover:calls=2,usec=2086,usec_per_call=1043.00,rejected_calls=0,failed_calls=0 cmdstat_cluster|info:calls=10362,usec=608706,usec_per_call=58.74,rejected_calls=0,failed_calls=0 cmdstat_cluster|shards:calls=7829,usec=1174093,usec_per_call=149.97,rejected_calls=0,failed_calls=0 cmdstat_cluster|slots:calls=30652,usec=1748475,usec_per_call=57.04,rejected_calls=0,failed_calls=0 cmdstat_cluster|myid:calls=7829,usec=8825,usec_per_call=1.13,rejected_calls=0,failed_calls=0 cmdstat_xread:calls=2171230,usec=17792555,usec_per_call=8.19,rejected_calls=13,failed_calls=0 cmdstat_del:calls=1505216,usec=13969476,usec_per_call=9.28,rejected_calls=6,failed_calls=0 cmdstat_ping:calls=11909,usec=8831,usec_per_call=0.74,rejected_calls=0,failed_calls=0 cmdstat_hdel:calls=159030,usec=1328618,usec_per_call=8.35,rejected_calls=0,failed_calls=0 cmdstat_getset:calls=73149,usec=342466,usec_per_call=4.68,rejected_calls=218222,failed_calls=0 cmdstat_decrby:calls=66129,usec=281379,usec_per_call=4.26,rejected_calls=0,failed_calls=0 cmdstat_command:calls=8543,usec=18255518,usec_per_call=2136.90,rejected_calls=0,failed_calls=0 cmdstat_command|docs:calls=1,usec=2371,usec_per_call=2371.00,rejected_calls=0,failed_calls=0 cmdstat_latency|histogram:calls=2533,usec=3141196,usec_per_call=1240.11,rejected_calls=0,failed_calls=0 cmdstat_latency|latest:calls=2533,usec=2491,usec_per_call=0.98,rejected_calls=0,failed_calls=0 cmdstat_get:calls=200574340,usec=347010724,usec_per_call=1.73,rejected_calls=8247,failed_calls=0 cmdstat_hmget:calls=253306780,usec=3961322198,usec_per_call=15.64,rejected_calls=11620,failed_calls=0 cmdstat_eval:calls=377020239,usec=3384815437,usec_per_call=8.98,rejected_calls=10322,failed_calls=0 cmdstat_scard:calls=1,usec=4,usec_per_call=4.00,rejected_calls=0,failed_calls=0 cmdstat_zadd:calls=1240871,usec=30082278,usec_per_call=24.24,rejected_calls=0,failed_calls=0 cmdstat_xdel:calls=57795,usec=395671,usec_per_call=6.85,rejected_calls=0,failed_calls=0 cmdstat_info:calls=3076,usec=2720469,usec_per_call=884.42,rejected_calls=0,failed_calls=0 cmdstat_set:calls=375923278,usec=896778024,usec_per_call=2.39,rejected_calls=0,failed_calls=0 cmdstat_client|setname:calls=2533,usec=6641,usec_per_call=2.62,rejected_calls=0,failed_calls=0 cmdstat_client|id:calls=7353,usec=3564,usec_per_call=0.48,rejected_calls=0,failed_calls=0 cmdstat_zscore:calls=52929,usec=87397,usec_per_call=1.65,rejected_calls=0,failed_calls=0 cmdstat_exists:calls=187659339,usec=149790615,usec_per_call=0.80,rejected_calls=1,failed_calls=0 cmdstat_zpopmin:calls=129827,usec=1797347,usec_per_call=13.84,rejected_calls=2,failed_calls=0 cmdstat_xgroup|create:calls=1,usec=7,usec_per_call=7.00,rejected_calls=0,failed_calls=0 cmdstat_xgroup|delconsumer:calls=6,usec=71,usec_per_call=11.83,rejected_calls=6,failed_calls=0 cmdstat_xreadgroup:calls=2843740,usec=17249333,usec_per_call=6.07,rejected_calls=0,failed_calls=0 cmdstat_zrangebyscore:calls=9888907,usec=36986821,usec_per_call=3.74,rejected_calls=595,failed_calls=0 cmdstat_multi:calls=58,usec=16,usec_per_call=0.28,rejected_calls=0,failed_calls=0 cmdstat_hmset:calls=196355079,usec=1632725410,usec_per_call=8.32,rejected_calls=25,failed_calls=0 cmdstat_expire:calls=150405,usec=659751,usec_per_call=4.39,rejected_calls=0,failed_calls=0 cmdstat_slowlog|len:calls=2533,usec=2875,usec_per_call=1.14,rejected_calls=0,failed_calls=0 cmdstat_slowlog|get:calls=2533,usec=33052,usec_per_call=13.05,rejected_calls=0,failed_calls=0 cmdstat_setex:calls=343,usec=3140,usec_per_call=9.15,rejected_calls=0,failed_calls=0 cmdstat_hgetall:calls=1320,usec=4323,usec_per_call=3.28,rejected_calls=0,failed_calls=0 cmdstat_xpending:calls=145940,usec=645652,usec_per_call=4.42,rejected_calls=0,failed_calls=269 cmdstat_xinfo|stream:calls=263,usec=2762,usec_per_call=10.50,rejected_calls=0,failed_calls=87 cmdstat_hget:calls=22752452,usec=57473068,usec_per_call=2.53,rejected_calls=1163,failed_calls=0 cmdstat_sadd:calls=525968,usec=3213123,usec_per_call=6.11,rejected_calls=0,failed_calls=0 cmdstat_config|get:calls=133511,usec=1294862,usec_per_call=9.70,rejected_calls=0,failed_calls=0 cmdstat_config|set:calls=7828,usec=92407,usec_per_call=11.80,rejected_calls=0,failed_calls=0 cmdstat_smembers:calls=352,usec=2138,usec_per_call=6.07,rejected_calls=0,failed_calls=0 cmdstat_hset:calls=454691,usec=3670011,usec_per_call=8.07,rejected_calls=0,failed_calls=0 cmdstat_xclaim:calls=17,usec=381,usec_per_call=22.41,rejected_calls=0,failed_calls=0 cmdstat_subscribe:calls=3,usec=38,usec_per_call=12.67,rejected_calls=0,failed_calls=0 cmdstat_zcard:calls=5058565,usec=11673524,usec_per_call=2.31,rejected_calls=1,failed_calls=0 cmdstat_scan:calls=77384,usec=6570766,usec_per_call=84.91,rejected_calls=0,failed_calls=0 cmdstat_zrange:calls=2586318,usec=26890462,usec_per_call=10.40,rejected_calls=0,failed_calls=0
Errorstats
errorstat_CLUSTERDOWN:count=249667 errorstat_ERR:count=1507 errorstat_MOVED:count=560 errorstat_NOGROUP:count=269 errorstat_NOSCRIPT:count=2 errorstat_UNBLOCKED:count=208
Latencystats
latency_percentiles_usec_incrby:p50=4.015,p99=12.031,p99.9=22.015 latency_percentiles_usec_hello:p50=3.007,p99=10.047,p99.9=20.095 latency_percentiles_usec_replconf:p50=1.003,p99=2.007,p99.9=4.015 latency_percentiles_usec_ttl:p50=2.007,p99=9.023,p99.9=17.023 latency_percentiles_usec_psync:p50=28.031,p99=45.055,p99.9=225.279 latency_percentiles_usec_xack:p50=2.007,p99=6.015,p99.9=13.055 latency_percentiles_usec_role:p50=5.023,p99=15.039,p99.9=26.111 latency_percentiles_usec_zrem:p50=12.031,p99=50.175,p99.9=88.063 latency_percentiles_usec_type:p50=2.007,p99=6.015,p99.9=11.007 latency_percentiles_usec_evalsha:p50=15.039,p99=64.255,p99.9=313.343 latency_percentiles_usec_select:p50=0.001,p99=1.003,p99.9=1.003 latency_percentiles_usec_srem:p50=8.031,p99=22.015,p99.9=39.167 latency_percentiles_usec_xadd:p50=8.031,p99=30.079,p99.9=54.015 latency_percentiles_usec_sismember:p50=2.007,p99=8.031,p99.9=8.031 latency_percentiles_usec_hkeys:p50=1.003,p99=14.015,p99.9=14.015 latency_percentiles_usec_exec:p50=21.119,p99=72.191,p99.9=479.231 latency_percentiles_usec_cluster|replicate:p50=313.343,p99=532.479,p99.9=688.127 latency_percentiles_usec_cluster|addslots:p50=118.271,p99=118.271,p99.9=118.271 latency_percentiles_usec_cluster|nodes:p50=151.551,p99=274.431,p99.9=387.071 latency_percentiles_usec_cluster|meet:p50=9.023,p99=22.015,p99.9=22.015 latency_percentiles_usec_cluster|failover:p50=946.175,p99=1146.879,p99.9=1146.879 latency_percentiles_usec_cluster|info:p50=52.223,p99=97.279,p99.9=152.575 latency_percentiles_usec_cluster|shards:p50=146.431,p99=239.615,p99.9=425.983 latency_percentiles_usec_cluster|slots:p50=60.159,p99=174.079,p99.9=224.255 latency_percentiles_usec_cluster|myid:p50=1.003,p99=2.007,p99.9=7.007 latency_percentiles_usec_xread:p50=5.023,p99=21.119,p99.9=37.119 latency_percentiles_usec_del:p50=6.015,p99=41.215,p99.9=62.207 latency_percentiles_usec_ping:p50=0.001,p99=5.023,p99.9=9.023 latency_percentiles_usec_hdel:p50=7.007,p99=22.015,p99.9=37.119 latency_percentiles_usec_getset:p50=4.015,p99=16.063,p99.9=29.055 latency_percentiles_usec_decrby:p50=4.015,p99=11.007,p99.9=19.071 latency_percentiles_usec_command:p50=2129.919,p99=2736.127,p99.9=3178.495 latency_percentiles_usec_command|docs:p50=2375.679,p99=2375.679,p99.9=2375.679 latency_percentiles_usec_latency|histogram:p50=1245.183,p99=2146.303,p99.9=2637.823 latency_percentiles_usec_latency|latest:p50=1.003,p99=2.007,p99.9=3.007 latency_percentiles_usec_get:p50=1.003,p99=7.007,p99.9=13.055 latency_percentiles_usec_hmget:p50=15.039,p99=41.215,p99.9=66.047 latency_percentiles_usec_eval:p50=8.031,p99=40.191,p99.9=58.111 latency_percentiles_usec_scard:p50=4.015,p99=4.015,p99.9=4.015 latency_percentiles_usec_zadd:p50=11.007,p99=47.103,p99.9=4128.767 latency_percentiles_usec_xdel:p50=7.007,p99=15.039,p99.9=26.111 latency_percentiles_usec_info:p50=1040.383,p99=1613.823,p99.9=2015.231 latency_percentiles_usec_set:p50=2.007,p99=7.007,p99.9=14.015 latency_percentiles_usec_client|setname:p50=2.007,p99=8.031,p99.9=15.039 latency_percentiles_usec_client|id:p50=0.001,p99=2.007,p99.9=9.023 latency_percentiles_usec_zscore:p50=2.007,p99=5.023,p99.9=10.047 latency_percentiles_usec_exists:p50=1.003,p99=2.007,p99.9=5.023 latency_percentiles_usec_zpopmin:p50=12.031,p99=37.119,p99.9=66.047 latency_percentiles_usec_xgroup|create:p50=7.007,p99=7.007,p99.9=7.007 latency_percentiles_usec_xgroup|delconsumer:p50=13.055,p99=20.095,p99.9=20.095 latency_percentiles_usec_xreadgroup:p50=28.031,p99=138.239,p99.9=290.815 latency_percentiles_usec_zrangebyscore:p50=3.007,p99=10.047,p99.9=21.119 latency_percentiles_usec_multi:p50=0.001,p99=1.003,p99.9=1.003 latency_percentiles_usec_hmset:p50=6.015,p99=41.215,p99.9=74.239 latency_percentiles_usec_expire:p50=4.015,p99=11.007,p99.9=24.063 latency_percentiles_usec_slowlog|len:p50=1.003,p99=4.015,p99.9=8.031 latency_percentiles_usec_slowlog|get:p50=12.031,p99=31.103,p99.9=50.175 latency_percentiles_usec_setex:p50=8.031,p99=20.095,p99.9=32.127 latency_percentiles_usec_hgetall:p50=3.007,p99=12.031,p99.9=23.039 latency_percentiles_usec_xpending:p50=4.015,p99=10.047,p99.9=20.095 latency_percentiles_usec_xinfo|stream:p50=10.047,p99=32.127,p99.9=42.239 latency_percentiles_usec_hget:p50=2.007,p99=8.031,p99.9=15.039 latency_percentiles_usec_sadd:p50=6.015,p99=19.071,p99.9=32.127 latency_percentiles_usec_config|get:p50=3.007,p99=307.199,p99.9=532.479 latency_percentiles_usec_config|set:p50=11.007,p99=26.111,p99.9=46.079 latency_percentiles_usec_smembers:p50=4.015,p99=26.111,p99.9=37.119 latency_percentiles_usec_hset:p50=7.007,p99=28.031,p99.9=55.039 latency_percentiles_usec_xclaim:p50=19.071,p99=47.103,p99.9=47.103 latency_percentiles_usec_subscribe:p50=13.055,p99=13.055,p99.9=13.055 latency_percentiles_usec_zcard:p50=2.007,p99=7.007,p99.9=14.015 latency_percentiles_usec_scan:p50=9.023,p99=4095.999,p99.9=6127.615 latency_percentiles_usec_zrange:p50=9.023,p99=29.055,p99.9=43.007
Cluster
cluster_enabled:1
Keyspace
db0:keys=447998,expires=37529,avg_ttl=30151823
Comment From: AlenChenGitHub
slave: Defaulted container "redis" out of: redis, redis-exporter
Server
redis_version:7.0.13 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:ec0335a785106d85 redis_mode:cluster os:Linux 4.18.0-372.49.1.el8_6.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:12.2.1 process_id:9 process_supervised:no run_id:1d63494420bdd7b8d5ea96041373e1fde3e02bbb tcp_port:6379 server_time_usec:1725503766187248 uptime_in_seconds:1107013 uptime_in_days:12 hz:10 configured_hz:10 lru_clock:14227734 executable:/data/redis-server config_file:/etc/config/redis.conf io_threads_active:0
Clients
connected_clients:67 cluster_connections:58 maxclients:100000 client_recent_max_input_buffer:20480 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0
Memory
used_memory:2856093880 used_memory_human:2.66G used_memory_rss:2973200384 used_memory_rss_human:2.77G used_memory_peak:3901671080 used_memory_peak_human:3.63G used_memory_peak_perc:73.20% used_memory_overhead:30079588 used_memory_startup:2126528 used_memory_dataset:2826014292 used_memory_dataset_perc:99.02% allocator_allocated:2857077160 allocator_active:2927587328 allocator_resident:2994311168 total_system_memory:472725020672 total_system_memory_human:440.26G used_memory_lua:55296 used_memory_vm_eval:55296 used_memory_lua_human:54.00K used_memory_scripts_eval:5104 number_of_cached_scripts:10 number_of_functions:0 number_of_libraries:0 used_memory_vm_functions:32768 used_memory_vm_total:88064 used_memory_vm_total_human:86.00K used_memory_functions:184 used_memory_scripts:5288 used_memory_scripts_human:5.16K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.02 allocator_frag_bytes:70510168 allocator_rss_ratio:1.02 allocator_rss_bytes:66723840 rss_overhead_ratio:0.99 rss_overhead_bytes:-21110784 mem_fragmentation_ratio:1.04 mem_fragmentation_bytes:116888800 mem_not_counted_for_evict:0 mem_replication_backlog:1086728 mem_total_replication_buffers:1086712 mem_clients_slaves:0 mem_clients_normal:120848 mem_cluster_links:75380 mem_aof_buffer:0 mem_allocator:jemalloc-5.2.1 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:2588497
Persistence
loading:0 async_loading:0 current_cow_peak:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1725503117 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:12 rdb_current_bgsave_time_sec:-1 rdb_saves:1 rdb_last_cow_size:6389760 rdb_last_load_keys_expired:0 rdb_last_load_keys_loaded:62229 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_rewrites:0 aof_rewrites_consecutive_failures:0 aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:557303 total_commands_processed:7064697530 instantaneous_ops_per_sec:4 total_net_input_bytes:16627495348636 total_net_output_bytes:10525654282808 total_net_repl_input_bytes:15748362650490 total_net_repl_output_bytes:8729836780033 instantaneous_input_kbps:0.15 instantaneous_output_kbps:18.95 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 rejected_connections:0 sync_full:21813 sync_partial_ok:11 sync_partial_err:21802 expired_keys:37821 expired_stale_perc:0.00 expired_time_cap_reached_count:77 expire_cycle_cpu_milliseconds:6132 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:533999674 keyspace_misses:11018535 pubsub_channels:0 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:99601 total_forks:6969 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:6769 dump_payload_sanitizations:0 total_reads_processed:6769648011 total_writes_processed:1351130558 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:1376605 reply_buffer_expands:3348820
Replication
role:slave master_host:19.40.2.11 master_port:6382 master_link_status:down master_last_io_seconds_ago:-1 master_sync_in_progress:0 slave_read_repl_offset:137853614286 slave_repl_offset:137853614286 master_link_down_since_seconds:-1 slave_priority:100 slave_read_only:1 replica_announced:1 connected_slaves:0 master_failover_state:no-failover master_replid:bc742bc367b1e899a0bc6d958817981d538e2ab2 master_replid2:b79f4ced1372529624c2380dd1a2ab51ecc19839 master_repl_offset:137853614286 second_repl_offset:137590816457 repl_backlog_active:1 repl_backlog_size:10485760 repl_backlog_first_byte_offset:137852550834 repl_backlog_histlen:1063453
CPU
used_cpu_sys:138551.080266 used_cpu_user:73692.714765 used_cpu_sys_children:7315.603875 used_cpu_user_children:40185.492418 used_cpu_sys_main_thread:137598.641521 used_cpu_user_main_thread:73493.118948
Modules
Commandstats
cmdstat_hset:calls=2400355,usec=29920406,usec_per_call=12.46,rejected_calls=4,failed_calls=0 cmdstat_cluster|shards:calls=215544,usec=52424092,usec_per_call=243.22,rejected_calls=0,failed_calls=0 cmdstat_cluster|failover:calls=9,usec=37514,usec_per_call=4168.22,rejected_calls=0,failed_calls=0 cmdstat_cluster|forget:calls=245,usec=9826,usec_per_call=40.11,rejected_calls=0,failed_calls=210 cmdstat_cluster|myid:calls=215544,usec=426076,usec_per_call=1.98,rejected_calls=63,failed_calls=0 cmdstat_cluster|nodes:calls=215545,usec=50197102,usec_per_call=232.88,rejected_calls=0,failed_calls=0 cmdstat_cluster|slots:calls=426256,usec=54029034,usec_per_call=126.75,rejected_calls=37,failed_calls=0 cmdstat_cluster|info:calls=436925,usec=50562934,usec_per_call=115.72,rejected_calls=18,failed_calls=0 cmdstat_cluster|meet:calls=81,usec=2005,usec_per_call=24.75,rejected_calls=0,failed_calls=0 cmdstat_cluster|replicate:calls=56253,usec=25809371,usec_per_call=458.81,rejected_calls=0,failed_calls=0 cmdstat_bgsave:calls=1,usec=99670,usec_per_call=99670.00,rejected_calls=0,failed_calls=0 cmdstat_client|id:calls=52,usec=88,usec_per_call=1.69,rejected_calls=0,failed_calls=0 cmdstat_client|setname:calls=221398,usec=1081224,usec_per_call=4.88,rejected_calls=0,failed_calls=0 cmdstat_incrby:calls=987818,usec=6967014,usec_per_call=7.05,rejected_calls=6,failed_calls=0 cmdstat_xreadgroup:calls=2738692,usec=22617363,usec_per_call=8.26,rejected_calls=13,failed_calls=0 cmdstat_zrem:calls=3734893,usec=68874708,usec_per_call=18.44,rejected_calls=3,failed_calls=0 cmdstat_config|set:calls=107071,usec=2381927,usec_per_call=22.25,rejected_calls=0,failed_calls=0 cmdstat_config|get:calls=3014074,usec=132851617,usec_per_call=44.08,rejected_calls=0,failed_calls=0 cmdstat_ttl:calls=801180,usec=2444469,usec_per_call=3.05,rejected_calls=0,failed_calls=0 cmdstat_hello:calls=18846,usec=164323,usec_per_call=8.72,rejected_calls=0,failed_calls=0 cmdstat_hmget:calls=202505791,usec=3112914669,usec_per_call=15.37,rejected_calls=634,failed_calls=0 cmdstat_psync:calls=21824,usec=743268,usec_per_call=34.06,rejected_calls=0,failed_calls=0 cmdstat_zadd:calls=11859808,usec=221262216,usec_per_call=18.66,rejected_calls=58,failed_calls=0 cmdstat_subscribe:calls=9,usec=192,usec_per_call=21.33,rejected_calls=0,failed_calls=0 cmdstat_hgetall:calls=2040,usec=12052,usec_per_call=5.91,rejected_calls=0,failed_calls=0 cmdstat_xack:calls=211818,usec=780923,usec_per_call=3.69,rejected_calls=1,failed_calls=0 cmdstat_hkeys:calls=48,usec=296,usec_per_call=6.17,rejected_calls=8,failed_calls=0 cmdstat_xclaim:calls=202816,usec=2435111,usec_per_call=12.01,rejected_calls=0,failed_calls=0 cmdstat_dbsize:calls=1,usec=2,usec_per_call=2.00,rejected_calls=0,failed_calls=0 cmdstat_zrange:calls=1610324,usec=24128644,usec_per_call=14.98,rejected_calls=8,failed_calls=0 cmdstat_replconf:calls=220930,usec=194237,usec_per_call=0.88,rejected_calls=0,failed_calls=0 cmdstat_hdel:calls=1116588,usec=12344094,usec_per_call=11.06,rejected_calls=9,failed_calls=0 cmdstat_scan:calls=16577,usec=97845864,usec_per_call=5902.51,rejected_calls=0,failed_calls=0 cmdstat_getset:calls=785,usec=11640,usec_per_call=14.83,rejected_calls=2481,failed_calls=0 cmdstat_info:calls=424013,usec=379869903,usec_per_call=895.89,rejected_calls=0,failed_calls=0 cmdstat_setex:calls=264,usec=4945,usec_per_call=18.73,rejected_calls=0,failed_calls=0 cmdstat_xrange:calls=1,usec=10,usec_per_call=10.00,rejected_calls=0,failed_calls=0 cmdstat_smembers:calls=552,usec=6589,usec_per_call=11.94,rejected_calls=2,failed_calls=0 cmdstat_latency|latest:calls=221398,usec=1628947,usec_per_call=7.36,rejected_calls=0,failed_calls=0 cmdstat_evalsha:calls=710439,usec=16795118,usec_per_call=23.64,rejected_calls=40,failed_calls=14 cmdstat_hget:calls=8821263,usec=33126643,usec_per_call=3.76,rejected_calls=27,failed_calls=0 cmdstat_del:calls=8241038,usec=133535903,usec_per_call=16.20,rejected_calls=148,failed_calls=0 cmdstat_xpending:calls=139582,usec=757383,usec_per_call=5.43,rejected_calls=7,failed_calls=0 cmdstat_slowlog|get:calls=221398,usec=3228726,usec_per_call=14.58,rejected_calls=0,failed_calls=0 cmdstat_slowlog|len:calls=221398,usec=377604,usec_per_call=1.71,rejected_calls=0,failed_calls=0 cmdstat_expire:calls=734953,usec=3642865,usec_per_call=4.96,rejected_calls=0,failed_calls=0 cmdstat_srem:calls=2963121,usec=26898311,usec_per_call=9.08,rejected_calls=33,failed_calls=0 cmdstat_zscore:calls=4787,usec=11914,usec_per_call=2.49,rejected_calls=0,failed_calls=0 cmdstat_set:calls=4039893856,usec=19734751435,usec_per_call=4.88,rejected_calls=3,failed_calls=0 cmdstat_get:calls=162168185,usec=289964918,usec_per_call=1.79,rejected_calls=586,failed_calls=0 cmdstat_xgroup|create:calls=1,usec=16,usec_per_call=16.00,rejected_calls=3,failed_calls=0 cmdstat_xgroup|delconsumer:calls=14,usec=373,usec_per_call=26.64,rejected_calls=15,failed_calls=0 cmdstat_zrangebyscore:calls=8274123,usec=37653839,usec_per_call=4.55,rejected_calls=86,failed_calls=0 cmdstat_zcard:calls=3547031,usec=8108203,usec_per_call=2.29,rejected_calls=26,failed_calls=0 cmdstat_decrby:calls=927160,usec=6522775,usec_per_call=7.04,rejected_calls=0,failed_calls=0 cmdstat_pexpireat:calls=1198553,usec=6612278,usec_per_call=5.52,rejected_calls=0,failed_calls=0 cmdstat_select:calls=32945,usec=52790,usec_per_call=1.60,rejected_calls=0,failed_calls=0 cmdstat_exists:calls=149470041,usec=146271681,usec_per_call=0.98,rejected_calls=0,failed_calls=0 cmdstat_zpopmin:calls=200617,usec=3975344,usec_per_call=19.82,rejected_calls=1,failed_calls=0 cmdstat_eval:calls=301233184,usec=3386422946,usec_per_call=11.24,rejected_calls=1260,failed_calls=0 cmdstat_xadd:calls=1657847,usec=26300242,usec_per_call=15.86,rejected_calls=0,failed_calls=0 cmdstat_ping:calls=96229,usec=58981,usec_per_call=0.61,rejected_calls=0,failed_calls=0 cmdstat_sadd:calls=3269682,usec=29461286,usec_per_call=9.01,rejected_calls=2,failed_calls=0 cmdstat_xdel:calls=211818,usec=2717826,usec_per_call=12.83,rejected_calls=0,failed_calls=0 cmdstat_hmset:calls=2129980565,usec=32959913011,usec_per_call=15.47,rejected_calls=193,failed_calls=0 cmdstat_type:calls=272,usec=865,usec_per_call=3.18,rejected_calls=0,failed_calls=0 cmdstat_xread:calls=889897,usec=8718160,usec_per_call=9.80,rejected_calls=163,failed_calls=0 cmdstat_command:calls=156,usec=661529,usec_per_call=4240.57,rejected_calls=0,failed_calls=0 cmdstat_command|docs:calls=5,usec=22158,usec_per_call=4431.60,rejected_calls=0,failed_calls=0 cmdstat_multi:calls=2636871,usec=682874,usec_per_call=0.26,rejected_calls=0,failed_calls=0 cmdstat_xinfo|stream:calls=392,usec=6995,usec_per_call=17.84,rejected_calls=1,failed_calls=120 cmdstat_exec:calls=2636871,usec=72692518,usec_per_call=27.57,rejected_calls=0,failed_calls=0 cmdstat_unlink:calls=91263,usec=1369385,usec_per_call=15.00,rejected_calls=0,failed_calls=0 cmdstat_role:calls=215604,usec=864421,usec_per_call=4.01,rejected_calls=0,failed_calls=0
Errorstats
errorstat_CLUSTERDOWN:count=2421 errorstat_ERR:count=344 errorstat_LOADING:count=118 errorstat_MOVED:count=3400 errorstat_NOSCRIPT:count=2 errorstat_UNBLOCKED:count=484
Latencystats
latency_percentiles_usec_hset:p50=11.007,p99=42.239,p99.9=89.087 latency_percentiles_usec_cluster|shards:p50=238.591,p99=477.183,p99.9=770.047 latency_percentiles_usec_cluster|failover:p50=3063.807,p99=15466.495,p99.9=15466.495 latency_percentiles_usec_cluster|forget:p50=6.015,p99=1589.247,p99.9=2277.375 latency_percentiles_usec_cluster|myid:p50=2.007,p99=5.023,p99.9=27.007 latency_percentiles_usec_cluster|nodes:p50=218.111,p99=532.479,p99.9=1032.191 latency_percentiles_usec_cluster|slots:p50=122.367,p99=276.479,p99.9=481.279 latency_percentiles_usec_cluster|info:p50=112.127,p99=260.095,p99.9=438.271 latency_percentiles_usec_cluster|meet:p50=20.095,p99=60.159,p99.9=134.143 latency_percentiles_usec_cluster|replicate:p50=442.367,p99=888.831,p99.9=2228.223 latency_percentiles_usec_bgsave:p50=100139.007,p99=100139.007,p99.9=100139.007 latency_percentiles_usec_client|id:p50=1.003,p99=2.007,p99.9=17.023 latency_percentiles_usec_client|setname:p50=4.015,p99=21.119,p99.9=52.223 latency_percentiles_usec_incrby:p50=6.015,p99=23.039,p99.9=53.247 latency_percentiles_usec_xreadgroup:p50=31.103,p99=166.911,p99.9=286.719 latency_percentiles_usec_zrem:p50=16.063,p99=59.135,p99.9=130.047 latency_percentiles_usec_config|set:p50=20.095,p99=75.263,p99.9=138.239 latency_percentiles_usec_config|get:p50=5.023,p99=679.935,p99.9=966.655 latency_percentiles_usec_ttl:p50=2.007,p99=14.015,p99.9=34.047 latency_percentiles_usec_hello:p50=8.031,p99=28.031,p99.9=79.359 latency_percentiles_usec_hmget:p50=13.055,p99=54.015,p99.9=111.103 latency_percentiles_usec_psync:p50=29.055,p99=93.183,p99.9=172.031 latency_percentiles_usec_zadd:p50=14.015,p99=56.063,p99.9=128.511 latency_percentiles_usec_subscribe:p50=23.039,p99=41.215,p99.9=41.215 latency_percentiles_usec_hgetall:p50=4.015,p99=29.055,p99.9=54.015 latency_percentiles_usec_xack:p50=3.007,p99=16.063,p99.9=42.239 latency_percentiles_usec_hkeys:p50=4.015,p99=24.063,p99.9=24.063 latency_percentiles_usec_xclaim:p50=9.023,p99=45.055,p99.9=92.159 latency_percentiles_usec_dbsize:p50=2.007,p99=2.007,p99.9=2.007 latency_percentiles_usec_zrange:p50=11.007,p99=61.183,p99.9=106.495 latency_percentiles_usec_replconf:p50=1.003,p99=3.007,p99.9=15.039 latency_percentiles_usec_hdel:p50=10.047,p99=34.047,p99.9=73.215 latency_percentiles_usec_scan:p50=5734.399,p99=13303.807,p99.9=23986.175 latency_percentiles_usec_getset:p50=14.015,p99=41.215,p99.9=79.359 latency_percentiles_usec_info:p50=745.471,p99=3063.807,p99.9=4112.383 latency_percentiles_usec_setex:p50=14.015,p99=82.431,p99.9=155.647 latency_percentiles_usec_xrange:p50=10.047,p99=10.047,p99.9=10.047 latency_percentiles_usec_smembers:p50=8.031,p99=59.135,p99.9=104.447 latency_percentiles_usec_latency|latest:p50=6.015,p99=28.031,p99.9=75.263 latency_percentiles_usec_evalsha:p50=16.063,p99=95.231,p99.9=227.327 latency_percentiles_usec_hget:p50=3.007,p99=16.063,p99.9=36.095 latency_percentiles_usec_del:p50=10.047,p99=71.167,p99.9=124.415 latency_percentiles_usec_xpending:p50=4.015,p99=19.071,p99.9=52.223 latency_percentiles_usec_slowlog|get:p50=12.031,p99=53.247,p99.9=99.327 latency_percentiles_usec_slowlog|len:p50=2.007,p99=4.015,p99.9=25.087 latency_percentiles_usec_expire:p50=4.015,p99=19.071,p99.9=48.127 latency_percentiles_usec_srem:p50=8.031,p99=31.103,p99.9=65.023 latency_percentiles_usec_zscore:p50=2.007,p99=10.047,p99.9=25.087 latency_percentiles_usec_set:p50=4.015,p99=18.047,p99.9=44.031 latency_percentiles_usec_get:p50=1.003,p99=10.047,p99.9=24.063 latency_percentiles_usec_xgroup|create:p50=16.063,p99=16.063,p99.9=16.063 latency_percentiles_usec_xgroup|delconsumer:p50=24.063,p99=57.087,p99.9=57.087 latency_percentiles_usec_zrangebyscore:p50=4.015,p99=17.023,p99.9=47.103 latency_percentiles_usec_zcard:p50=2.007,p99=10.047,p99.9=25.087 latency_percentiles_usec_decrby:p50=6.015,p99=23.039,p99.9=53.247 latency_percentiles_usec_pexpireat:p50=5.023,p99=21.119,p99.9=47.103 latency_percentiles_usec_select:p50=1.003,p99=3.007,p99.9=35.071 latency_percentiles_usec_exists:p50=1.003,p99=3.007,p99.9=15.039 latency_percentiles_usec_zpopmin:p50=18.047,p99=58.111,p99.9=120.319 latency_percentiles_usec_eval:p50=9.023,p99=47.103,p99.9=116.223 latency_percentiles_usec_xadd:p50=14.015,p99=51.199,p99.9=103.423 latency_percentiles_usec_ping:p50=0.001,p99=6.015,p99.9=16.063 latency_percentiles_usec_sadd:p50=8.031,p99=31.103,p99.9=68.095 latency_percentiles_usec_xdel:p50=11.007,p99=43.007,p99.9=89.087 latency_percentiles_usec_hmset:p50=12.031,p99=60.159,p99.9=112.127 latency_percentiles_usec_type:p50=2.007,p99=18.047,p99.9=33.023 latency_percentiles_usec_xread:p50=6.015,p99=35.071,p99.9=74.239 latency_percentiles_usec_command:p50=4980.735,p99=6946.815,p99.9=7995.391 latency_percentiles_usec_command|docs:p50=5505.023,p99=5931.007,p99.9=5931.007 latency_percentiles_usec_multi:p50=0.001,p99=1.003,p99.9=5.023 latency_percentiles_usec_xinfo|stream:p50=14.015,p99=60.159,p99.9=79.359 latency_percentiles_usec_exec:p50=24.063,p99=89.087,p99.9=360.447 latency_percentiles_usec_unlink:p50=13.055,p99=44.031,p99.9=94.207 latency_percentiles_usec_role:p50=3.007,p99=17.023,p99.9=42.239
Cluster
cluster_enabled:1
Keyspace
db0:keys=394857,expires=9477,avg_ttl=75291831
Comment From: AlenChenGitHub
@sundb I used the following configuration to restore data replication between the primary and secondary nodes. repl-diskless-sync no
Comment From: sundb
@AlenChenGitHub did you ever do failover?
Comment From: AlenChenGitHub
@sundb Yes, before this situation occurs, a failover has been performed, and this situation occurs on some nodes of the standby node. According to some information, the disk memory may not be enough to support the generation of rdb files.
Comment From: sundb
@sundb Yes, before this situation occurs, a failover has been performed, and this situation occurs on some nodes of the standby node. According to some information, the disk memory may not be enough to support the generation of rdb files.
do you mean disk space?
Comment From: AlenChenGitHub
@sundb Yes, before this situation occurs, a failover has been performed, and this situation occurs on some nodes of the standby node. According to some information, the disk memory may not be enough to support the generation of rdb files.
do you mean disk space?
I think so, but I'm not sure. Can I draw that conclusion from this picture?
![Redis [QUESTION]In redis cluster mode, when there is a large amount of data, the slave node repeatedly reports the problem of full sync, and the info replication displays down. May I ask from what aspects should I consider the solution? Improve the cpu and memory of redis nodes?](https://www.nineya.com/upload/2025/1/0300173962739244)
Comment From: AlenChenGitHub
@sundb In a large number of cases, I configured repl-diskless-sync no. Is that appropriate? If both clusters crash, but there is no RDB file, will the data become unrecoverable?
Comment From: sundb
used_memory_peak_perc is
@sundb Yes, before this situation occurs, a failover has been performed, and this situation occurs on some nodes of the standby node. According to some information, the disk memory may not be enough to support the generation of rdb files.
do you mean disk space?
I think so, but I'm not sure. Can I draw that conclusion from this picture?
no, it's RAM, not HDD. in this case, you should enable rep-diskless-sync, otherwise the fork may fail and the full synchronization can't be performed.
If both clusters crash, but there is no RDB file, will the data become unrecoverable?
yes, unless you're still turn on aof.
Comment From: AlenChenGitHub
used_memory_peak_percis@sundb Yes, before this situation occurs, a failover has been performed, and this situation occurs on some nodes of the standby node. According to some information, the disk memory may not be enough to support the generation of rdb files.
do you mean disk space?
I think so, but I'm not sure. Can I draw that conclusion from this picture?
no, it's RAM, not HDD. in this case, you should enable rep-diskless-sync, otherwise the fork may fail and the full synchronization can't be performed.
If both clusters crash, but there is no RDB file, will the data become unrecoverable?
yes, unless you're still turn on aof.
Thank you very much for your reply. I greatly appreciate it!
Comment From: sundb
@AlenChenGitHub can you share the logs from master?
Comment From: AlenChenGitHub
@AlenChenGitHub can you share the logs from master?
2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # Redis version=7.0.13, bits=64, commit=00000000, modified=0, pid=8, just started 2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # Configuration loaded 2024-09-04T05:29:06.252870054Z 8:M 04 Sep 2024 05:29:06.252 * monotonic clock: POSIX clock_gettime 2024-09-04T05:29:06.253247239Z 8:M 04 Sep 2024 05:29:06.253 * No cluster configuration found, I'm 86926eddfaf2b8c9bf08a4a014bb7fef722384a7 2024-09-04T05:29:06.254068394Z 8:M 04 Sep 2024 05:29:06.254 * Running mode=cluster, port=6379. 2024-09-04T05:29:06.254111751Z 8:M 04 Sep 2024 05:29:06.254 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 2024-09-04T05:29:06.254124008Z 8:M 04 Sep 2024 05:29:06.254 # Server initialized 2024-09-04T05:29:06.255313917Z 8:M 04 Sep 2024 05:29:06.255 * Ready to accept connections 2024-09-04T05:29:06.347872482Z 8:M 04 Sep 2024 05:29:06.347 # IP address for this node updated to 10.128.1.12 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Full resync requested by replica 18.40.2.16:6383 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Replication backlog created, my new replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000' 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.377527128Z 8:M 04 Sep 2024 05:29:06.377 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:29:06.377534878Z 8:M 04 Sep 2024 05:29:06.377 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '5ddc4b5a47e1ca96e0f4e6727a996e0d3d9a9657', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:06.377539654Z 8:M 04 Sep 2024 05:29:06.377 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Replica 18.40.2.16:6384 asks for synchronization 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Full resync requested by replica 18.40.2.16:6384 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Replica 19.40.2.11:6383 asks for synchronization 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'fe2d83c0173365ffd8dc29fb072cc95663bd46e3', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:07.110273067Z 8:M 04 Sep 2024 05:29:07.110 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '5ddc4b5a47e1ca96e0f4e6727a996e0d3d9a9657', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:08.524320648Z 8:M 04 Sep 2024 05:29:08.524 # Cluster state changed: ok 2024-09-04T05:29:13.429293296Z 8:M 04 Sep 2024 05:29:13.429 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:29:13.546692677Z 8:M 04 Sep 2024 05:29:13.546 # Connection with replica 18.40.2.16:6384 lost. 2024-09-04T05:29:14.053199805Z 8:M 04 Sep 2024 05:29:14.053 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:29:14.056677916Z 8:M 04 Sep 2024 05:29:14.056 * Background RDB transfer started by pid 13 2024-09-04T05:29:14.080286211Z 13:C 04 Sep 2024 05:29:14.080 * Fork CoW for RDB: current 6 MB, peak 6 MB, average 6 MB 2024-09-04T05:29:14.258013068Z 8:M 04 Sep 2024 05:29:14.257 # Diskless rdb transfer, done reading from pipe, 2 replicas still up. 2024-09-04T05:29:14.268410574Z 8:M 04 Sep 2024 05:29:14.268 * Background RDB transfer terminated with success 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Streamed RDB transfer with replica 19.40.2.11:6383 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Synchronization with replica 19.40.2.11:6383 succeeded 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Streamed RDB transfer with replica 18.40.2.16:6382 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:29:14.268441216Z 8:M 04 Sep 2024 05:29:14.268 * Synchronization with replica 18.40.2.16:6382 succeeded 2024-09-04T05:29:32.819602599Z 8:M 04 Sep 2024 05:29:32.819 * Marking node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30 as failing (quorum reached). 2024-09-04T05:29:38.738636690Z 8:M 04 Sep 2024 05:29:38.738 # Address updated for node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30, now 19.40.2.11:6391 2024-09-04T05:29:38.867427487Z 8:M 04 Sep 2024 05:29:38.867 * Clear FAIL state for node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30: is reachable again and nobody is serving its slots after some time. 2024-09-04T05:37:39.493554652Z 8:M 04 Sep 2024 05:37:39.493 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:37:39.498579541Z 8:M 04 Sep 2024 05:37:39.498 # Connection with replica 19.40.2.11:6383 lost. 2024-09-04T05:37:39.499244132Z 8:M 04 Sep 2024 05:37:39.499 # Configuration change detected. Reconfiguring myself as a replica of 1bea42d01e08ea767239eee03ab8777963828a1e 2024-09-04T05:37:39.499666983Z 8:S 04 Sep 2024 05:37:39.499 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 2024-09-04T05:37:39.499673383Z 8:S 04 Sep 2024 05:37:39.499 * Connecting to MASTER 18.40.2.16:6382 2024-09-04T05:37:39.499806939Z 8:S 04 Sep 2024 05:37:39.499 * MASTER <-> REPLICA sync started 2024-09-04T05:37:39.502002151Z 8:S 04 Sep 2024 05:37:39.501 * Non blocking connect for SYNC fired the event. 2024-09-04T05:37:39.502262152Z 8:S 04 Sep 2024 05:37:39.502 * Master replied to PING, replication can continue... 2024-09-04T05:37:39.502530027Z 8:S 04 Sep 2024 05:37:39.502 * Trying a partial resynchronization (request 23f9944fb8633643f348c43a803047128a90a8cc:627113695). 2024-09-04T05:37:39.502726549Z 8:S 04 Sep 2024 05:37:39.502 * Successful partial resynchronization with master. 2024-09-04T05:37:39.502731281Z 8:S 04 Sep 2024 05:37:39.502 # Master replication ID changed to 2bbb3e0d236184caf578d5a8eb422befe94553ad 2024-09-04T05:37:39.502741464Z 8:S 04 Sep 2024 05:37:39.502 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization. 2024-09-04T05:51:51.870298204Z 8:S 04 Sep 2024 05:51:51.870 # Taking over the master (user request). 2024-09-04T05:51:51.870298204Z 8:S 04 Sep 2024 05:51:51.870 # New configEpoch set to 134 2024-09-04T05:51:51.870298204Z 8:M 04 Sep 2024 05:51:51.870 # Connection with master lost. 2024-09-04T05:51:51.870298204Z 8:M 04 Sep 2024 05:51:51.870 * Caching the disconnected master state. 2024-09-04T05:51:51.870346814Z 8:M 04 Sep 2024 05:51:51.870 * Discarding previously cached master state. 2024-09-04T05:51:51.870346814Z 8:M 04 Sep 2024 05:51:51.870 # Setting secondary replication ID to 2bbb3e0d236184caf578d5a8eb422befe94553ad, valid up to offset: 629591655. New replication ID is b79f4ced1372529624c2380dd1a2ab51ecc19839 2024-09-04T05:51:51.872938649Z 8:M 04 Sep 2024 05:51:51.872 * Replica 19.40.2.11:6384 asks for synchronization 2024-09-04T05:51:51.872975372Z 8:M 04 Sep 2024 05:51:51.872 * Partial resynchronization request from 19.40.2.11:6384 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.873234499Z 8:M 04 Sep 2024 05:51:51.873 * Replica 19.40.2.11:6383 asks for synchronization 2024-09-04T05:51:51.873249682Z 8:M 04 Sep 2024 05:51:51.873 * Partial resynchronization request from 19.40.2.11:6383 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.873554307Z 8:M 04 Sep 2024 05:51:51.873 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:51.873559073Z 8:M 04 Sep 2024 05:51:51.873 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:51.873563419Z 8:M 04 Sep 2024 05:51:51.873 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:51.876507853Z 8:M 04 Sep 2024 05:51:51.876 * Replica 18.40.2.16:6384 asks for synchronization 2024-09-04T05:51:51.876525518Z 8:M 04 Sep 2024 05:51:51.876 * Partial resynchronization request from 18.40.2.16:6384 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.880682694Z 8:M 04 Sep 2024 05:51:51.880 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:51.880689927Z 8:M 04 Sep 2024 05:51:51.880 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:51.880689927Z 8:M 04 Sep 2024 05:51:51.880 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:52.040739440Z 8:M 04 Sep 2024 05:51:52.040 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:51:52.042318876Z 8:M 04 Sep 2024 05:51:52.042 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:52.042325684Z 8:M 04 Sep 2024 05:51:52.042 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:52.042330671Z 8:M 04 Sep 2024 05:51:52.042 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:52.061688414Z 8:M 04 Sep 2024 05:51:52.061 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:51:52.062193838Z 8:M 04 Sep 2024 05:51:52.062 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:52.062200042Z 8:M 04 Sep 2024 05:51:52.062 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:52.062204821Z 8:M 04 Sep 2024 05:51:52.062 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:55.318614097Z 8:M 04 Sep 2024 05:51:55.318 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:51:55.319445641Z 8:M 04 Sep 2024 05:51:55.319 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:55.319453377Z 8:M 04 Sep 2024 05:51:55.319 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:55.319458352Z 8:M 04 Sep 2024 05:51:55.319 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:57.171963040Z 8:M 04 Sep 2024 05:51:57.171 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:51:57.173282296Z 8:M 04 Sep 2024 05:51:57.173 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:57.173289640Z 8:M 04 Sep 2024 05:51:57.173 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:57.173294424Z 8:M 04 Sep 2024 05:51:57.173 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:00.484081150Z 8:M 04 Sep 2024 05:52:00.484 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:00.516370409Z 8:M 04 Sep 2024 05:52:00.516 * Background RDB transfer started by pid 14 2024-09-04T05:52:02.061538026Z 8:M 04 Sep 2024 05:52:02.061 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:02.062617218Z 8:M 04 Sep 2024 05:52:02.062 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:02.062624389Z 8:M 04 Sep 2024 05:52:02.062 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:02.062629594Z 8:M 04 Sep 2024 05:52:02.062 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:02.079198038Z 8:M 04 Sep 2024 05:52:02.079 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:02.079226845Z 8:M 04 Sep 2024 05:52:02.079 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:02.079241246Z 14:signal-handler (1725429122) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:02.079953772Z 8:M 04 Sep 2024 05:52:02.079 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:02.079959658Z 8:M 04 Sep 2024 05:52:02.079 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:02.079964258Z 8:M 04 Sep 2024 05:52:02.079 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:02.124153532Z 8:M 04 Sep 2024 05:52:02.124 # Background transfer terminated by signal 10 2024-09-04T05:52:06.330690463Z 8:M 04 Sep 2024 05:52:06.330 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:06.332202302Z 8:M 04 Sep 2024 05:52:06.332 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:06.332210111Z 8:M 04 Sep 2024 05:52:06.332 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:06.332210111Z 8:M 04 Sep 2024 05:52:06.332 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:07.432689833Z 8:M 04 Sep 2024 05:52:07.432 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:07.434066537Z 8:M 04 Sep 2024 05:52:07.434 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:07.434073495Z 8:M 04 Sep 2024 05:52:07.434 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:07.434078292Z 8:M 04 Sep 2024 05:52:07.434 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:11.585986582Z 8:M 04 Sep 2024 05:52:11.585 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:11.613714462Z 8:M 04 Sep 2024 05:52:11.613 * Background RDB transfer started by pid 15 2024-09-04T05:52:12.081007928Z 8:M 04 Sep 2024 05:52:12.080 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:12.082203135Z 8:M 04 Sep 2024 05:52:12.082 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:12.082216921Z 8:M 04 Sep 2024 05:52:12.082 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:12.082216921Z 8:M 04 Sep 2024 05:52:12.082 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:12.097930229Z 8:M 04 Sep 2024 05:52:12.097 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:12.098036681Z 8:M 04 Sep 2024 05:52:12.098 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:12.098048311Z 15:signal-handler (1725429132) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:12.098760195Z 8:M 04 Sep 2024 05:52:12.098 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:12.098765335Z 8:M 04 Sep 2024 05:52:12.098 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:12.098769880Z 8:M 04 Sep 2024 05:52:12.098 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:12.217055146Z 8:M 04 Sep 2024 05:52:12.217 # Background transfer terminated by signal 10 2024-09-04T05:52:16.462264461Z 8:M 04 Sep 2024 05:52:16.462 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:16.463187387Z 8:M 04 Sep 2024 05:52:16.463 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:16.463195067Z 8:M 04 Sep 2024 05:52:16.463 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:16.463200027Z 8:M 04 Sep 2024 05:52:16.463 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:17.656697604Z 8:M 04 Sep 2024 05:52:17.656 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:17.683470711Z 8:M 04 Sep 2024 05:52:17.683 * Background RDB transfer started by pid 16 2024-09-04T05:52:17.684135713Z 8:M 04 Sep 2024 05:52:17.684 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:17.685366184Z 8:M 04 Sep 2024 05:52:17.685 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:17.685372021Z 8:M 04 Sep 2024 05:52:17.685 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:17.685376901Z 8:M 04 Sep 2024 05:52:17.685 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.099579147Z 8:M 04 Sep 2024 05:52:22.099 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:22.101294573Z 8:M 04 Sep 2024 05:52:22.101 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:22.101301725Z 8:M 04 Sep 2024 05:52:22.101 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:22.101306436Z 8:M 04 Sep 2024 05:52:22.101 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.114737205Z 8:M 04 Sep 2024 05:52:22.114 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:22.114968948Z 8:M 04 Sep 2024 05:52:22.114 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:22.114975771Z 16:signal-handler (1725429142) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:22.115644538Z 8:M 04 Sep 2024 05:52:22.115 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:22.115649480Z 8:M 04 Sep 2024 05:52:22.115 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:22.115654209Z 8:M 04 Sep 2024 05:52:22.115 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.198642344Z 8:M 04 Sep 2024 05:52:22.198 # Background transfer terminated by signal 10 2024-09-04T05:52:26.503687889Z 8:M 04 Sep 2024 05:52:26.503 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:26.504170313Z 8:M 04 Sep 2024 05:52:26.504 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:26.504177291Z 8:M 04 Sep 2024 05:52:26.504 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:26.504182057Z 8:M 04 Sep 2024 05:52:26.504 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:27.717143083Z 8:M 04 Sep 2024 05:52:27.717 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:27.718440897Z 8:M 04 Sep 2024 05:52:27.718 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:27.718450469Z 8:M 04 Sep 2024 05:52:27.718 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:27.718450469Z 8:M 04 Sep 2024 05:52:27.718 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:31.769645231Z 8:M 04 Sep 2024 05:52:31.769 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:31.795978382Z 8:M 04 Sep 2024 05:52:31.795 * Background RDB transfer started by pid 17 2024-09-04T05:52:32.120289617Z 8:M 04 Sep 2024 05:52:32.120 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:32.121800456Z 8:M 04 Sep 2024 05:52:32.121 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:32.121807464Z 8:M 04 Sep 2024 05:52:32.121 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:32.121812555Z 8:M 04 Sep 2024 05:52:32.121 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:32.134125348Z 8:M 04 Sep 2024 05:52:32.134 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:32.134197407Z 8:M 04 Sep 2024 05:52:32.134 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:32.134214929Z 17:signal-handler (1725429152) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:32.135162596Z 8:M 04 Sep 2024 05:52:32.135 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:32.135168818Z 8:M 04 Sep 2024 05:52:32.135 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:32.135173484Z 8:M 04 Sep 2024 05:52:32.135 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:32.198874129Z 8:M 04 Sep 2024 05:52:32.198 # Background transfer terminated by signal 10 2024-09-04T05:52:36.541623930Z 8:M 04 Sep 2024 05:52:36.541 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:36.542231754Z 8:M 04 Sep 2024 05:52:36.542 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:36.542238721Z 8:M 04 Sep 2024 05:52:36.542 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:36.542243381Z 8:M 04 Sep 2024 05:52:36.542 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:37.762891304Z 8:M 04 Sep 2024 05:52:37.762 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:37.764631830Z 8:M 04 Sep 2024 05:52:37.764 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:37.764655361Z 8:M 04 Sep 2024 05:52:37.764 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:37.764671492Z 8:M 04 Sep 2024 05:52:37.764 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:41.867372471Z 8:M 04 Sep 2024 05:52:41.867 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:41.893296744Z 8:M 04 Sep 2024 05:52:41.893 * Background RDB transfer started by pid 18 2024-09-04T05:52:42.141380326Z 8:M 04 Sep 2024 05:52:42.141 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:42.142328056Z 8:M 04 Sep 2024 05:52:42.142 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:42.142335304Z 8:M 04 Sep 2024 05:52:42.142 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:42.142340150Z 8:M 04 Sep 2024 05:52:42.142 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:42.156539396Z 8:M 04 Sep 2024 05:52:42.156 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:42.156598566Z 8:M 04 Sep 2024 05:52:42.156 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:42.156616111Z 18:signal-handler (1725429162) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:42.157419070Z 8:M 04 Sep 2024 05:52:42.157 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:42.157425755Z 8:M 04 Sep 2024 05:52:42.157 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:42.157430602Z 8:M 04 Sep 2024 05:52:42.157 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:42.196165975Z 8:M 04 Sep 2024 05:52:42.196 # Background transfer terminated by signal 10 2024-09-04T05:52:46.585451159Z 8:M 04 Sep 2024 05:52:46.585 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:46.585986221Z 8:M 04 Sep 2024 05:52:46.585 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:46.585994940Z 8:M 04 Sep 2024 05:52:46.585 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:46.585994940Z 8:M 04 Sep 2024 05:52:46.585 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:47.802989676Z 8:M 04 Sep 2024 05:52:47.802 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:47.804299836Z 8:M 04 Sep 2024 05:52:47.804 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:47.804307810Z 8:M 04 Sep 2024 05:52:47.804 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:47.804307810Z 8:M 04 Sep 2024 05:52:47.804 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:51.965330270Z 8:M 04 Sep 2024 05:52:51.965 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:51.991870371Z 8:M 04 Sep 2024 05:52:51.991 * Background RDB transfer started by pid 19 2024-09-04T05:52:52.162715852Z 8:M 04 Sep 2024 05:52:52.162 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:52.163942196Z 8:M 04 Sep 2024 05:52:52.163 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:52.163949690Z 8:M 04 Sep 2024 05:52:52.163 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:52.163954653Z 8:M 04 Sep 2024 05:52:52.163 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:52.177492702Z 8:M 04 Sep 2024 05:52:52.177 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:52.177518258Z 8:M 04 Sep 2024 05:52:52.177 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:52.177524830Z 19:signal-handler (1725429172) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:52.178197346Z 8:M 04 Sep 2024 05:52:52.178 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:52.178204361Z 8:M 04 Sep 2024 05:52:52.178 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:52.178209214Z 8:M 04 Sep 2024 05:52:52.178 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:52.295236394Z 8:M 04 Sep 2024 05:52:52.295 # Background transfer terminated by signal 10 2024-09-04T05:52:56.629547710Z 8:M 04 Sep 2024 05:52:56.629 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:56.630332936Z 8:M 04 Sep 2024 05:52:56.630 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:56.630355927Z 8:M 04 Sep 2024 05:52:56.630 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:56.630371647Z 8:M 04 Sep 2024 05:52:56.630 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:56.726374145Z 8:M 04 Sep 2024 05:52:56.726 # Failover auth granted to 9279b0187ca03dd29b5348685780b25796539f90 for epoch 138 2024-09-04T05:52:57.032276763Z 8:M 04 Sep 2024 05:52:57.032 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:57.058642104Z 8:M 04 Sep 2024 05:52:57.058 * Background RDB transfer started by pid 20 2024-09-04T05:52:57.934849476Z 8:M 04 Sep 2024 05:52:57.934 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:57.936380404Z 8:M 04 Sep 2024 05:52:57.936 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:57.936387463Z 8:M 04 Sep 2024 05:52:57.936 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:57.936392066Z 8:M 04 Sep 2024 05:52:57.936 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:01.782689786Z 20:C 04 Sep 2024 05:53:01.782 * Fork CoW for RDB: current 5 MB, peak 5 MB, average 4 MB 2024-09-04T05:53:01.782689786Z 8:M 04 Sep 2024 05:53:01.782 # Diskless rdb transfer, done reading from pipe, 1 replicas still up. 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Background RDB transfer terminated with success 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Streamed RDB transfer with replica 18.40.2.16:6383 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Synchronization with replica 18.40.2.16:6383 succeeded 2024-09-04T05:53:02.075879219Z 8:M 04 Sep 2024 05:53:02.075 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:02.101839168Z 8:M 04 Sep 2024 05:53:02.101 * Background RDB transfer started by pid 21 2024-09-04T05:53:02.184402119Z 8:M 04 Sep 2024 05:53:02.184 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:02.184478073Z 8:M 04 Sep 2024 05:53:02.184 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:02.184485044Z 21:signal-handler (1725429182) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:02.185803735Z 8:M 04 Sep 2024 05:53:02.185 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:02.185810553Z 8:M 04 Sep 2024 05:53:02.185 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:02.185815247Z 8:M 04 Sep 2024 05:53:02.185 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:02.304942025Z 8:M 04 Sep 2024 05:53:02.304 # Background transfer terminated by signal 10 2024-09-04T05:53:07.140923428Z 8:M 04 Sep 2024 05:53:07.140 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:07.169290761Z 8:M 04 Sep 2024 05:53:07.169 * Background RDB transfer started by pid 22 2024-09-04T05:53:08.705104549Z 8:M 04 Sep 2024 05:53:08.705 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:08.705147835Z 8:M 04 Sep 2024 05:53:08.705 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:08.705156380Z 22:signal-handler (1725429188) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:08.708202879Z 8:M 04 Sep 2024 05:53:08.708 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:08.708209271Z 8:M 04 Sep 2024 05:53:08.708 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:08.708214156Z 8:M 04 Sep 2024 05:53:08.708 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:08.775553201Z 8:M 04 Sep 2024 05:53:08.775 # Background transfer terminated by signal 10 2024-09-04T05:53:12.205206786Z 8:M 04 Sep 2024 05:53:12.205 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:12.206799951Z 8:M 04 Sep 2024 05:53:12.206 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:12.206806543Z 8:M 04 Sep 2024 05:53:12.206 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:12.206810998Z 8:M 04 Sep 2024 05:53:12.206 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:17.236168176Z 8:M 04 Sep 2024 05:53:17.236 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:17.262140392Z 8:M 04 Sep 2024 05:53:17.262 * Background RDB transfer started by pid 23 2024-09-04T05:53:18.744048527Z 8:M 04 Sep 2024 05:53:18.744 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:18.744222746Z 8:M 04 Sep 2024 05:53:18.744 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:18.744234131Z 23:signal-handler (1725429198) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:18.745388570Z 8:M 04 Sep 2024 05:53:18.745 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:18.745396901Z 8:M 04 Sep 2024 05:53:18.745 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:18.745396901Z 8:M 04 Sep 2024 05:53:18.745 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:18.874539076Z 8:M 04 Sep 2024 05:53:18.874 # Background transfer terminated by signal 10 2024-09-04T05:53:22.226054812Z 8:M 04 Sep 2024 05:53:22.226 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:22.227434209Z 8:M 04 Sep 2024 05:53:22.227 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:22.227442102Z 8:M 04 Sep 2024 05:53:22.227 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:22.227442102Z 8:M 04 Sep 2024 05:53:22.227 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:27.329261859Z 8:M 04 Sep 2024 05:53:27.329 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:27.355406622Z 8:M 04 Sep 2024 05:53:27.355 * Background RDB transfer started by pid 24 2024-09-04T05:53:28.790079853Z 8:M 04 Sep 2024 05:53:28.790 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:28.790106752Z 8:M 04 Sep 2024 05:53:28.790 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:28.790120304Z 24:signal-handler (1725429208) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:28.791686129Z 8:M 04 Sep 2024 05:53:28.791 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:28.791692328Z 8:M 04 Sep 2024 05:53:28.791 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:28.791697237Z 8:M 04 Sep 2024 05:53:28.791 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:28.861835229Z 8:M 04 Sep 2024 05:53:28.861 # Background transfer terminated by signal 10 2024-09-04T05:53:32.246138295Z 8:M 04 Sep 2024 05:53:32.245 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:32.247323300Z 8:M 04 Sep 2024 05:53:32.247 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:32.247332979Z 8:M 04 Sep 2024 05:53:32.247 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:32.247332979Z 8:M 04 Sep 2024 05:53:32.247 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:37.415509701Z 8:M 04 Sep 2024 05:53:37.415 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:37.441385277Z 8:M 04 Sep 2024 05:53:37.441 * Background RDB transfer started by pid 25 2024-09-04T05:53:38.890083119Z 8:M 04 Sep 2024 05:53:38.890 # Diskless rdb transfer, write error sending DB to replica: Connection reset by peer 2024-09-04T05:53:38.890083119Z 8:M 04 Sep 2024 05:53:38.890 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:38.890112580Z 8:M 04 Sep 2024 05:53:38.890 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:38.890124162Z 25:signal-handler (1725429218) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:38.891743284Z 8:M 04 Sep 2024 05:53:38.891 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:38.891749578Z 8:M 04 Sep 2024 05:53:38.891 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:38.891754536Z 8:M 04 Sep 2024 05:53:38.891 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:38.947730347Z 8:M 04 Sep 2024 05:53:38.947 # Background transfer terminated by signal 10 2024-09-04T05:53:42.267494567Z 8:M 04 Sep 2024 05:53:42.267 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:42.268964041Z 8:M 04 Sep 2024 05:53:42.268 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:42.268970652Z 8:M 04 Sep 2024 05:53:42.268 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:42.268975386Z 8:M 04 Sep 2024 05:53:42.268 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:47.507359891Z 8:M 04 Sep 2024 05:53:47.507 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:47.533568754Z 8:M 04 Sep 2024 05:53:47.533 * Background RDB transfer started by pid 26 2024-09-04T05:53:49.434222246Z 8:M 04 Sep 2024 05:53:49.434 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:49.434270388Z 8:M 04 Sep 2024 05:53:49.434 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:49.434287980Z 26:signal-handler (1725429229) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:49.435209172Z 8:M 04 Sep 2024 05:53:49.435 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:49.435215837Z 8:M 04 Sep 2024 05:53:49.435 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:49.435231112Z 8:M 04 Sep 2024 05:53:49.435 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:49.542232297Z 8:M 04 Sep 2024 05:53:49.542 # Background transfer terminated by signal 10 2024-09-04T05:53:52.288927903Z 8:M 04 Sep 2024 05:53:52.288 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:52.290560694Z 8:M 04 Sep 2024 05:53:52.290 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:52.290567771Z 8:M 04 Sep 2024 05:53:52.290 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:52.290572619Z 8:M 04 Sep 2024 05:53:52.290 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:57.594600151Z 8:M 04 Sep 2024 05:53:57.594 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:57.620919124Z 8:M 04 Sep 2024 05:53:57.620 * Background RDB transfer started by pid 27 2024-09-04T05:53:59.481807734Z 8:M 04 Sep 2024 05:53:59.481 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:59.481846238Z 8:M 04 Sep 2024 05:53:59.481 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:59.481846238Z 27:signal-handler (1725429239) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:59.482688748Z 8:M 04 Sep 2024 05:53:59.482 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:59.482695390Z 8:M 04 Sep 2024 05:53:59.482 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:59.482700030Z 8:M 04 Sep 2024 05:53:59.482 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:59.529274962Z 8:M 04 Sep 2024 05:53:59.529 # Background transfer terminated by signal 10 2024-09-04T05:54:02.308877752Z 8:M 04 Sep 2024 05:54:02.308 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:07.679413142Z 8:M 04 Sep 2024 05:54:07.679 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:07.709925483Z 8:M 04 Sep 2024 05:54:07.709 * Background RDB transfer started by pid 28 2024-09-04T05:54:09.528059127Z 8:M 04 Sep 2024 05:54:09.528 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:09.528102942Z 8:M 04 Sep 2024 05:54:09.528 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:09.528102942Z 28:signal-handler (1725429249) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:09.529283739Z 8:M 04 Sep 2024 05:54:09.529 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:09.529294551Z 8:M 04 Sep 2024 05:54:09.529 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:09.529294551Z 8:M 04 Sep 2024 05:54:09.529 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:09.616873388Z 8:M 04 Sep 2024 05:54:09.616 # Background transfer terminated by signal 10 2024-09-04T05:54:12.330808256Z 8:M 04 Sep 2024 05:54:12.330 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:12.332065010Z 8:M 04 Sep 2024 05:54:12.332 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:12.332075827Z 8:M 04 Sep 2024 05:54:12.332 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:12.332075827Z 8:M 04 Sep 2024 05:54:12.332 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:17.771918316Z 8:M 04 Sep 2024 05:54:17.771 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:17.802199497Z 8:M 04 Sep 2024 05:54:17.802 * Background RDB transfer started by pid 29 2024-09-04T05:54:19.740090467Z 8:M 04 Sep 2024 05:54:19.740 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:19.740130540Z 8:M 04 Sep 2024 05:54:19.740 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:19.740142017Z 29:signal-handler (1725429259) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:19.741479993Z 8:M 04 Sep 2024 05:54:19.741 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:19.741486385Z 8:M 04 Sep 2024 05:54:19.741 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:19.741491161Z 8:M 04 Sep 2024 05:54:19.741 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:19.810141343Z 8:M 04 Sep 2024 05:54:19.810 # Background transfer terminated by signal 10 2024-09-04T05:54:22.350761223Z 8:M 04 Sep 2024 05:54:22.350 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:22.352594415Z 8:M 04 Sep 2024 05:54:22.352 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:22.352601685Z 8:M 04 Sep 2024 05:54:22.352 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:22.352606506Z 8:M 04 Sep 2024 05:54:22.352 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:27.865504567Z 8:M 04 Sep 2024 05:54:27.865 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:27.891277470Z 8:M 04 Sep 2024 05:54:27.891 * Background RDB transfer started by pid 30 2024-09-04T05:54:29.780454599Z 8:M 04 Sep 2024 05:54:29.780 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:29.780491148Z 8:M 04 Sep 2024 05:54:29.780 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:29.780491148Z 30:signal-handler (1725429269) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:29.781557476Z 8:M 04 Sep 2024 05:54:29.781 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:29.781557476Z 8:M 04 Sep 2024 05:54:29.781 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:29.781568122Z 8:M 04 Sep 2024 05:54:29.781 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:29.899191914Z 8:M 04 Sep 2024 05:54:29.899 # Background transfer terminated by signal 10 2024-09-04T05:54:32.373324203Z 8:M 04 Sep 2024 05:54:32.373 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:32.374770669Z 8:M 04 Sep 2024 05:54:32.374 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:32.374770669Z 8:M 04 Sep 2024 05:54:32.374 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:32.374785245Z 8:M 04 Sep 2024 05:54:32.374 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:37.956206699Z 8:M 04 Sep 2024 05:54:37.956 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:37.982095075Z 8:M 04 Sep 2024 05:54:37.982 * Background RDB transfer started by pid 31 2024-09-04T05:54:39.841955582Z 8:M 04 Sep 2024 05:54:39.841 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:39.842046390Z 8:M 04 Sep 2024 05:54:39.842 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:39.842054121Z 31:signal-handler (1725429279) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:39.843188909Z 8:M 04 Sep 2024 05:54:39.843 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:39.843196043Z 8:M 04 Sep 2024 05:54:39.843 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:39.843201092Z 8:M 04 Sep 2024 05:54:39.843 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:39.889895156Z 8:M 04 Sep 2024 05:54:39.889 # Background transfer terminated by signal 10 2024-09-04T05:54:42.393838021Z 8:M 04 Sep 2024 05:54:42.393 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:47.041788381Z 8:M 04 Sep 2024 05:54:47.041 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:47.068197290Z 8:M 04 Sep 2024 05:54:47.068 * Background RDB transfer started by pid 32 2024-09-04T05:54:50.101226932Z 8:M 04 Sep 2024 05:54:50.101 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:50.101280884Z 8:M 04 Sep 2024 05:54:50.101 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:50.101297656Z 32:signal-handler (1725429290) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:50.102655956Z 8:M 04 Sep 2024 05:54:50.102 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:50.102664377Z 8:M 04 Sep 2024 05:54:50.102 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:50.102664377Z 8:M 04 Sep 2024 05:54:50.102 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:50.178262897Z 8:M 04 Sep 2024 05:54:50.178 # Background transfer terminated by signal 10 2024-09-04T05:54:52.439280330Z 8:M 04 Sep 2024 05:54:52.439 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:57.121692048Z 8:M 04 Sep 2024 05:54:57.121 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:57.148149712Z 8:M 04 Sep 2024 05:54:57.148 * Background RDB transfer started by pid 33 2024-09-04T05:55:00.236641481Z 8:M 04 Sep 2024 05:55:00.236 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:00.236764536Z 8:M 04 Sep 2024 05:55:00.236 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:00.236771797Z 33:signal-handler (1725429300) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:00.237936392Z 8:M 04 Sep 2024 05:55:00.237 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:00.237951345Z 8:M 04 Sep 2024 05:55:00.237 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:00.237951345Z 8:M 04 Sep 2024 05:55:00.237 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:00.360012176Z 8:M 04 Sep 2024 05:55:00.359 # Background transfer terminated by signal 10 2024-09-04T05:55:02.463033734Z 8:M 04 Sep 2024 05:55:02.462 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:02.464873994Z 8:M 04 Sep 2024 05:55:02.464 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:02.464881477Z 8:M 04 Sep 2024 05:55:02.464 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:02.464886455Z 8:M 04 Sep 2024 05:55:02.464 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:07.209151256Z 8:M 04 Sep 2024 05:55:07.209 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:07.236078355Z 8:M 04 Sep 2024 05:55:07.236 * Background RDB transfer started by pid 34 2024-09-04T05:55:10.291860724Z 8:M 04 Sep 2024 05:55:10.291 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:10.291969355Z 8:M 04 Sep 2024 05:55:10.291 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:10.291994271Z 34:signal-handler (1725429310) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:10.292968362Z 8:M 04 Sep 2024 05:55:10.292 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:10.292974452Z 8:M 04 Sep 2024 05:55:10.292 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:10.292979284Z 8:M 04 Sep 2024 05:55:10.292 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:10.348548812Z 8:M 04 Sep 2024 05:55:10.348 # Background transfer terminated by signal 10 2024-09-04T05:55:12.484938105Z 8:M 04 Sep 2024 05:55:12.484 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:12.486255656Z 8:M 04 Sep 2024 05:55:12.486 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:12.486297424Z 8:M 04 Sep 2024 05:55:12.486 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:12.486297424Z 8:M 04 Sep 2024 05:55:12.486 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:17.292725369Z 8:M 04 Sep 2024 05:55:17.292 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:17.318742904Z 8:M 04 Sep 2024 05:55:17.318 * Background RDB transfer started by pid 35 2024-09-04T05:55:20.571174106Z 8:M 04 Sep 2024 05:55:20.571 # Diskless rdb transfer, write error sending DB to replica: Connection reset by peer 2024-09-04T05:55:20.571174106Z 8:M 04 Sep 2024 05:55:20.571 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:20.571201198Z 8:M 04 Sep 2024 05:55:20.571 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:20.571201198Z 35:signal-handler (1725429320) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:20.572526692Z 8:M 04 Sep 2024 05:55:20.572 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:20.572533694Z 8:M 04 Sep 2024 05:55:20.572 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:20.572538346Z 8:M 04 Sep 2024 05:55:20.572 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:20.629479288Z 8:M 04 Sep 2024 05:55:20.629 # Background transfer terminated by signal 10 2024-09-04T05:55:22.504719021Z 8:M 04 Sep 2024 05:55:22.504 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:22.505507967Z 8:M 04 Sep 2024 05:55:22.505 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:22.505514930Z 8:M 04 Sep 2024 05:55:22.505 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:22.505521212Z 8:M 04 Sep 2024 05:55:22.505 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:27.374762775Z 8:M 04 Sep 2024 05:55:27.374 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:27.400782624Z 8:M 04 Sep 2024 05:55:27.400 * Background RDB transfer started by pid 36 2024-09-04T05:55:31.279018770Z 8:M 04 Sep 2024 05:55:31.278 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:31.279063794Z 8:M 04 Sep 2024 05:55:31.279 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:31.279077244Z 36:signal-handler (1725429331) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:31.280259678Z 8:M 04 Sep 2024 05:55:31.280 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:31.280267969Z 8:M 04 Sep 2024 05:55:31.280 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:31.280267969Z 8:M 04 Sep 2024 05:55:31.280 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:31.313647733Z 8:M 04 Sep 2024 05:55:31.313 # Background transfer terminated by signal 10 2024-09-04T05:55:32.527116296Z 8:M 04 Sep 2024 05:55:32.527 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:32.528462143Z 8:M 04 Sep 2024 05:55:32.528 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:32.528469550Z 8:M 04 Sep 2024 05:55:32.528 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:32.528474584Z 8:M 04 Sep 2024 05:55:32.528 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:37.453927030Z 8:M 04 Sep 2024 05:55:37.453 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:37.480918164Z 8:M 04 Sep 2024 05:55:37.480 * Background RDB transfer started by pid 37 2024-09-04T05:55:41.487443584Z 8:M 04 Sep 2024 05:55:41.487 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:41.487528203Z 8:M 04 Sep 2024 05:55:41.487 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:41.487535546Z 37:signal-handler (1725429341) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:41.488858701Z 8:M 04 Sep 2024 05:55:41.488 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:41.488865032Z 8:M 04 Sep 2024 05:55:41.488 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:41.488869579Z 8:M 04 Sep 2024 05:55:41.488 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:41.596714223Z 8:M 04 Sep 2024 05:55:41.596 # Background transfer terminated by signal 10 2024-09-04T05:55:42.551322362Z 8:M 04 Sep 2024 05:55:42.551 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:42.552698908Z 8:M 04 Sep 2024 05:55:42.552 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:42.552707428Z 8:M 04 Sep 2024 05:55:42.552 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:42.552707428Z 8:M 04 Sep 2024 05:55:42.552 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:47.536613295Z 8:M 04 Sep 2024 05:55:47.536 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:47.563424808Z 8:M 04 Sep 2024 05:55:47.563 * Background RDB transfer started by pid 38 2024-09-04T05:55:51.579334377Z 8:M 04 Sep 2024 05:55:51.579 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:51.579456724Z 8:M 04 Sep 2024 05:55:51.579 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:51.579468327Z 38:signal-handler (1725429351) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:51.581028560Z 8:M 04 Sep 2024 05:55:51.581 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:51.581038387Z 8:M 04 Sep 2024 05:55:51.581 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:51.581038387Z 8:M 04 Sep 2024 05:55:51.581 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:51.676434434Z 8:M 04 Sep 2024 05:55:51.676 # Background transfer terminated by signal 10 2024-09-04T05:55:52.575937845Z 8:M 04 Sep 2024 05:55:52.575 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:52.576943654Z 8:M 04 Sep 2024 05:55:52.576 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:52.576966370Z 8:M 04 Sep 2024 05:55:52.576 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:52.576975611Z 8:M 04 Sep 2024 05:55:52.576 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:57.616822260Z 8:M 04 Sep 2024 05:55:57.616 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:57.642849683Z 8:M 04 Sep 2024 05:55:57.642 * Background RDB transfer started by pid 39 2024-09-04T05:56:01.634177253Z 8:M 04 Sep 2024 05:56:01.634 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:01.634528573Z 8:M 04 Sep 2024 05:56:01.634 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:01.634528573Z 39:signal-handler (1725429361) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:01.635170120Z 8:M 04 Sep 2024 05:56:01.635 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:01.635178620Z 8:M 04 Sep 2024 05:56:01.635 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:01.635178620Z 8:M 04 Sep 2024 05:56:01.635 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:01.755344411Z 8:M 04 Sep 2024 05:56:01.755 # Background transfer terminated by signal 10 2024-09-04T05:56:02.596185868Z 8:M 04 Sep 2024 05:56:02.596 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:02.597782941Z 8:M 04 Sep 2024 05:56:02.597 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:02.597790454Z 8:M 04 Sep 2024 05:56:02.597 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:02.597795319Z 8:M 04 Sep 2024 05:56:02.597 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:07.689386204Z 8:M 04 Sep 2024 05:56:07.689 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:07.715938079Z 8:M 04 Sep 2024 05:56:07.715 * Background RDB transfer started by pid 40 2024-09-04T05:56:11.709514046Z 8:M 04 Sep 2024 05:56:11.709 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:11.709745672Z 8:M 04 Sep 2024 05:56:11.709 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:11.709763901Z 40:signal-handler (1725429371) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:11.710799151Z 8:M 04 Sep 2024 05:56:11.710 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:11.710805660Z 8:M 04 Sep 2024 05:56:11.710 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:11.710810345Z 8:M 04 Sep 2024 05:56:11.710 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:11.829254501Z 8:M 04 Sep 2024 05:56:11.829 # Background transfer terminated by signal 10 2024-09-04T05:56:12.618565167Z 8:M 04 Sep 2024 05:56:12.618 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:12.619642192Z 8:M 04 Sep 2024 05:56:12.619 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:12.619656253Z 8:M 04 Sep 2024 05:56:12.619 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:12.619656253Z 8:M 04 Sep 2024 05:56:12.619 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:17.768273392Z 8:M 04 Sep 2024 05:56:17.768 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:17.796352141Z 8:M 04 Sep 2024 05:56:17.796 * Background RDB transfer started by pid 41 2024-09-04T05:56:21.754616110Z 8:M 04 Sep 2024 05:56:21.754 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:21.754642113Z 8:M 04 Sep 2024 05:56:21.754 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:21.754661664Z 41:signal-handler (1725429381) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:21.756742761Z 8:M 04 Sep 2024 05:56:21.756 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:21.756750474Z 8:M 04 Sep 2024 05:56:21.756 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:21.756755122Z 8:M 04 Sep 2024 05:56:21.756 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:21.811165715Z 8:M 04 Sep 2024 05:56:21.811 # Background transfer terminated by signal 10 2024-09-04T05:56:22.644073933Z 8:M 04 Sep 2024 05:56:22.643 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:22.645913532Z 8:M 04 Sep 2024 05:56:22.645 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:22.645922219Z 8:M 04 Sep 2024 05:56:22.645 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:22.645922219Z 8:M 04 Sep 2024 05:56:22.645 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:27.851821183Z 8:M 04 Sep 2024 05:56:27.851 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:27.878957620Z 8:M 04 Sep 2024 05:56:27.878 * Background RDB transfer started by pid 42 2024-09-04T05:56:31.837916514Z 8:M 04 Sep 2024 05:56:31.837 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:31.838120139Z 8:M 04 Sep 2024 05:56:31.838 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:31.838137476Z 42:signal-handler (1725429391) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:31.839142382Z 8:M 04 Sep 2024 05:56:31.839 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:31.839149268Z 8:M 04 Sep 2024 05:56:31.839 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:31.839154027Z 8:M 04 Sep 2024 05:56:31.839 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:31.891728631Z 8:M 04 Sep 2024 05:56:31.891 # Background transfer terminated by signal 10 2024-09-04T05:56:32.664209090Z 8:M 04 Sep 2024 05:56:32.664 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:32.666451305Z 8:M 04 Sep 2024 05:56:32.666 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:32.666476597Z 8:M 04 Sep 2024 05:56:32.666 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:32.666492735Z 8:M 04 Sep 2024 05:56:32.666 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:37.934947998Z 8:M 04 Sep 2024 05:56:37.934 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:37.962195736Z 8:M 04 Sep 2024 05:56:37.962 * Background RDB transfer started by pid 43 2024-09-04T05:56:41.906069307Z 8:M 04 Sep 2024 05:56:41.906 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:41.906096663Z 8:M 04 Sep 2024 05:56:41.906 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:41.906096663Z 43:signal-handler (1725429401) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:41.907073857Z 8:M 04 Sep 2024 05:56:41.907 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:41.907080292Z 8:M 04 Sep 2024 05:56:41.907 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:41.907085390Z 8:M 04 Sep 2024 05:56:41.907 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:41.975548377Z 8:M 04 Sep 2024 05:56:41.975 # Background transfer terminated by signal 10 2024-09-04T05:56:42.685822680Z 8:M 04 Sep 2024 05:56:42.685 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:42.686944574Z 8:M 04 Sep 2024 05:56:42.686 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:42.686952958Z 8:M 04 Sep 2024 05:56:42.686 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:42.686952958Z 8:M 04 Sep 2024 05:56:42.686 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:47.011762006Z 8:M 04 Sep 2024 05:56:47.011 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:47.039604225Z 8:M 04 Sep 2024 05:56:47.039 * Background RDB transfer started by pid 44 2024-09-04T05:56:51.909815714Z 44:C 04 Sep 2024 05:56:51.909 * Fork CoW for RDB: current 7 MB, peak 7 MB, average 5 MB 2024-09-04T05:56:51.909815714Z 8:M 04 Sep 2024 05:56:51.909 # Diskless rdb transfer, done reading from pipe, 1 replicas still up. 2024-09-04T05:56:51.956471320Z 8:M 04 Sep 2024 05:56:51.956 * Background RDB transfer terminated with success 2024-09-04T05:56:51.956529721Z 8:M 04 Sep 2024 05:56:51.956 * Streamed RDB transfer with replica 18.40.2.16:6382 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:56:51.956529721Z 8:M 04 Sep 2024 05:56:51.956 * Synchronization with replica 18.40.2.16:6382 succeeded
Comment From: AlenChenGitHub
@AlenChenGitHub can you share the logs from master?
2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # Redis version=7.0.13, bits=64, commit=00000000, modified=0, pid=8, just started 2024-09-04T05:29:06.252533976Z 8:C 04 Sep 2024 05:29:06.252 # Configuration loaded 2024-09-04T05:29:06.252870054Z 8:M 04 Sep 2024 05:29:06.252 * monotonic clock: POSIX clock_gettime 2024-09-04T05:29:06.253247239Z 8:M 04 Sep 2024 05:29:06.253 * No cluster configuration found, I'm 86926eddfaf2b8c9bf08a4a014bb7fef722384a7 2024-09-04T05:29:06.254068394Z 8:M 04 Sep 2024 05:29:06.254 * Running mode=cluster, port=6379. 2024-09-04T05:29:06.254111751Z 8:M 04 Sep 2024 05:29:06.254 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 2024-09-04T05:29:06.254124008Z 8:M 04 Sep 2024 05:29:06.254 # Server initialized 2024-09-04T05:29:06.255313917Z 8:M 04 Sep 2024 05:29:06.255 * Ready to accept connections 2024-09-04T05:29:06.347872482Z 8:M 04 Sep 2024 05:29:06.347 # IP address for this node updated to 10.128.1.12 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Full resync requested by replica 18.40.2.16:6383 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Replication backlog created, my new replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000' 2024-09-04T05:29:06.375640090Z 8:M 04 Sep 2024 05:29:06.375 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.377527128Z 8:M 04 Sep 2024 05:29:06.377 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:29:06.377534878Z 8:M 04 Sep 2024 05:29:06.377 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '5ddc4b5a47e1ca96e0f4e6727a996e0d3d9a9657', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:06.377539654Z 8:M 04 Sep 2024 05:29:06.377 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Replica 18.40.2.16:6384 asks for synchronization 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Full resync requested by replica 18.40.2.16:6384 2024-09-04T05:29:06.588816407Z 8:M 04 Sep 2024 05:29:06.588 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Replica 19.40.2.11:6383 asks for synchronization 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'fe2d83c0173365ffd8dc29fb072cc95663bd46e3', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:06.870012028Z 8:M 04 Sep 2024 05:29:06.869 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:07.110273067Z 8:M 04 Sep 2024 05:29:07.110 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '5ddc4b5a47e1ca96e0f4e6727a996e0d3d9a9657', my replication IDs are '23f9944fb8633643f348c43a803047128a90a8cc' and '0000000000000000000000000000000000000000') 2024-09-04T05:29:07.187578895Z 8:M 04 Sep 2024 05:29:07.187 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:29:08.524320648Z 8:M 04 Sep 2024 05:29:08.524 # Cluster state changed: ok 2024-09-04T05:29:13.429293296Z 8:M 04 Sep 2024 05:29:13.429 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:29:13.546692677Z 8:M 04 Sep 2024 05:29:13.546 # Connection with replica 18.40.2.16:6384 lost. 2024-09-04T05:29:14.053199805Z 8:M 04 Sep 2024 05:29:14.053 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:29:14.056677916Z 8:M 04 Sep 2024 05:29:14.056 * Background RDB transfer started by pid 13 2024-09-04T05:29:14.080286211Z 13:C 04 Sep 2024 05:29:14.080 * Fork CoW for RDB: current 6 MB, peak 6 MB, average 6 MB 2024-09-04T05:29:14.258013068Z 8:M 04 Sep 2024 05:29:14.257 # Diskless rdb transfer, done reading from pipe, 2 replicas still up. 2024-09-04T05:29:14.268410574Z 8:M 04 Sep 2024 05:29:14.268 * Background RDB transfer terminated with success 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Streamed RDB transfer with replica 19.40.2.11:6383 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Synchronization with replica 19.40.2.11:6383 succeeded 2024-09-04T05:29:14.268434534Z 8:M 04 Sep 2024 05:29:14.268 * Streamed RDB transfer with replica 18.40.2.16:6382 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:29:14.268441216Z 8:M 04 Sep 2024 05:29:14.268 * Synchronization with replica 18.40.2.16:6382 succeeded 2024-09-04T05:29:32.819602599Z 8:M 04 Sep 2024 05:29:32.819 * Marking node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30 as failing (quorum reached). 2024-09-04T05:29:38.738636690Z 8:M 04 Sep 2024 05:29:38.738 # Address updated for node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30, now 19.40.2.11:6391 2024-09-04T05:29:38.867427487Z 8:M 04 Sep 2024 05:29:38.867 * Clear FAIL state for node 6ada6904bc6443e2ae02103fffdd8b8eaf9a7b30: is reachable again and nobody is serving its slots after some time. 2024-09-04T05:37:39.493554652Z 8:M 04 Sep 2024 05:37:39.493 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:37:39.498579541Z 8:M 04 Sep 2024 05:37:39.498 # Connection with replica 19.40.2.11:6383 lost. 2024-09-04T05:37:39.499244132Z 8:M 04 Sep 2024 05:37:39.499 # Configuration change detected. Reconfiguring myself as a replica of 1bea42d01e08ea767239eee03ab8777963828a1e 2024-09-04T05:37:39.499666983Z 8:S 04 Sep 2024 05:37:39.499 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 2024-09-04T05:37:39.499673383Z 8:S 04 Sep 2024 05:37:39.499 * Connecting to MASTER 18.40.2.16:6382 2024-09-04T05:37:39.499806939Z 8:S 04 Sep 2024 05:37:39.499 * MASTER <-> REPLICA sync started 2024-09-04T05:37:39.502002151Z 8:S 04 Sep 2024 05:37:39.501 * Non blocking connect for SYNC fired the event. 2024-09-04T05:37:39.502262152Z 8:S 04 Sep 2024 05:37:39.502 * Master replied to PING, replication can continue... 2024-09-04T05:37:39.502530027Z 8:S 04 Sep 2024 05:37:39.502 * Trying a partial resynchronization (request 23f9944fb8633643f348c43a803047128a90a8cc:627113695). 2024-09-04T05:37:39.502726549Z 8:S 04 Sep 2024 05:37:39.502 * Successful partial resynchronization with master. 2024-09-04T05:37:39.502731281Z 8:S 04 Sep 2024 05:37:39.502 # Master replication ID changed to 2bbb3e0d236184caf578d5a8eb422befe94553ad 2024-09-04T05:37:39.502741464Z 8:S 04 Sep 2024 05:37:39.502 * MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization. 2024-09-04T05:51:51.870298204Z 8:S 04 Sep 2024 05:51:51.870 # Taking over the master (user request). 2024-09-04T05:51:51.870298204Z 8:S 04 Sep 2024 05:51:51.870 # New configEpoch set to 134 2024-09-04T05:51:51.870298204Z 8:M 04 Sep 2024 05:51:51.870 # Connection with master lost. 2024-09-04T05:51:51.870298204Z 8:M 04 Sep 2024 05:51:51.870 * Caching the disconnected master state. 2024-09-04T05:51:51.870346814Z 8:M 04 Sep 2024 05:51:51.870 * Discarding previously cached master state. 2024-09-04T05:51:51.870346814Z 8:M 04 Sep 2024 05:51:51.870 # Setting secondary replication ID to 2bbb3e0d236184caf578d5a8eb422befe94553ad, valid up to offset: 629591655. New replication ID is b79f4ced1372529624c2380dd1a2ab51ecc19839 2024-09-04T05:51:51.872938649Z 8:M 04 Sep 2024 05:51:51.872 * Replica 19.40.2.11:6384 asks for synchronization 2024-09-04T05:51:51.872975372Z 8:M 04 Sep 2024 05:51:51.872 * Partial resynchronization request from 19.40.2.11:6384 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.873234499Z 8:M 04 Sep 2024 05:51:51.873 * Replica 19.40.2.11:6383 asks for synchronization 2024-09-04T05:51:51.873249682Z 8:M 04 Sep 2024 05:51:51.873 * Partial resynchronization request from 19.40.2.11:6383 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.873554307Z 8:M 04 Sep 2024 05:51:51.873 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:51.873559073Z 8:M 04 Sep 2024 05:51:51.873 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:51.873563419Z 8:M 04 Sep 2024 05:51:51.873 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:51.876507853Z 8:M 04 Sep 2024 05:51:51.876 * Replica 18.40.2.16:6384 asks for synchronization 2024-09-04T05:51:51.876525518Z 8:M 04 Sep 2024 05:51:51.876 * Partial resynchronization request from 18.40.2.16:6384 accepted. Sending 0 bytes of backlog starting from offset 629591655. 2024-09-04T05:51:51.880682694Z 8:M 04 Sep 2024 05:51:51.880 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:51.880689927Z 8:M 04 Sep 2024 05:51:51.880 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:51.880689927Z 8:M 04 Sep 2024 05:51:51.880 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:52.040739440Z 8:M 04 Sep 2024 05:51:52.040 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:51:52.042318876Z 8:M 04 Sep 2024 05:51:52.042 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:52.042325684Z 8:M 04 Sep 2024 05:51:52.042 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:52.042330671Z 8:M 04 Sep 2024 05:51:52.042 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:52.061688414Z 8:M 04 Sep 2024 05:51:52.061 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:51:52.062193838Z 8:M 04 Sep 2024 05:51:52.062 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:52.062200042Z 8:M 04 Sep 2024 05:51:52.062 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:52.062204821Z 8:M 04 Sep 2024 05:51:52.062 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:55.318614097Z 8:M 04 Sep 2024 05:51:55.318 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:51:55.319445641Z 8:M 04 Sep 2024 05:51:55.319 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:51:55.319453377Z 8:M 04 Sep 2024 05:51:55.319 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:55.319458352Z 8:M 04 Sep 2024 05:51:55.319 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:51:57.171963040Z 8:M 04 Sep 2024 05:51:57.171 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:51:57.173282296Z 8:M 04 Sep 2024 05:51:57.173 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:51:57.173289640Z 8:M 04 Sep 2024 05:51:57.173 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:51:57.173294424Z 8:M 04 Sep 2024 05:51:57.173 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:00.484081150Z 8:M 04 Sep 2024 05:52:00.484 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:00.516370409Z 8:M 04 Sep 2024 05:52:00.516 * Background RDB transfer started by pid 14 2024-09-04T05:52:02.061538026Z 8:M 04 Sep 2024 05:52:02.061 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:02.062617218Z 8:M 04 Sep 2024 05:52:02.062 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:02.062624389Z 8:M 04 Sep 2024 05:52:02.062 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:02.062629594Z 8:M 04 Sep 2024 05:52:02.062 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:02.079198038Z 8:M 04 Sep 2024 05:52:02.079 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:02.079226845Z 8:M 04 Sep 2024 05:52:02.079 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:02.079241246Z 14:signal-handler (1725429122) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:02.079953772Z 8:M 04 Sep 2024 05:52:02.079 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:02.079959658Z 8:M 04 Sep 2024 05:52:02.079 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:02.079964258Z 8:M 04 Sep 2024 05:52:02.079 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:02.124153532Z 8:M 04 Sep 2024 05:52:02.124 # Background transfer terminated by signal 10 2024-09-04T05:52:06.330690463Z 8:M 04 Sep 2024 05:52:06.330 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:06.332202302Z 8:M 04 Sep 2024 05:52:06.332 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:06.332210111Z 8:M 04 Sep 2024 05:52:06.332 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:06.332210111Z 8:M 04 Sep 2024 05:52:06.332 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:07.432689833Z 8:M 04 Sep 2024 05:52:07.432 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:07.434066537Z 8:M 04 Sep 2024 05:52:07.434 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:07.434073495Z 8:M 04 Sep 2024 05:52:07.434 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:07.434078292Z 8:M 04 Sep 2024 05:52:07.434 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:11.585986582Z 8:M 04 Sep 2024 05:52:11.585 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:11.613714462Z 8:M 04 Sep 2024 05:52:11.613 * Background RDB transfer started by pid 15 2024-09-04T05:52:12.081007928Z 8:M 04 Sep 2024 05:52:12.080 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:12.082203135Z 8:M 04 Sep 2024 05:52:12.082 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:12.082216921Z 8:M 04 Sep 2024 05:52:12.082 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:12.082216921Z 8:M 04 Sep 2024 05:52:12.082 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:12.097930229Z 8:M 04 Sep 2024 05:52:12.097 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:12.098036681Z 8:M 04 Sep 2024 05:52:12.098 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:12.098048311Z 15:signal-handler (1725429132) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:12.098760195Z 8:M 04 Sep 2024 05:52:12.098 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:12.098765335Z 8:M 04 Sep 2024 05:52:12.098 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:12.098769880Z 8:M 04 Sep 2024 05:52:12.098 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:12.217055146Z 8:M 04 Sep 2024 05:52:12.217 # Background transfer terminated by signal 10 2024-09-04T05:52:16.462264461Z 8:M 04 Sep 2024 05:52:16.462 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:16.463187387Z 8:M 04 Sep 2024 05:52:16.463 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:16.463195067Z 8:M 04 Sep 2024 05:52:16.463 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:16.463200027Z 8:M 04 Sep 2024 05:52:16.463 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:17.656697604Z 8:M 04 Sep 2024 05:52:17.656 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:17.683470711Z 8:M 04 Sep 2024 05:52:17.683 * Background RDB transfer started by pid 16 2024-09-04T05:52:17.684135713Z 8:M 04 Sep 2024 05:52:17.684 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:17.685366184Z 8:M 04 Sep 2024 05:52:17.685 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:17.685372021Z 8:M 04 Sep 2024 05:52:17.685 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:17.685376901Z 8:M 04 Sep 2024 05:52:17.685 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.099579147Z 8:M 04 Sep 2024 05:52:22.099 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:22.101294573Z 8:M 04 Sep 2024 05:52:22.101 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:22.101301725Z 8:M 04 Sep 2024 05:52:22.101 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:22.101306436Z 8:M 04 Sep 2024 05:52:22.101 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.114737205Z 8:M 04 Sep 2024 05:52:22.114 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:22.114968948Z 8:M 04 Sep 2024 05:52:22.114 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:22.114975771Z 16:signal-handler (1725429142) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:22.115644538Z 8:M 04 Sep 2024 05:52:22.115 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:22.115649480Z 8:M 04 Sep 2024 05:52:22.115 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:22.115654209Z 8:M 04 Sep 2024 05:52:22.115 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:22.198642344Z 8:M 04 Sep 2024 05:52:22.198 # Background transfer terminated by signal 10 2024-09-04T05:52:26.503687889Z 8:M 04 Sep 2024 05:52:26.503 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:26.504170313Z 8:M 04 Sep 2024 05:52:26.504 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:26.504177291Z 8:M 04 Sep 2024 05:52:26.504 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:26.504182057Z 8:M 04 Sep 2024 05:52:26.504 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:27.717143083Z 8:M 04 Sep 2024 05:52:27.717 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:27.718440897Z 8:M 04 Sep 2024 05:52:27.718 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:27.718450469Z 8:M 04 Sep 2024 05:52:27.718 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:27.718450469Z 8:M 04 Sep 2024 05:52:27.718 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:31.769645231Z 8:M 04 Sep 2024 05:52:31.769 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:31.795978382Z 8:M 04 Sep 2024 05:52:31.795 * Background RDB transfer started by pid 17 2024-09-04T05:52:32.120289617Z 8:M 04 Sep 2024 05:52:32.120 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:32.121800456Z 8:M 04 Sep 2024 05:52:32.121 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:32.121807464Z 8:M 04 Sep 2024 05:52:32.121 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:32.121812555Z 8:M 04 Sep 2024 05:52:32.121 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:32.134125348Z 8:M 04 Sep 2024 05:52:32.134 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:32.134197407Z 8:M 04 Sep 2024 05:52:32.134 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:32.134214929Z 17:signal-handler (1725429152) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:32.135162596Z 8:M 04 Sep 2024 05:52:32.135 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:32.135168818Z 8:M 04 Sep 2024 05:52:32.135 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:32.135173484Z 8:M 04 Sep 2024 05:52:32.135 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:32.198874129Z 8:M 04 Sep 2024 05:52:32.198 # Background transfer terminated by signal 10 2024-09-04T05:52:36.541623930Z 8:M 04 Sep 2024 05:52:36.541 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:36.542231754Z 8:M 04 Sep 2024 05:52:36.542 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:36.542238721Z 8:M 04 Sep 2024 05:52:36.542 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:36.542243381Z 8:M 04 Sep 2024 05:52:36.542 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:37.762891304Z 8:M 04 Sep 2024 05:52:37.762 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:37.764631830Z 8:M 04 Sep 2024 05:52:37.764 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:37.764655361Z 8:M 04 Sep 2024 05:52:37.764 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:37.764671492Z 8:M 04 Sep 2024 05:52:37.764 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:41.867372471Z 8:M 04 Sep 2024 05:52:41.867 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:41.893296744Z 8:M 04 Sep 2024 05:52:41.893 * Background RDB transfer started by pid 18 2024-09-04T05:52:42.141380326Z 8:M 04 Sep 2024 05:52:42.141 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:42.142328056Z 8:M 04 Sep 2024 05:52:42.142 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:42.142335304Z 8:M 04 Sep 2024 05:52:42.142 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:42.142340150Z 8:M 04 Sep 2024 05:52:42.142 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:42.156539396Z 8:M 04 Sep 2024 05:52:42.156 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:42.156598566Z 8:M 04 Sep 2024 05:52:42.156 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:42.156616111Z 18:signal-handler (1725429162) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:42.157419070Z 8:M 04 Sep 2024 05:52:42.157 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:42.157425755Z 8:M 04 Sep 2024 05:52:42.157 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:42.157430602Z 8:M 04 Sep 2024 05:52:42.157 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:42.196165975Z 8:M 04 Sep 2024 05:52:42.196 # Background transfer terminated by signal 10 2024-09-04T05:52:46.585451159Z 8:M 04 Sep 2024 05:52:46.585 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:46.585986221Z 8:M 04 Sep 2024 05:52:46.585 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:46.585994940Z 8:M 04 Sep 2024 05:52:46.585 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:46.585994940Z 8:M 04 Sep 2024 05:52:46.585 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:47.802989676Z 8:M 04 Sep 2024 05:52:47.802 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:47.804299836Z 8:M 04 Sep 2024 05:52:47.804 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:47.804307810Z 8:M 04 Sep 2024 05:52:47.804 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:47.804307810Z 8:M 04 Sep 2024 05:52:47.804 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:51.965330270Z 8:M 04 Sep 2024 05:52:51.965 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:51.991870371Z 8:M 04 Sep 2024 05:52:51.991 * Background RDB transfer started by pid 19 2024-09-04T05:52:52.162715852Z 8:M 04 Sep 2024 05:52:52.162 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:52.163942196Z 8:M 04 Sep 2024 05:52:52.163 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:52.163949690Z 8:M 04 Sep 2024 05:52:52.163 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:52.163954653Z 8:M 04 Sep 2024 05:52:52.163 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:52.177492702Z 8:M 04 Sep 2024 05:52:52.177 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:52.177518258Z 8:M 04 Sep 2024 05:52:52.177 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:52:52.177524830Z 19:signal-handler (1725429172) Received SIGUSR1 in child, exiting now. 2024-09-04T05:52:52.178197346Z 8:M 04 Sep 2024 05:52:52.178 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:52.178204361Z 8:M 04 Sep 2024 05:52:52.178 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:52.178209214Z 8:M 04 Sep 2024 05:52:52.178 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:52:52.295236394Z 8:M 04 Sep 2024 05:52:52.295 # Background transfer terminated by signal 10 2024-09-04T05:52:56.629547710Z 8:M 04 Sep 2024 05:52:56.629 # Connection with replica 18.40.2.16:6383 lost. 2024-09-04T05:52:56.630332936Z 8:M 04 Sep 2024 05:52:56.630 * Replica 18.40.2.16:6383 asks for synchronization 2024-09-04T05:52:56.630355927Z 8:M 04 Sep 2024 05:52:56.630 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:56.630371647Z 8:M 04 Sep 2024 05:52:56.630 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:52:56.726374145Z 8:M 04 Sep 2024 05:52:56.726 # Failover auth granted to 9279b0187ca03dd29b5348685780b25796539f90 for epoch 138 2024-09-04T05:52:57.032276763Z 8:M 04 Sep 2024 05:52:57.032 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:52:57.058642104Z 8:M 04 Sep 2024 05:52:57.058 * Background RDB transfer started by pid 20 2024-09-04T05:52:57.934849476Z 8:M 04 Sep 2024 05:52:57.934 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:52:57.936380404Z 8:M 04 Sep 2024 05:52:57.936 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:52:57.936387463Z 8:M 04 Sep 2024 05:52:57.936 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:52:57.936392066Z 8:M 04 Sep 2024 05:52:57.936 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:01.782689786Z 20:C 04 Sep 2024 05:53:01.782 * Fork CoW for RDB: current 5 MB, peak 5 MB, average 4 MB 2024-09-04T05:53:01.782689786Z 8:M 04 Sep 2024 05:53:01.782 # Diskless rdb transfer, done reading from pipe, 1 replicas still up. 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Background RDB transfer terminated with success 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Streamed RDB transfer with replica 18.40.2.16:6383 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:53:01.874791595Z 8:M 04 Sep 2024 05:53:01.874 * Synchronization with replica 18.40.2.16:6383 succeeded 2024-09-04T05:53:02.075879219Z 8:M 04 Sep 2024 05:53:02.075 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:02.101839168Z 8:M 04 Sep 2024 05:53:02.101 * Background RDB transfer started by pid 21 2024-09-04T05:53:02.184402119Z 8:M 04 Sep 2024 05:53:02.184 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:02.184478073Z 8:M 04 Sep 2024 05:53:02.184 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:02.184485044Z 21:signal-handler (1725429182) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:02.185803735Z 8:M 04 Sep 2024 05:53:02.185 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:02.185810553Z 8:M 04 Sep 2024 05:53:02.185 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:02.185815247Z 8:M 04 Sep 2024 05:53:02.185 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:02.304942025Z 8:M 04 Sep 2024 05:53:02.304 # Background transfer terminated by signal 10 2024-09-04T05:53:07.140923428Z 8:M 04 Sep 2024 05:53:07.140 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:07.169290761Z 8:M 04 Sep 2024 05:53:07.169 * Background RDB transfer started by pid 22 2024-09-04T05:53:08.705104549Z 8:M 04 Sep 2024 05:53:08.705 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:08.705147835Z 8:M 04 Sep 2024 05:53:08.705 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:08.705156380Z 22:signal-handler (1725429188) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:08.708202879Z 8:M 04 Sep 2024 05:53:08.708 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:08.708209271Z 8:M 04 Sep 2024 05:53:08.708 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:08.708214156Z 8:M 04 Sep 2024 05:53:08.708 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:08.775553201Z 8:M 04 Sep 2024 05:53:08.775 # Background transfer terminated by signal 10 2024-09-04T05:53:12.205206786Z 8:M 04 Sep 2024 05:53:12.205 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:12.206799951Z 8:M 04 Sep 2024 05:53:12.206 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:12.206806543Z 8:M 04 Sep 2024 05:53:12.206 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:12.206810998Z 8:M 04 Sep 2024 05:53:12.206 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:17.236168176Z 8:M 04 Sep 2024 05:53:17.236 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:17.262140392Z 8:M 04 Sep 2024 05:53:17.262 * Background RDB transfer started by pid 23 2024-09-04T05:53:18.744048527Z 8:M 04 Sep 2024 05:53:18.744 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:18.744222746Z 8:M 04 Sep 2024 05:53:18.744 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:18.744234131Z 23:signal-handler (1725429198) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:18.745388570Z 8:M 04 Sep 2024 05:53:18.745 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:18.745396901Z 8:M 04 Sep 2024 05:53:18.745 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:18.745396901Z 8:M 04 Sep 2024 05:53:18.745 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:18.874539076Z 8:M 04 Sep 2024 05:53:18.874 # Background transfer terminated by signal 10 2024-09-04T05:53:22.226054812Z 8:M 04 Sep 2024 05:53:22.226 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:22.227434209Z 8:M 04 Sep 2024 05:53:22.227 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:22.227442102Z 8:M 04 Sep 2024 05:53:22.227 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:22.227442102Z 8:M 04 Sep 2024 05:53:22.227 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:27.329261859Z 8:M 04 Sep 2024 05:53:27.329 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:27.355406622Z 8:M 04 Sep 2024 05:53:27.355 * Background RDB transfer started by pid 24 2024-09-04T05:53:28.790079853Z 8:M 04 Sep 2024 05:53:28.790 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:28.790106752Z 8:M 04 Sep 2024 05:53:28.790 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:28.790120304Z 24:signal-handler (1725429208) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:28.791686129Z 8:M 04 Sep 2024 05:53:28.791 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:28.791692328Z 8:M 04 Sep 2024 05:53:28.791 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:28.791697237Z 8:M 04 Sep 2024 05:53:28.791 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:28.861835229Z 8:M 04 Sep 2024 05:53:28.861 # Background transfer terminated by signal 10 2024-09-04T05:53:32.246138295Z 8:M 04 Sep 2024 05:53:32.245 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:32.247323300Z 8:M 04 Sep 2024 05:53:32.247 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:32.247332979Z 8:M 04 Sep 2024 05:53:32.247 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:32.247332979Z 8:M 04 Sep 2024 05:53:32.247 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:37.415509701Z 8:M 04 Sep 2024 05:53:37.415 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:37.441385277Z 8:M 04 Sep 2024 05:53:37.441 * Background RDB transfer started by pid 25 2024-09-04T05:53:38.890083119Z 8:M 04 Sep 2024 05:53:38.890 # Diskless rdb transfer, write error sending DB to replica: Connection reset by peer 2024-09-04T05:53:38.890083119Z 8:M 04 Sep 2024 05:53:38.890 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:38.890112580Z 8:M 04 Sep 2024 05:53:38.890 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:38.890124162Z 25:signal-handler (1725429218) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:38.891743284Z 8:M 04 Sep 2024 05:53:38.891 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:38.891749578Z 8:M 04 Sep 2024 05:53:38.891 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:38.891754536Z 8:M 04 Sep 2024 05:53:38.891 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:38.947730347Z 8:M 04 Sep 2024 05:53:38.947 # Background transfer terminated by signal 10 2024-09-04T05:53:42.267494567Z 8:M 04 Sep 2024 05:53:42.267 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:42.268964041Z 8:M 04 Sep 2024 05:53:42.268 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:42.268970652Z 8:M 04 Sep 2024 05:53:42.268 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:42.268975386Z 8:M 04 Sep 2024 05:53:42.268 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:47.507359891Z 8:M 04 Sep 2024 05:53:47.507 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:47.533568754Z 8:M 04 Sep 2024 05:53:47.533 * Background RDB transfer started by pid 26 2024-09-04T05:53:49.434222246Z 8:M 04 Sep 2024 05:53:49.434 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:49.434270388Z 8:M 04 Sep 2024 05:53:49.434 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:49.434287980Z 26:signal-handler (1725429229) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:49.435209172Z 8:M 04 Sep 2024 05:53:49.435 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:49.435215837Z 8:M 04 Sep 2024 05:53:49.435 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:49.435231112Z 8:M 04 Sep 2024 05:53:49.435 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:49.542232297Z 8:M 04 Sep 2024 05:53:49.542 # Background transfer terminated by signal 10 2024-09-04T05:53:52.288927903Z 8:M 04 Sep 2024 05:53:52.288 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:52.290560694Z 8:M 04 Sep 2024 05:53:52.290 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:52.290567771Z 8:M 04 Sep 2024 05:53:52.290 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:52.290572619Z 8:M 04 Sep 2024 05:53:52.290 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:53:57.594600151Z 8:M 04 Sep 2024 05:53:57.594 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:53:57.620919124Z 8:M 04 Sep 2024 05:53:57.620 * Background RDB transfer started by pid 27 2024-09-04T05:53:59.481807734Z 8:M 04 Sep 2024 05:53:59.481 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:53:59.481846238Z 8:M 04 Sep 2024 05:53:59.481 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:53:59.481846238Z 27:signal-handler (1725429239) Received SIGUSR1 in child, exiting now. 2024-09-04T05:53:59.482688748Z 8:M 04 Sep 2024 05:53:59.482 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:53:59.482695390Z 8:M 04 Sep 2024 05:53:59.482 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:53:59.482700030Z 8:M 04 Sep 2024 05:53:59.482 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:53:59.529274962Z 8:M 04 Sep 2024 05:53:59.529 # Background transfer terminated by signal 10 2024-09-04T05:54:02.308877752Z 8:M 04 Sep 2024 05:54:02.308 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:02.310157188Z 8:M 04 Sep 2024 05:54:02.310 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:07.679413142Z 8:M 04 Sep 2024 05:54:07.679 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:07.709925483Z 8:M 04 Sep 2024 05:54:07.709 * Background RDB transfer started by pid 28 2024-09-04T05:54:09.528059127Z 8:M 04 Sep 2024 05:54:09.528 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:09.528102942Z 8:M 04 Sep 2024 05:54:09.528 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:09.528102942Z 28:signal-handler (1725429249) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:09.529283739Z 8:M 04 Sep 2024 05:54:09.529 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:09.529294551Z 8:M 04 Sep 2024 05:54:09.529 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:09.529294551Z 8:M 04 Sep 2024 05:54:09.529 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:09.616873388Z 8:M 04 Sep 2024 05:54:09.616 # Background transfer terminated by signal 10 2024-09-04T05:54:12.330808256Z 8:M 04 Sep 2024 05:54:12.330 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:12.332065010Z 8:M 04 Sep 2024 05:54:12.332 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:12.332075827Z 8:M 04 Sep 2024 05:54:12.332 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:12.332075827Z 8:M 04 Sep 2024 05:54:12.332 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:17.771918316Z 8:M 04 Sep 2024 05:54:17.771 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:17.802199497Z 8:M 04 Sep 2024 05:54:17.802 * Background RDB transfer started by pid 29 2024-09-04T05:54:19.740090467Z 8:M 04 Sep 2024 05:54:19.740 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:19.740130540Z 8:M 04 Sep 2024 05:54:19.740 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:19.740142017Z 29:signal-handler (1725429259) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:19.741479993Z 8:M 04 Sep 2024 05:54:19.741 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:19.741486385Z 8:M 04 Sep 2024 05:54:19.741 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:19.741491161Z 8:M 04 Sep 2024 05:54:19.741 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:19.810141343Z 8:M 04 Sep 2024 05:54:19.810 # Background transfer terminated by signal 10 2024-09-04T05:54:22.350761223Z 8:M 04 Sep 2024 05:54:22.350 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:22.352594415Z 8:M 04 Sep 2024 05:54:22.352 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:22.352601685Z 8:M 04 Sep 2024 05:54:22.352 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:22.352606506Z 8:M 04 Sep 2024 05:54:22.352 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:27.865504567Z 8:M 04 Sep 2024 05:54:27.865 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:27.891277470Z 8:M 04 Sep 2024 05:54:27.891 * Background RDB transfer started by pid 30 2024-09-04T05:54:29.780454599Z 8:M 04 Sep 2024 05:54:29.780 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:29.780491148Z 8:M 04 Sep 2024 05:54:29.780 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:29.780491148Z 30:signal-handler (1725429269) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:29.781557476Z 8:M 04 Sep 2024 05:54:29.781 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:29.781557476Z 8:M 04 Sep 2024 05:54:29.781 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:29.781568122Z 8:M 04 Sep 2024 05:54:29.781 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:29.899191914Z 8:M 04 Sep 2024 05:54:29.899 # Background transfer terminated by signal 10 2024-09-04T05:54:32.373324203Z 8:M 04 Sep 2024 05:54:32.373 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:32.374770669Z 8:M 04 Sep 2024 05:54:32.374 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:32.374770669Z 8:M 04 Sep 2024 05:54:32.374 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:32.374785245Z 8:M 04 Sep 2024 05:54:32.374 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:37.956206699Z 8:M 04 Sep 2024 05:54:37.956 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:37.982095075Z 8:M 04 Sep 2024 05:54:37.982 * Background RDB transfer started by pid 31 2024-09-04T05:54:39.841955582Z 8:M 04 Sep 2024 05:54:39.841 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:39.842046390Z 8:M 04 Sep 2024 05:54:39.842 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:39.842054121Z 31:signal-handler (1725429279) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:39.843188909Z 8:M 04 Sep 2024 05:54:39.843 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:39.843196043Z 8:M 04 Sep 2024 05:54:39.843 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:39.843201092Z 8:M 04 Sep 2024 05:54:39.843 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:39.889895156Z 8:M 04 Sep 2024 05:54:39.889 # Background transfer terminated by signal 10 2024-09-04T05:54:42.393838021Z 8:M 04 Sep 2024 05:54:42.393 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:42.395215128Z 8:M 04 Sep 2024 05:54:42.395 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:47.041788381Z 8:M 04 Sep 2024 05:54:47.041 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:47.068197290Z 8:M 04 Sep 2024 05:54:47.068 * Background RDB transfer started by pid 32 2024-09-04T05:54:50.101226932Z 8:M 04 Sep 2024 05:54:50.101 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:50.101280884Z 8:M 04 Sep 2024 05:54:50.101 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:54:50.101297656Z 32:signal-handler (1725429290) Received SIGUSR1 in child, exiting now. 2024-09-04T05:54:50.102655956Z 8:M 04 Sep 2024 05:54:50.102 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:50.102664377Z 8:M 04 Sep 2024 05:54:50.102 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:50.102664377Z 8:M 04 Sep 2024 05:54:50.102 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:54:50.178262897Z 8:M 04 Sep 2024 05:54:50.178 # Background transfer terminated by signal 10 2024-09-04T05:54:52.439280330Z 8:M 04 Sep 2024 05:54:52.439 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:54:52.441323641Z 8:M 04 Sep 2024 05:54:52.441 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:54:57.121692048Z 8:M 04 Sep 2024 05:54:57.121 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:54:57.148149712Z 8:M 04 Sep 2024 05:54:57.148 * Background RDB transfer started by pid 33 2024-09-04T05:55:00.236641481Z 8:M 04 Sep 2024 05:55:00.236 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:00.236764536Z 8:M 04 Sep 2024 05:55:00.236 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:00.236771797Z 33:signal-handler (1725429300) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:00.237936392Z 8:M 04 Sep 2024 05:55:00.237 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:00.237951345Z 8:M 04 Sep 2024 05:55:00.237 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:00.237951345Z 8:M 04 Sep 2024 05:55:00.237 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:00.360012176Z 8:M 04 Sep 2024 05:55:00.359 # Background transfer terminated by signal 10 2024-09-04T05:55:02.463033734Z 8:M 04 Sep 2024 05:55:02.462 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:02.464873994Z 8:M 04 Sep 2024 05:55:02.464 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:02.464881477Z 8:M 04 Sep 2024 05:55:02.464 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:02.464886455Z 8:M 04 Sep 2024 05:55:02.464 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:07.209151256Z 8:M 04 Sep 2024 05:55:07.209 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:07.236078355Z 8:M 04 Sep 2024 05:55:07.236 * Background RDB transfer started by pid 34 2024-09-04T05:55:10.291860724Z 8:M 04 Sep 2024 05:55:10.291 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:10.291969355Z 8:M 04 Sep 2024 05:55:10.291 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:10.291994271Z 34:signal-handler (1725429310) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:10.292968362Z 8:M 04 Sep 2024 05:55:10.292 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:10.292974452Z 8:M 04 Sep 2024 05:55:10.292 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:10.292979284Z 8:M 04 Sep 2024 05:55:10.292 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:10.348548812Z 8:M 04 Sep 2024 05:55:10.348 # Background transfer terminated by signal 10 2024-09-04T05:55:12.484938105Z 8:M 04 Sep 2024 05:55:12.484 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:12.486255656Z 8:M 04 Sep 2024 05:55:12.486 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:12.486297424Z 8:M 04 Sep 2024 05:55:12.486 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:12.486297424Z 8:M 04 Sep 2024 05:55:12.486 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:17.292725369Z 8:M 04 Sep 2024 05:55:17.292 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:17.318742904Z 8:M 04 Sep 2024 05:55:17.318 * Background RDB transfer started by pid 35 2024-09-04T05:55:20.571174106Z 8:M 04 Sep 2024 05:55:20.571 # Diskless rdb transfer, write error sending DB to replica: Connection reset by peer 2024-09-04T05:55:20.571174106Z 8:M 04 Sep 2024 05:55:20.571 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:20.571201198Z 8:M 04 Sep 2024 05:55:20.571 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:20.571201198Z 35:signal-handler (1725429320) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:20.572526692Z 8:M 04 Sep 2024 05:55:20.572 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:20.572533694Z 8:M 04 Sep 2024 05:55:20.572 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:20.572538346Z 8:M 04 Sep 2024 05:55:20.572 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:20.629479288Z 8:M 04 Sep 2024 05:55:20.629 # Background transfer terminated by signal 10 2024-09-04T05:55:22.504719021Z 8:M 04 Sep 2024 05:55:22.504 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:22.505507967Z 8:M 04 Sep 2024 05:55:22.505 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:22.505514930Z 8:M 04 Sep 2024 05:55:22.505 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:22.505521212Z 8:M 04 Sep 2024 05:55:22.505 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:27.374762775Z 8:M 04 Sep 2024 05:55:27.374 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:27.400782624Z 8:M 04 Sep 2024 05:55:27.400 * Background RDB transfer started by pid 36 2024-09-04T05:55:31.279018770Z 8:M 04 Sep 2024 05:55:31.278 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:31.279063794Z 8:M 04 Sep 2024 05:55:31.279 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:31.279077244Z 36:signal-handler (1725429331) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:31.280259678Z 8:M 04 Sep 2024 05:55:31.280 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:31.280267969Z 8:M 04 Sep 2024 05:55:31.280 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:31.280267969Z 8:M 04 Sep 2024 05:55:31.280 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:31.313647733Z 8:M 04 Sep 2024 05:55:31.313 # Background transfer terminated by signal 10 2024-09-04T05:55:32.527116296Z 8:M 04 Sep 2024 05:55:32.527 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:32.528462143Z 8:M 04 Sep 2024 05:55:32.528 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:32.528469550Z 8:M 04 Sep 2024 05:55:32.528 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:32.528474584Z 8:M 04 Sep 2024 05:55:32.528 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:37.453927030Z 8:M 04 Sep 2024 05:55:37.453 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:37.480918164Z 8:M 04 Sep 2024 05:55:37.480 * Background RDB transfer started by pid 37 2024-09-04T05:55:41.487443584Z 8:M 04 Sep 2024 05:55:41.487 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:41.487528203Z 8:M 04 Sep 2024 05:55:41.487 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:41.487535546Z 37:signal-handler (1725429341) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:41.488858701Z 8:M 04 Sep 2024 05:55:41.488 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:41.488865032Z 8:M 04 Sep 2024 05:55:41.488 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:41.488869579Z 8:M 04 Sep 2024 05:55:41.488 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:41.596714223Z 8:M 04 Sep 2024 05:55:41.596 # Background transfer terminated by signal 10 2024-09-04T05:55:42.551322362Z 8:M 04 Sep 2024 05:55:42.551 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:42.552698908Z 8:M 04 Sep 2024 05:55:42.552 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:42.552707428Z 8:M 04 Sep 2024 05:55:42.552 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:42.552707428Z 8:M 04 Sep 2024 05:55:42.552 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:47.536613295Z 8:M 04 Sep 2024 05:55:47.536 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:47.563424808Z 8:M 04 Sep 2024 05:55:47.563 * Background RDB transfer started by pid 38 2024-09-04T05:55:51.579334377Z 8:M 04 Sep 2024 05:55:51.579 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:51.579456724Z 8:M 04 Sep 2024 05:55:51.579 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:55:51.579468327Z 38:signal-handler (1725429351) Received SIGUSR1 in child, exiting now. 2024-09-04T05:55:51.581028560Z 8:M 04 Sep 2024 05:55:51.581 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:51.581038387Z 8:M 04 Sep 2024 05:55:51.581 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:51.581038387Z 8:M 04 Sep 2024 05:55:51.581 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:55:51.676434434Z 8:M 04 Sep 2024 05:55:51.676 # Background transfer terminated by signal 10 2024-09-04T05:55:52.575937845Z 8:M 04 Sep 2024 05:55:52.575 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:55:52.576943654Z 8:M 04 Sep 2024 05:55:52.576 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:55:52.576966370Z 8:M 04 Sep 2024 05:55:52.576 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:55:52.576975611Z 8:M 04 Sep 2024 05:55:52.576 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:55:57.616822260Z 8:M 04 Sep 2024 05:55:57.616 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:55:57.642849683Z 8:M 04 Sep 2024 05:55:57.642 * Background RDB transfer started by pid 39 2024-09-04T05:56:01.634177253Z 8:M 04 Sep 2024 05:56:01.634 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:01.634528573Z 8:M 04 Sep 2024 05:56:01.634 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:01.634528573Z 39:signal-handler (1725429361) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:01.635170120Z 8:M 04 Sep 2024 05:56:01.635 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:01.635178620Z 8:M 04 Sep 2024 05:56:01.635 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:01.635178620Z 8:M 04 Sep 2024 05:56:01.635 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:01.755344411Z 8:M 04 Sep 2024 05:56:01.755 # Background transfer terminated by signal 10 2024-09-04T05:56:02.596185868Z 8:M 04 Sep 2024 05:56:02.596 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:02.597782941Z 8:M 04 Sep 2024 05:56:02.597 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:02.597790454Z 8:M 04 Sep 2024 05:56:02.597 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:02.597795319Z 8:M 04 Sep 2024 05:56:02.597 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:07.689386204Z 8:M 04 Sep 2024 05:56:07.689 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:07.715938079Z 8:M 04 Sep 2024 05:56:07.715 * Background RDB transfer started by pid 40 2024-09-04T05:56:11.709514046Z 8:M 04 Sep 2024 05:56:11.709 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:11.709745672Z 8:M 04 Sep 2024 05:56:11.709 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:11.709763901Z 40:signal-handler (1725429371) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:11.710799151Z 8:M 04 Sep 2024 05:56:11.710 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:11.710805660Z 8:M 04 Sep 2024 05:56:11.710 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:11.710810345Z 8:M 04 Sep 2024 05:56:11.710 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:11.829254501Z 8:M 04 Sep 2024 05:56:11.829 # Background transfer terminated by signal 10 2024-09-04T05:56:12.618565167Z 8:M 04 Sep 2024 05:56:12.618 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:12.619642192Z 8:M 04 Sep 2024 05:56:12.619 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:12.619656253Z 8:M 04 Sep 2024 05:56:12.619 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:12.619656253Z 8:M 04 Sep 2024 05:56:12.619 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:17.768273392Z 8:M 04 Sep 2024 05:56:17.768 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:17.796352141Z 8:M 04 Sep 2024 05:56:17.796 * Background RDB transfer started by pid 41 2024-09-04T05:56:21.754616110Z 8:M 04 Sep 2024 05:56:21.754 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:21.754642113Z 8:M 04 Sep 2024 05:56:21.754 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:21.754661664Z 41:signal-handler (1725429381) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:21.756742761Z 8:M 04 Sep 2024 05:56:21.756 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:21.756750474Z 8:M 04 Sep 2024 05:56:21.756 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:21.756755122Z 8:M 04 Sep 2024 05:56:21.756 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:21.811165715Z 8:M 04 Sep 2024 05:56:21.811 # Background transfer terminated by signal 10 2024-09-04T05:56:22.644073933Z 8:M 04 Sep 2024 05:56:22.643 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:22.645913532Z 8:M 04 Sep 2024 05:56:22.645 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:22.645922219Z 8:M 04 Sep 2024 05:56:22.645 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:22.645922219Z 8:M 04 Sep 2024 05:56:22.645 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:27.851821183Z 8:M 04 Sep 2024 05:56:27.851 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:27.878957620Z 8:M 04 Sep 2024 05:56:27.878 * Background RDB transfer started by pid 42 2024-09-04T05:56:31.837916514Z 8:M 04 Sep 2024 05:56:31.837 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:31.838120139Z 8:M 04 Sep 2024 05:56:31.838 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:31.838137476Z 42:signal-handler (1725429391) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:31.839142382Z 8:M 04 Sep 2024 05:56:31.839 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:31.839149268Z 8:M 04 Sep 2024 05:56:31.839 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:31.839154027Z 8:M 04 Sep 2024 05:56:31.839 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:31.891728631Z 8:M 04 Sep 2024 05:56:31.891 # Background transfer terminated by signal 10 2024-09-04T05:56:32.664209090Z 8:M 04 Sep 2024 05:56:32.664 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:32.666451305Z 8:M 04 Sep 2024 05:56:32.666 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:32.666476597Z 8:M 04 Sep 2024 05:56:32.666 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:32.666492735Z 8:M 04 Sep 2024 05:56:32.666 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:37.934947998Z 8:M 04 Sep 2024 05:56:37.934 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:37.962195736Z 8:M 04 Sep 2024 05:56:37.962 * Background RDB transfer started by pid 43 2024-09-04T05:56:41.906069307Z 8:M 04 Sep 2024 05:56:41.906 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:41.906096663Z 8:M 04 Sep 2024 05:56:41.906 # Diskless rdb transfer, last replica dropped, killing fork child. 2024-09-04T05:56:41.906096663Z 43:signal-handler (1725429401) Received SIGUSR1 in child, exiting now. 2024-09-04T05:56:41.907073857Z 8:M 04 Sep 2024 05:56:41.907 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:41.907080292Z 8:M 04 Sep 2024 05:56:41.907 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:41.907085390Z 8:M 04 Sep 2024 05:56:41.907 * Current BGSAVE has socket target. Waiting for next BGSAVE for SYNC 2024-09-04T05:56:41.975548377Z 8:M 04 Sep 2024 05:56:41.975 # Background transfer terminated by signal 10 2024-09-04T05:56:42.685822680Z 8:M 04 Sep 2024 05:56:42.685 # Connection with replica 18.40.2.16:6382 lost. 2024-09-04T05:56:42.686944574Z 8:M 04 Sep 2024 05:56:42.686 * Replica 18.40.2.16:6382 asks for synchronization 2024-09-04T05:56:42.686952958Z 8:M 04 Sep 2024 05:56:42.686 * Partial resynchronization not accepted: Requested offset for second ID was 629605539, but I can reply up to 629591655 2024-09-04T05:56:42.686952958Z 8:M 04 Sep 2024 05:56:42.686 * Delay next BGSAVE for diskless SYNC 2024-09-04T05:56:47.011762006Z 8:M 04 Sep 2024 05:56:47.011 * Starting BGSAVE for SYNC with target: replicas sockets 2024-09-04T05:56:47.039604225Z 8:M 04 Sep 2024 05:56:47.039 * Background RDB transfer started by pid 44 2024-09-04T05:56:51.909815714Z 44:C 04 Sep 2024 05:56:51.909 * Fork CoW for RDB: current 7 MB, peak 7 MB, average 5 MB 2024-09-04T05:56:51.909815714Z 8:M 04 Sep 2024 05:56:51.909 # Diskless rdb transfer, done reading from pipe, 1 replicas still up. 2024-09-04T05:56:51.956471320Z 8:M 04 Sep 2024 05:56:51.956 * Background RDB transfer terminated with success 2024-09-04T05:56:51.956529721Z 8:M 04 Sep 2024 05:56:51.956 * Streamed RDB transfer with replica 18.40.2.16:6382 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming 2024-09-04T05:56:51.956529721Z 8:M 04 Sep 2024 05:56:51.956 * Synchronization with replica 18.40.2.16:6382 succeeded
The ip address of the faulty slave node is 18.40.2.16:6382
Comment From: AlenChenGitHub
@sundb Sorry, my answer to the above question is wrong, correct it.
In the original cluster, we gave the default configuration of repl-diskless-sync no, and later our team changed it to repl-diskless-sync yes. The status of the slave node lost connection was restored.
Comment From: sundb
@AlenChenGitHub thanks, i'll try to reproduce it in my local.
Comment From: sundb
@AlenChenGitHub sorry for the late reply, why closing this issue?