Crash report
1:S 19 Nov 2023 08:07:41.051 # ------------------------------------------------
1:S 19 Nov 2023 08:07:41.051 # !!! Software Failure. Press left mouse button to continue
1:S 19 Nov 2023 08:07:41.051 # Guru Meditation: Replica was unable to write command to disk. #server.c:4029
------ STACK TRACE ------
Backtrace:
redis-server *:6377 [cluster](+0x99641)[0x55be7622d641]
redis-server *:6377 [cluster](processInputBuffer+0x100)[0x55be76250760]
redis-server *:6377 [cluster](readQueryFromClient+0x350)[0x55be76250ca0]
redis-server *:6377 [cluster](+0x1babcc)[0x55be7634ebcc]
redis-server *:6377 [cluster](aeMain+0x101)[0x55be76221471]
redis-server *:6377 [cluster](main+0x3af)[0x55be7621664f]
/lib/x86_64-linux-gnu/libc.so.6(+0x271ca)[0x7fca341d21ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7fca341d2285]
redis-server *:6377 [cluster](_start+0x21)[0x55be76216e51]
------ INFO OUTPUT ------
# Server
redis_version:7.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:104833399552fd99
redis_mode:cluster
os:Linux 6.2.0-1013-gcp x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:12.2.0
process_id:1
process_supervised:no
run_id:f1baa1a84c1446bd56ab32143732b5b5dd8117fe
tcp_port:6377
server_time_usec:1700381261051397
uptime_in_seconds:2871
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5882445
executable:/data/redis-server
config_file:/usr/local/etc/redis/redis.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6377
# Clients
connected_clients:13
cluster_connections:8
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:620745520
used_memory_human:591.99M
used_memory_rss:616660992
used_memory_rss_human:588.09M
used_memory_peak:1069180296
used_memory_peak_human:1019.65M
used_memory_peak_perc:58.06%
used_memory_overhead:261987552
used_memory_startup:1582976
used_memory_dataset:358757968
used_memory_dataset_perc:57.94%
allocator_allocated:621252472
allocator_active:621498368
allocator_resident:632594432
total_system_memory:16758276096
total_system_memory_human:15.61G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:8589934592
maxmemory_human:8.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:245896
allocator_rss_ratio:1.02
allocator_rss_bytes:11096064
rss_overhead_ratio:0.97
rss_overhead_bytes:-15933440
mem_fragmentation_ratio:0.99
mem_fragmentation_bytes:-4082424
mem_not_counted_for_evict:37048
mem_replication_backlog:1086728
mem_total_replication_buffers:1086712
mem_clients_slaves:0
mem_clients_normal:128888
mem_cluster_links:8576
mem_aof_buffer:3072
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:57162
rdb_bgsave_in_progress:0
rdb_last_save_time:1700381196
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:4
rdb_current_bgsave_time_sec:-1
rdb_saves:45
rdb_last_cow_size:23814144
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:11
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:4
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:7
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:25878528
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:188403632
aof_base_size:136108460
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:1237
total_commands_processed:3032722
instantaneous_ops_per_sec:1008
total_net_input_bytes:606447973
total_net_output_bytes:9089472
total_net_repl_input_bytes:571412741
total_net_repl_output_bytes:0
instantaneous_input_kbps:74.96
instantaneous_output_kbps:2.23
instantaneous_input_repl_kbps:60.49
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:239614
keyspace_misses:514410
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:19063
total_forks:52
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:2912357
total_writes_processed:751197
io_threaded_reads_processed:250
io_threaded_writes_processed:687
reply_buffer_shrinks:76
reply_buffer_expands:55
eventloop_cycles:2925129
eventloop_duration_sum:98480076
eventloop_duration_cmd_sum:32684084
instantaneous_eventloop_cycles_per_sec:865
instantaneous_eventloop_duration_usec:31
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:slave
master_host:10.0.0.46
master_port:6373
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:574556074
slave_repl_offset:574555992
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:02bc66b44a2935ba66cdfa7d26b5659f04c07cea
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:574555992
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:573487249
repl_backlog_histlen:1068744
# CPU
used_cpu_sys:48.500003
used_cpu_user:65.533084
used_cpu_sys_children:8.873903
used_cpu_user_children:62.588725
used_cpu_sys_main_thread:48.121616
used_cpu_user_main_thread:64.542304
# Modules
# Commandstats
cmdstat_sadd:calls=22872,usec=123725,usec_per_call=5.41,rejected_calls=0,failed_calls=0
cmdstat_readonly:calls=340,usec=122,usec_per_call=0.36,rejected_calls=0,failed_calls=0
cmdstat_smembers:calls=13469,usec=72865,usec_per_call=5.41,rejected_calls=0,failed_calls=0
cmdstat_pexpireat:calls=22876,usec=52745,usec_per_call=2.31,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=7,usec=8,usec_per_call=1.14,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=1217,usec=2042,usec_per_call=1.68,rejected_calls=0,failed_calls=0
cmdstat_client|list:calls=7,usec=527,usec_per_call=75.29,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=3239,usec=632,usec_per_call=0.20,rejected_calls=0,failed_calls=0
cmdstat_set:calls=2201446,usec=9629694,usec_per_call=4.37,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=269,usec=69,usec_per_call=0.26,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=5,usec=56,usec_per_call=11.20,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=3239,usec=40612,usec_per_call=12.54,rejected_calls=0,failed_calls=0
cmdstat_scard:calls=2434,usec=4603,usec_per_call=1.89,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=1237,usec=7812,usec_per_call=6.32,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=113,usec=21186777,usec_per_call=187493.59,rejected_calls=0,failed_calls=0
cmdstat_select:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|meet:calls=2,usec=17,usec_per_call=8.50,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=1289,usec=39819,usec_per_call=30.89,rejected_calls=0,failed_calls=0
cmdstat_cluster|set-config-epoch:calls=2,usec=11,usec_per_call=5.50,rejected_calls=0,failed_calls=1
cmdstat_cluster|nodes:calls=14,usec=1031,usec_per_call=73.64,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=4,usec=121,usec_per_call=30.25,rejected_calls=0,failed_calls=0
cmdstat_cluster|replicate:calls=4,usec=320,usec_per_call=80.00,rejected_calls=0,failed_calls=2
cmdstat_get:calls=738121,usec=1421416,usec_per_call=1.93,rejected_calls=0,failed_calls=0
cmdstat_info:calls=54,usec=3124,usec_per_call=57.85,rejected_calls=0,failed_calls=0
cmdstat_del:calls=20461,usec=131169,usec_per_call=6.41,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_ERR:count=3
# Latencystats
latency_percentiles_usec_sadd:p50=5.023,p99=22.015,p99.9=40.191
latency_percentiles_usec_readonly:p50=0.001,p99=1.003,p99.9=13.055
latency_percentiles_usec_smembers:p50=5.023,p99=24.063,p99.9=42.239
latency_percentiles_usec_pexpireat:p50=2.007,p99=13.055,p99.9=28.031
latency_percentiles_usec_dbsize:p50=1.003,p99=3.007,p99.9=3.007
latency_percentiles_usec_client|setname:p50=1.003,p99=12.031,p99.9=19.071
latency_percentiles_usec_client|list:p50=77.311,p99=83.455,p99.9=83.455
latency_percentiles_usec_multi:p50=0.001,p99=1.003,p99.9=13.055
latency_percentiles_usec_set:p50=4.015,p99=19.071,p99.9=38.143
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_slowlog|get:p50=10.047,p99=25.087,p99.9=25.087
latency_percentiles_usec_exec:p50=11.007,p99=39.167,p99.9=57.087
latency_percentiles_usec_scard:p50=2.007,p99=9.023,p99.9=27.007
latency_percentiles_usec_auth:p50=5.023,p99=25.087,p99.9=47.103
latency_percentiles_usec_keys:p50=190840.831,p99=425721.855,p99.9=476053.503
latency_percentiles_usec_select:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_cluster|meet:p50=1.003,p99=16.063,p99.9=16.063
latency_percentiles_usec_cluster|slots:p50=23.039,p99=76.287,p99.9=147.455
latency_percentiles_usec_cluster|set-config-epoch:p50=1.003,p99=10.047,p99.9=10.047
latency_percentiles_usec_cluster|nodes:p50=73.215,p99=103.423,p99.9=103.423
latency_percentiles_usec_cluster|info:p50=19.071,p99=45.055,p99.9=45.055
latency_percentiles_usec_cluster|replicate:p50=6.015,p99=206.847,p99.9=206.847
latency_percentiles_usec_get:p50=2.007,p99=7.007,p99.9=25.087
latency_percentiles_usec_info:p50=55.039,p99=110.079,p99.9=118.271
latency_percentiles_usec_del:p50=6.015,p99=24.063,p99.9=46.079
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=2191138,expires=2191138,avg_ttl=0
# Cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:5
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:1
cluster_stats_messages_ping_sent:4243
cluster_stats_messages_pong_sent:4267
cluster_stats_messages_meet_sent:1
cluster_stats_messages_sent:8511
cluster_stats_messages_ping_received:4267
cluster_stats_messages_pong_received:4244
cluster_stats_messages_received:8511
total_cluster_links_buffer_limit_exceeded:0
------ CLUSTER NODES OUTPUT ------
42e12fb9192eaee13206061da902eebc1195cbb9 10.0.0.46:6374@16374,,tls-port=0,shard-id=7bf255b2cd7056542f36f1c1376385d778087abe master - 0 1700381258622 2 connected 5461-10922
3627912670c959bf744e02c96855f1ef271441d7 10.0.0.46:6373@16373,,tls-port=0,shard-id=f4cac3f0a5beff8b75a06c12644e90932cbc349d master - 0 1700381260647 1 connected 0-5460
c2362bd23028319ca61ea84bc3e5cefd92df26ef 10.0.0.46:6375@16375,,tls-port=0,shard-id=0ac5adf9be8614894680d44abf1954028f7afd0a master - 0 1700381259658 3 connected 10923-16383
5ff6e748ca1c5b41f78642236c39016834a67bf3 10.0.0.46:6377@16377,,tls-port=0,shard-id=f4cac3f0a5beff8b75a06c12644e90932cbc349d myself,slave 3627912670c959bf744e02c96855f1ef271441d7 0 1700381260000 1 connected
04027e0af52953c8ebc6f3f44b00d6db0e129557 10.0.0.46:6376@16376,,tls-port=0,shard-id=7bf255b2cd7056542f36f1c1376385d778087abe slave 42e12fb9192eaee13206061da902eebc1195cbb9 0 1700381259000 2 connected
------ CLIENT LIST OUTPUT ------
id=957 addr=10.152.4.180:42810 laddr=10.0.0.46:6377 fd=34 name= age=2422 idle=149 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
id=911 addr=10.0.0.46:36518 laddr=10.0.0.46:6377 fd=26 name= age=2732 idle=82 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|slots user=default redir=-1 resp=2 lib-name= lib-ver=
id=905 addr=10.0.0.46:6373 laddr=10.0.0.46:46158 fd=21 name= age=2736 idle=0 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=82 qbuf-free=40872 argv-mem=46 multi-mem=0 rbs=1024 rbp=67 obl=0 oll=0 omem=0 tot-mem=42966 events=r cmd=set user=(superuser) redir=-1 resp=2 lib-name= lib-ver=
id=906 addr=10.152.6.142:60504 laddr=10.0.0.46:6377 fd=11 name= age=2736 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=12 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
id=907 addr=10.0.0.46:36470 laddr=10.0.0.46:6377 fd=22 name= age=2732 idle=72 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|slots user=default redir=-1 resp=2 lib-name= lib-ver=
id=981 addr=10.152.1.21:56178 laddr=10.0.0.46:6377 fd=30 name= age=2210 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=167 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
id=919 addr=10.152.5.203:35966 laddr=10.0.0.46:6377 fd=29 name= age=2674 idle=34 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
id=908 addr=10.0.0.46:36482 laddr=10.0.0.46:6377 fd=23 name= age=2732 idle=57 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|slots user=default redir=-1 resp=2 lib-name= lib-ver=
id=909 addr=10.0.0.46:36492 laddr=10.0.0.46:6377 fd=24 name= age=2732 idle=12 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|slots user=default redir=-1 resp=2 lib-name= lib-ver=
id=910 addr=10.0.0.46:36502 laddr=10.0.0.46:6377 fd=25 name= age=2732 idle=7 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=cluster|slots user=default redir=-1 resp=2 lib-name= lib-ver=
id=912 addr=10.152.5.168:41852 laddr=10.0.0.46:6377 fd=27 name= age=2730 idle=1 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=scard user=default redir=-1 resp=2 lib-name= lib-ver=
id=928 addr=10.152.8.3:55184 laddr=10.0.0.46:6377 fd=33 name= age=2624 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=15 obl=0 oll=0 omem=0 tot-mem=22400 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
id=983 addr=10.152.8.4:41722 laddr=10.0.0.46:6377 fd=28 name= age=2195 idle=0 flags=r db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23424 events=r cmd=get user=default redir=-1 resp=2 lib-name= lib-ver=
------ CURRENT CLIENT INFO ------
id=905 addr=10.0.0.46:6373 laddr=10.0.0.46:46158 fd=21 name= age=2736 idle=0 flags=M db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=82 qbuf-free=40872 argv-mem=46 multi-mem=0 rbs=1024 rbp=67 obl=0 oll=0 omem=0 tot-mem=42966 events=r cmd=set user=(superuser) redir=-1 resp=2 lib-name= lib-ver=
argc: '5'
argv[0]: '"SET"'
argv[1]: '"floats:itemsl:34404699024"'
argv[2]: '"1"'
argv[3]: '"PXAT"'
argv[4]: '"1700399261051"'
------ MODULES INFO OUTPUT ------
------ CONFIG DEBUG OUTPUT ------
list-compress-depth 0
activedefrag yes
io-threads-do-reads yes
replica-read-only yes
slave-read-only yes
repl-diskless-sync yes
lazyfree-lazy-expire no
client-query-buffer-limit 1gb
repl-diskless-load disabled
lazyfree-lazy-server-del no
proto-max-bulk-len 512mb
lazyfree-lazy-user-del no
io-threads 2
lazyfree-lazy-eviction no
lazyfree-lazy-user-flush no
sanitize-dump-payload no
------ FAST MEMORY TEST ------
1:S 19 Nov 2023 08:07:41.052 # Bio worker thread #0 terminated
1:S 19 Nov 2023 08:07:41.052 # Bio worker thread #1 terminated
1:S 19 Nov 2023 08:07:41.053 # Bio worker thread #2 terminated
1:S 19 Nov 2023 08:07:41.053 # IO thread(tid:140506398967488) terminated
*** Preparing to test memory region 55be764fe000 (2273280 bytes)
*** Preparing to test memory region 55be77c05000 (397312 bytes)
*** Preparing to test memory region 7fc9bda00000 (1723858944 bytes)
*** Preparing to test memory region 7fca247ff000 (37748736 bytes)
*** Preparing to test memory region 7fca26c00000 (8388608 bytes)
*** Preparing to test memory region 7fca27400000 (2097152 bytes)
*** Preparing to test memory region 7fca276ff000 (59768832 bytes)
*** Preparing to test memory region 7fca2b000000 (8388608 bytes)
*** Preparing to test memory region 7fca2b800000 (2097152 bytes)
*** Preparing to test memory region 7fca2bb00000 (5378048 bytes)
*** Preparing to test memory region 7fca30200000 (8388608 bytes)
*** Preparing to test memory region 7fca30a00000 (2097152 bytes)
*** Preparing to test memory region 7fca30d7b000 (2621440 bytes)
*** Preparing to test memory region 7fca30ffc000 (8388608 bytes)
*** Preparing to test memory region 7fca317fd000 (8388608 bytes)
*** Preparing to test memory region 7fca31ffe000 (8388608 bytes)
*** Preparing to test memory region 7fca327ff000 (8388608 bytes)
*** Preparing to test memory region 7fca33000000 (8388608 bytes)
*** Preparing to test memory region 7fca33800000 (8388608 bytes)
*** Preparing to test memory region 7fca341a6000 (20480 bytes)
*** Preparing to test memory region 7fca3437f000 (53248 bytes)
*** Preparing to test memory region 7fca3480b000 (12288 bytes)
*** Preparing to test memory region 7fca34999000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
Additional information 1. Docker image: node:current-slim 2. Library: ioredis@5.3.2
Comment From: enjoy-binbin
can you check replica-ignore-disk-write-errors conf in the redis.conf and see if it can help?
Comment From: oranagra
this "crash" is intentional, and i wouldn't recommend to turn on that config. it is there to warn you that the replica is unable to persist. i.e. if it's unreliable, better kill it early, rather than think it's there, rely on it for durability and then find out it's bad when you actually need it..
i see your info has rdb_last_bgsave_status:err, can you check earlier in the log to see what happened?
Comment From: alexhype17
this "crash" is intentional, and i wouldn't recommend to turn on that config. it is there to warn you that the replica is unable to persist. i.e. if it's unreliable, better kill it early, rather than think it's there, rely on it for durability and then find out it's bad when you actually need it..
i see your info has
rdb_last_bgsave_status:err, can you check earlier in the log to see what happened?
1:S 19 Nov 2023 08:05:28.105 * Background saving started by pid 72
72:C 19 Nov 2023 08:05:31.353 * DB saved on disk
72:C 19 Nov 2023 08:05:31.372 * Fork CoW for RDB: current 24 MB, peak 24 MB, average 14 MB
1:S 19 Nov 2023 08:05:31.427 * Background saving terminated with success
1:S 19 Nov 2023 08:06:32.100 * 10000 changes in 60 seconds. Saving...
1:S 19 Nov 2023 08:06:32.118 * Background saving started by pid 73
73:C 19 Nov 2023 08:06:35.921 * DB saved on disk
73:C 19 Nov 2023 08:06:35.935 * Fork CoW for RDB: current 22 MB, peak 22 MB, average 12 MB
1:S 19 Nov 2023 08:06:36.048 * Background saving terminated with success
1:S 19 Nov 2023 08:07:37.059 * 10000 changes in 60 seconds. Saving...
1:S 19 Nov 2023 08:07:37.078 * Background saving started by pid 74
74:C 19 Nov 2023 08:07:40.931 # Error moving temp DB file temp-74.rdb on the final destination dump.rdb (in server root dir /var/lib/redis): No such file or directory
1:S 19 Nov 2023 08:07:41.046 # Background saving error
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:S 19 Nov 2023 08:07:41.051 # ------------------------------------------------
1:S 19 Nov 2023 08:07:41.051 # !!! Software Failure. Press left mouse button to continue
1:S 19 Nov 2023 08:07:41.051 # Guru Meditation: Replica was unable to write command to disk. #server.c:4029
Comment From: alexhype17
latest logs with replica-ignore-disk-write-errors yes sometime replicas disconnecting and reconnecting after 2-5 minutes downtime
1:S 19 Nov 2023 17:48:14.236 * Discarding previously cached master state.
1:S 19 Nov 2023 17:48:14.236 * MASTER <-> REPLICA sync: Flushing old data
1:S 19 Nov 2023 17:48:24.064 * MASTER <-> REPLICA sync: Loading DB in memory
1:S 19 Nov 2023 17:48:24.088 * Loading RDB produced by version 7.2.3
1:S 19 Nov 2023 17:48:24.088 * RDB age 82 seconds
1:S 19 Nov 2023 17:48:24.088 * RDB memory usage when created 1486.59 Mb
1:S 19 Nov 2023 17:48:46.628 * Done loading RDB, keys loaded: 10369059, keys expired: 0.
1:S 19 Nov 2023 17:48:46.713 * MASTER <-> REPLICA sync: Finished with success
1:S 19 Nov 2023 17:48:46.713 * Creating AOF incr file temp-appendonly.aof.incr on background rewrite
1:S 19 Nov 2023 17:48:46.776 * Background append only file rewriting started by pid 259
1:S 19 Nov 2023 17:49:03.009 * FAIL message received from 5c0d4d53e06da6dfcd727dc9437cd53651655d9d () about 1566ceb3932da0460daedef419526f554409141a ()
1:S 19 Nov 2023 17:49:19.901 * Clear FAIL state for node 1566ceb3932da0460daedef419526f554409141a ():replica is reachable again.
1:S 19 Nov 2023 17:49:19.901 * FAIL message received from 8b2853c1c99b2e493601e8c9f10ddcd8f51efefc () about 1566ceb3932da0460daedef419526f554409141a ()
259:C 19 Nov 2023 17:49:22.377 * Successfully created the temporary AOF base file temp-rewriteaof-bg-259.aof
259:C 19 Nov 2023 17:49:22.481 * Fork CoW for AOF rewrite: current 203 MB, peak 203 MB, average 154 MB
1:S 19 Nov 2023 17:49:31.774 * Background AOF rewrite terminated with success
1:S 19 Nov 2023 17:49:31.774 * Successfully renamed the temporary AOF base file temp-rewriteaof-bg-259.aof into appendonly.aof.19.base.rdb
1:S 19 Nov 2023 17:49:31.774 * Successfully renamed the temporary AOF incr file temp-appendonly.aof.incr into appendonly.aof.19.incr.aof
1:S 19 Nov 2023 17:49:31.780 * Removing the history file appendonly.aof.18.incr.aof in the background
1:S 19 Nov 2023 17:49:31.780 * Removing the history file appendonly.aof.18.base.rdb in the background
1:S 19 Nov 2023 17:49:31.788 * Background AOF rewrite finished successfully
1:S 19 Nov 2023 17:49:38.510 * 10000 changes in 60 seconds. Saving...
1:S 19 Nov 2023 17:49:38.609 * Background saving started by pid 260
1:S 19 Nov 2023 17:49:47.125 * Clear FAIL state for node 1566ceb3932da0460daedef419526f554409141a ():replica is reachable again.
260:C 19 Nov 2023 17:50:24.237 * DB saved on disk
260:C 19 Nov 2023 17:50:24.427 * Fork CoW for RDB: current 174 MB, peak 175 MB, average 137 MB
1:S 19 Nov 2023 17:50:31.389 * FAIL message received from 5c0d4d53e06da6dfcd727dc9437cd53651655d9d () about 1566ceb3932da0460daedef419526f554409141a ()
1:S 19 Nov 2023 17:50:31.390 * Background saving terminated with success
1:S 19 Nov 2023 17:51:14.787 * Clear FAIL state for node 1566ceb3932da0460daedef419526f554409141a ():replica is reachable again.
1:S 19 Nov 2023 17:51:45.732 * 10000 changes in 60 seconds. Saving...
1:S 19 Nov 2023 17:51:45.854 * Background saving started by pid 261
1:S 19 Nov 2023 17:52:17.499 * FAIL message received from 6b5031a18f7f760308213240e0f33c5a4205ed46 () about 1566ceb3932da0460daedef419526f554409141a ()
261:C 19 Nov 2023 17:52:29.524 * DB saved on disk
261:C 19 Nov 2023 17:52:29.650 * Fork CoW for RDB: current 123 MB, peak 123 MB, average 50 MB
1:S 19 Nov 2023 17:52:37.695 * Background saving terminated with success
Comment From: oranagra
Error moving temp DB file temp-74.rdb on the final destination dump.rdb (in server root dir /var/lib/redis): No such file or directory
i don't understand how come the file is missing. we didn't print any error while creating it. could it be that some external tool deleted it while it's forming?