Notice! - If a Redis module was involved, please open an issue in the module's repo instead! - If you're using docker on Apple M1, please make sure the image you're using was compiled for ARM!
=== REDIS BUG REPORT START: Cut & paste starting from here === 16886:S 18 May 2024 17:37:58.437 # ------------------------------------------------ 16886:S 18 May 2024 17:37:58.437 # !!! Software Failure. Press left mouse button to continue 16886:S 18 May 2024 17:37:58.437 # Guru Meditation: Replica was unable to write command to disk. #server.c:3890
------ STACK TRACE ------
Backtrace: /usr/bin/redis-server 0.0.0.0:6379(processCommand+0x79f)[0x55a138a865df] /usr/bin/redis-server 0.0.0.0:6379(processInputBuffer+0x107)[0x55a138aa2917] /usr/bin/redis-server 0.0.0.0:6379(readQueryFromClient+0x318)[0x55a138aa2e58] /usr/bin/redis-server 0.0.0.0:6379(+0x17165c)[0x55a138b7665c] /usr/bin/redis-server 0.0.0.0:6379(aeProcessEvents+0x1e2)[0x55a138a7c2c2] /usr/bin/redis-server 0.0.0.0:6379(aeMain+0x1d)[0x55a138a7c5fd] /usr/bin/redis-server 0.0.0.0:6379(main+0x360)[0x55a138a73e80] /lib64/libc.so.6(+0x3feb0)[0x7fbef203feb0] /lib64/libc.so.6(__libc_start_main+0x80)[0x7fbef203ff60] /usr/bin/redis-server 0.0.0.0:6379(_start+0x25)[0x55a138a74565]
------ INFO OUTPUT ------
Server
redis_version:7.0.12 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:f05f74181a461f8a redis_mode:standalone os:Linux 5.14.0-362.24.1.el9_3.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.1 process_id:16886 process_supervised:systemd run_id:3652a80751d87a5650b157a9b38d1503f12fc7b5 tcp_port:6379 server_time_usec:1716046678436036 uptime_in_seconds:2049619 uptime_in_days:23 hz:10 configured_hz:10 lru_clock:4770646 executable:/usr/bin/redis-server config_file:/etc/redis/redis.conf io_threads_active:0
Clients
connected_clients:11 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:20480 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0
Memory
used_memory:36766572832 used_memory_human:34.24G used_memory_rss:36651757568 used_memory_rss_human:34.13G used_memory_peak:36784468960 used_memory_peak_human:34.26G used_memory_peak_perc:99.95% used_memory_overhead:620491024 used_memory_startup:862632 used_memory_dataset:36146081808 used_memory_dataset_perc:98.31% allocator_allocated:36766720024 allocator_active:36771098624 allocator_resident:37209387008 total_system_memory:52464754688 total_system_memory_human:48.86G used_memory_lua:31744 used_memory_vm_eval:31744 used_memory_lua_human:31.00K used_memory_scripts_eval:0 number_of_cached_scripts:0 number_of_functions:0 number_of_libraries:0 used_memory_vm_functions:32768 used_memory_vm_total:64512 used_memory_vm_total_human:63.00K used_memory_functions:184 used_memory_scripts:184 used_memory_scripts_human:184B maxmemory:42949672960 maxmemory_human:40.00G maxmemory_policy:volatile-ttl allocator_frag_ratio:1.00 allocator_frag_bytes:4378600 allocator_rss_ratio:1.01 allocator_rss_bytes:438288384 rss_overhead_ratio:0.99 rss_overhead_bytes:-557629440 mem_fragmentation_ratio:1.00 mem_fragmentation_bytes:-114813200 mem_not_counted_for_evict:13472 mem_replication_backlog:1066224 mem_total_replication_buffers:1066208 mem_clients_slaves:0 mem_clients_normal:215488 mem_cluster_links:0 mem_aof_buffer:0 mem_allocator:jemalloc-5.2.1 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0
Persistence
loading:0 async_loading:0 current_cow_peak:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:83842 rdb_bgsave_in_progress:0 rdb_last_save_time:1716046123 rdb_last_bgsave_status:err rdb_last_bgsave_time_sec:494 rdb_current_bgsave_time_sec:-1 rdb_saves:3987 rdb_last_cow_size:145416192 rdb_last_load_keys_expired:0 rdb_last_load_keys_loaded:1820668 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_rewrites:0 aof_rewrites_consecutive_failures:0 aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:29315977 total_commands_processed:725190990 instantaneous_ops_per_sec:496 total_net_input_bytes:120380326392 total_net_output_bytes:109210186001 total_net_repl_input_bytes:76777739780 total_net_repl_output_bytes:0 instantaneous_input_kbps:47.68 instantaneous_output_kbps:21.89 instantaneous_input_repl_kbps:16.89 instantaneous_output_repl_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:426493285 keyspace_misses:18244237 pubsub_channels:1 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:856381 total_forks:3987 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:5484 dump_payload_sanitizations:0 total_reads_processed:661053299 total_writes_processed:561920760 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:4184073 reply_buffer_expands:612366
Replication
role:slave master_host:10.13.38.148 master_port:6379 master_link_status:up master_last_io_seconds_ago:0 master_sync_in_progress:0 slave_read_repl_offset:54769285133 slave_repl_offset:54769284731 slave_priority:100 slave_read_only:1 replica_announced:1 connected_slaves:0 master_failover_state:no-failover master_replid:4c5398b41765fbf5e0041a5cff19a0581cd16d01 master_replid2:b2e8d8d0904eda43b2b8259e1e1cac6cceef4b0a master_repl_offset:54769284731 second_repl_offset:36018919782 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:54768233876 repl_backlog_histlen:1050856
CPU
used_cpu_sys:40010.436250 used_cpu_user:24854.856420 used_cpu_sys_children:143388.891904 used_cpu_user_children:1395613.069525 used_cpu_sys_main_thread:39952.302710 used_cpu_user_main_thread:24826.489939
Modules
Commandstats
cmdstat_zrange:calls=2859,usec=772442453,usec_per_call=270179.25,rejected_calls=1,failed_calls=0 cmdstat_latency|latest:calls=6,usec=16,usec_per_call=2.67,rejected_calls=0,failed_calls=0 cmdstat_hdel:calls=12076000,usec=145043577,usec_per_call=12.01,rejected_calls=0,failed_calls=0 cmdstat_info:calls=846076,usec=297637651,usec_per_call=351.79,rejected_calls=0,failed_calls=0 cmdstat_hget:calls=12891011,usec=154422419,usec_per_call=11.98,rejected_calls=0,failed_calls=0 cmdstat_zrem:calls=7369735,usec=41586346,usec_per_call=5.64,rejected_calls=0,failed_calls=0 cmdstat_set:calls=18505972,usec=611987829,usec_per_call=33.07,rejected_calls=1,failed_calls=0 cmdstat_auth:calls=29302173,usec=289659479,usec_per_call=9.89,rejected_calls=0,failed_calls=0 cmdstat_srem:calls=1875700,usec=47671715,usec_per_call=25.42,rejected_calls=0,failed_calls=0 cmdstat_exec:calls=1259274,usec=42668948,usec_per_call=33.88,rejected_calls=0,failed_calls=142 cmdstat_hexists:calls=1966544,usec=21849983,usec_per_call=11.11,rejected_calls=0,failed_calls=0 cmdstat_hgetall:calls=297813912,usec=2789077662,usec_per_call=9.37,rejected_calls=0,failed_calls=0 cmdstat_quit:calls=13756,usec=9293,usec_per_call=0.68,rejected_calls=0,failed_calls=0 cmdstat_slowlog|get:calls=6,usec=37,usec_per_call=6.17,rejected_calls=0,failed_calls=0 cmdstat_slowlog|len:calls=6,usec=4,usec_per_call=0.67,rejected_calls=0,failed_calls=0 cmdstat_ping:calls=6077891,usec=15438644,usec_per_call=2.54,rejected_calls=2506,failed_calls=0 cmdstat_incr:calls=22311233,usec=168461766,usec_per_call=7.55,rejected_calls=0,failed_calls=0 cmdstat_config|get:calls=6,usec=1323,usec_per_call=220.50,rejected_calls=0,failed_calls=0 cmdstat_config|rewrite:calls=5,usec=50418,usec_per_call=10083.60,rejected_calls=0,failed_calls=0 cmdstat_select:calls=29067546,usec=54213043,usec_per_call=1.87,rejected_calls=0,failed_calls=0 cmdstat_client|kill:calls=10,usec=1030,usec_per_call=103.00,rejected_calls=0,failed_calls=0 cmdstat_client|setname:calls=15737,usec=36360,usec_per_call=2.31,rejected_calls=0,failed_calls=0 cmdstat_get:calls=90148693,usec=590493180,usec_per_call=6.55,rejected_calls=2510,failed_calls=0 cmdstat_slaveof:calls=5,usec=2910,usec_per_call=582.00,rejected_calls=142,failed_calls=0 cmdstat_exists:calls=2547170,usec=9261162,usec_per_call=3.64,rejected_calls=176,failed_calls=0 cmdstat_subscribe:calls=160,usec=295,usec_per_call=1.84,rejected_calls=0,failed_calls=0 cmdstat_publish:calls=6026494,usec=115568802,usec_per_call=19.18,rejected_calls=0,failed_calls=0 cmdstat_zadd:calls=7850829,usec=635140806,usec_per_call=80.90,rejected_calls=0,failed_calls=0 cmdstat_multi:calls=1259275,usec=1008203,usec_per_call=0.80,rejected_calls=0,failed_calls=0 cmdstat_hset:calls=86428287,usec=830155119,usec_per_call=9.61,rejected_calls=0,failed_calls=0 cmdstat_srandmember:calls=844299,usec=13648961,usec_per_call=16.17,rejected_calls=0,failed_calls=0 cmdstat_sismember:calls=844298,usec=5782886,usec_per_call=6.85,rejected_calls=0,failed_calls=0 cmdstat_hincrby:calls=14307336,usec=360183608,usec_per_call=25.17,rejected_calls=0,failed_calls=0 cmdstat_role:calls=29067510,usec=148660371,usec_per_call=5.11,rejected_calls=0,failed_calls=0 cmdstat_del:calls=6792440,usec=91974871,usec_per_call=13.54,rejected_calls=0,failed_calls=0 cmdstat_scard:calls=37678736,usec=219223604,usec_per_call=5.82,rejected_calls=0,failed_calls=0
Errorstats
errorstat_ERR:count=6 errorstat_EXECABORT:count=142 errorstat_LOADING:count=5335 errorstat_READONLY:count=1
Latencystats
latency_percentiles_usec_zrange:p50=329252.863,p99=603979.775,p99.9=692060.159 latency_percentiles_usec_latency|latest:p50=2.007,p99=8.031,p99.9=8.031 latency_percentiles_usec_hdel:p50=7.007,p99=55.039,p99.9=165.887 latency_percentiles_usec_info:p50=205.823,p99=1613.823,p99.9=6422.527 latency_percentiles_usec_hget:p50=9.023,p99=46.079,p99.9=140.287 latency_percentiles_usec_zrem:p50=3.007,p99=31.103,p99.9=82.431 latency_percentiles_usec_set:p50=27.007,p99=122.367,p99.9=454.655 latency_percentiles_usec_auth:p50=7.007,p99=44.031,p99.9=239.615 latency_percentiles_usec_srem:p50=22.015,p99=72.191,p99.9=164.863 latency_percentiles_usec_exec:p50=26.111,p99=126.463,p99.9=301.055 latency_percentiles_usec_hexists:p50=9.023,p99=45.055,p99.9=106.495 latency_percentiles_usec_hgetall:p50=8.031,p99=38.143,p99.9=104.447 latency_percentiles_usec_quit:p50=1.003,p99=2.007,p99.9=24.063 latency_percentiles_usec_slowlog|get:p50=5.023,p99=10.047,p99.9=10.047 latency_percentiles_usec_slowlog|len:p50=1.003,p99=1.003,p99.9=1.003 latency_percentiles_usec_ping:p50=2.007,p99=23.039,p99.9=87.039 latency_percentiles_usec_incr:p50=7.007,p99=30.079,p99.9=76.287 latency_percentiles_usec_config|get:p50=236.543,p99=262.143,p99.9=262.143 latency_percentiles_usec_config|rewrite:p50=7471.103,p99=19660.799,p99.9=19660.799 latency_percentiles_usec_select:p50=2.007,p99=6.015,p99.9=36.095 latency_percentiles_usec_client|kill:p50=92.159,p99=184.319,p99.9=184.319 latency_percentiles_usec_client|setname:p50=2.007,p99=14.015,p99.9=51.199 latency_percentiles_usec_get:p50=6.015,p99=26.111,p99.9=73.215 latency_percentiles_usec_slaveof:p50=647.167,p99=704.511,p99.9=704.511 latency_percentiles_usec_exists:p50=3.007,p99=9.023,p99.9=48.127 latency_percentiles_usec_subscribe:p50=1.003,p99=10.047,p99.9=31.103 latency_percentiles_usec_publish:p50=13.055,p99=119.295,p99.9=471.039 latency_percentiles_usec_zadd:p50=74.239,p99=197.631,p99.9=444.415 latency_percentiles_usec_multi:p50=1.003,p99=2.007,p99.9=23.039 latency_percentiles_usec_hset:p50=2.007,p99=68.095,p99.9=141.311 latency_percentiles_usec_srandmember:p50=13.055,p99=52.223,p99.9=129.023 latency_percentiles_usec_sismember:p50=6.015,p99=28.031,p99.9=74.239 latency_percentiles_usec_hincrby:p50=20.095,p99=93.183,p99.9=419.839 latency_percentiles_usec_role:p50=5.023,p99=20.095,p99.9=60.159 latency_percentiles_usec_del:p50=8.031,p99=74.239,p99.9=309.247 latency_percentiles_usec_scard:p50=5.023,p99=21.119,p99.9=64.255
Cluster
cluster_enabled:0
Keyspace
db0:keys=9700468,expires=2606484,avg_ttl=0
------ CLIENT LIST OUTPUT ------ id=29308112 addr=10.13.38.146:34164 laddr=10.13.38.146:6379 fd=13 name=sentinel-8b90a335-cmd age=373 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23296 events=r cmd=ping user=default redir=-1 resp=2 id=29309462 addr=10.13.36.129:46564 laddr=10.13.38.146:6379 fd=12 name= age=311 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=53 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=hgetall user=default redir=-1 resp=2 id=18526 addr=10.13.38.148:6379 laddr=10.13.38.146:48956 fd=11 name= age=2046840 idle=0 flags=Mx db=0 sub=0 psub=0 ssub=0 multi=0 qbuf=402 qbuf-free=20040 argv-mem=117 multi-mem=0 rbs=1024 rbp=45 obl=0 oll=0 omem=0 tot-mem=22405 events=r cmd=del user=(superuser) redir=-1 resp=2 id=29315983 addr=10.13.36.129:48770 laddr=10.13.38.146:6379 fd=18 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37632 events=r cmd=get user=default redir=-1 resp=2 id=29315985 addr=10.13.36.129:59978 laddr=10.13.38.146:6379 fd=19 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=16384 rbp=0 obl=0 oll=0 omem=0 tot-mem=37632 events=r cmd=get user=default redir=-1 resp=2 id=29315980 addr=10.13.36.195:51336 laddr=10.13.38.146:6379 fd=15 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=get user=default redir=-1 resp=2 id=6185520 addr=10.13.38.147:35742 laddr=10.13.38.146:6379 fd=14 name=sentinel-76a03ce8-pubsub age=1404646 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=1856 events=r cmd=subscribe user=default redir=-1 resp=2 id=29291111 addr=10.13.38.148:36412 laddr=10.13.38.146:6379 fd=21 name=sentinel-ebd169d8-cmd age=1152 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2 id=6185509 addr=10.13.38.148:39524 laddr=10.13.38.146:6379 fd=8 name=sentinel-ebd169d8-pubsub age=1404649 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=1856 events=r cmd=subscribe user=default redir=-1 resp=2 id=6185506 addr=10.13.38.146:31346 laddr=10.13.38.146:6379 fd=10 name=sentinel-8b90a335-pubsub age=1404650 idle=0 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=138 obl=0 oll=0 omem=0 tot-mem=1856 events=r cmd=subscribe user=default redir=-1 resp=2 id=29279010 addr=10.13.38.147:40338 laddr=10.13.38.146:6379 fd=16 name=sentinel-76a03ce8-cmd age=1701 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22272 events=r cmd=ping user=default redir=-1 resp=2
------ CURRENT CLIENT INFO ------ id=18526 addr=10.13.38.148:6379 laddr=10.13.38.146:48956 fd=11 name= age=2046840 idle=0 flags=Mx db=0 sub=0 psub=0 ssub=0 multi=0 qbuf=402 qbuf-free=20040 argv-mem=117 multi-mem=0 rbs=1024 rbp=45 obl=0 oll=0 omem=0 tot-mem=22405 events=r cmd=del user=(superuser) redir=-1 resp=2 argv[0]: '"DEL"' argv[1]: '"pb-rewards-rabattcode-gluecksrad-mai-de-2024:app-container:pb-rewards-rabattcode-gluecksrad-mai-de-2024-fe-5-q46dd"' 16886:S 18 May 2024 17:37:58.463 # key 'pb-rewards-rabattcode-gluecksrad-mai-de-2024:app-container:pb-rewards-rabattcode-gluecksrad-mai-de-2024-fe-5-q46dd' found in DB containing the following object: 16886:S 18 May 2024 17:37:58.463 # Object type: 0 16886:S 18 May 2024 17:37:58.463 # Object encoding: 0 16886:S 18 May 2024 17:37:58.463 # Object refcount: 1
------ MODULES INFO OUTPUT ------
------ CONFIG DEBUG OUTPUT ------ proto-max-bulk-len 512mb io-threads-do-reads no sanitize-dump-payload no repl-diskless-sync yes client-query-buffer-limit 1gb lazyfree-lazy-user-del no repl-diskless-load disabled slave-read-only yes replica-read-only yes activedefrag no lazyfree-lazy-expire no lazyfree-lazy-server-del no list-compress-depth 0 io-threads 1 lazyfree-lazy-eviction no lazyfree-lazy-user-flush no
------ FAST MEMORY TEST ------ 16886:S 18 May 2024 17:37:58.479 # Bio thread for job type #0 terminated 16886:S 18 May 2024 17:37:58.479 # Bio thread for job type #1 terminated 16886:S 18 May 2024 17:37:58.481 # Bio thread for job type #2 terminated *** Preparing to test memory region 55a138d2f000 (2306048 bytes) *** Preparing to test memory region 55a139a4c000 (270336 bytes) *** Preparing to test memory region 7fb46a000000 (27950977024 bytes) *** Preparing to test memory region 7fbaf1a00000 (8388608 bytes) *** Preparing to test memory region 7fbaf2200000 (16747855872 bytes) *** Preparing to test memory region 7fbed8700000 (378535936 bytes) *** Preparing to test memory region 7fbeef1fd000 (8388608 bytes) *** Preparing to test memory region 7fbeef9fe000 (8388608 bytes) *** Preparing to test memory region 7fbef01ff000 (8388608 bytes) *** Preparing to test memory region 7fbef0a00000 (8388608 bytes) *** Preparing to test memory region 7fbef1600000 (8388608 bytes) *** Preparing to test memory region 7fbef1f25000 (16384 bytes) *** Preparing to test memory region 7fbef21fc000 (53248 bytes) *** Preparing to test memory region 7fbef220c000 (8192 bytes) *** Preparing to test memory region 7fbef2234000 (8192 bytes) *** Preparing to test memory region 7fbef224f000 (4096 bytes) *** Preparing to test memory region 7fbef228e000 (4096 bytes) *** Preparing to test memory region 7fbef22ba000 (4096 bytes) *** Preparing to test memory region 7fbef23f3000 (12288 bytes) *** Preparing to test memory region 7fbef282f000 (12288 bytes) *** Preparing to test memory region 7fbef29b5000 (4096 bytes) *** Preparing to test memory region 7fbef2a96000 (8192 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. === Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
Additional information
- RedHat 9.4
- Steps to reproduce (if any): none, redis crashed during normal operations
Comment From: sundb
@noris-spfab can you check whether there are any errors about rdb saving in the log file.
like can't save in background: fork, Background saving error.
Comment From: noris-spfab
Hello, yes, there was the following message immediately before:
1339361:C 18 May 2024 17:28:41.583 * DB saved on disk 1339361:C 18 May 2024 17:28:42.423 * Fork CoW for RDB: current 138 MB, peak 138 MB, average 88 MB 16886:S 18 May 2024 17:28:43.462 * Background saving terminated with success 16886:S 18 May 2024 17:29:44.021 * 10000 changes in 60 seconds. Saving... 16886:S 18 May 2024 17:29:44.878 * Background saving started by pid 1339680 1339680:C 18 May 2024 17:37:54.291 # Write error saving DB on disk(rdbSaveRio): No space left on device 16886:S 18 May 2024 17:37:58.431 # Background saving error
=== REDIS BUG REPORT START: Cut & paste starting from here === 16886:S 18 May 2024 17:37:58.437 # ------------------------------------------------ 16886:S 18 May 2024 17:37:58.437 # !!! Software Failure. Press left mouse button to continue 16886:S 18 May 2024 17:37:58.437 # Guru Meditation: Replica was unable to write command to disk. #server.c:3890
------ STACK TRACE ------
Backtrace: /usr/bin/redis-server 0.0.0.0:6379(processCommand+0x79f)[0x55a138a865df] /usr/bin/redis-server 0.0.0.0:6379(processInputBuffer+0x107)[0x55a138aa2917] /usr/bin/redis-server 0.0.0.0:6379(readQueryFromClient+0x318)[0x55a138aa2e58] /usr/bin/redis-server 0.0.0.0:6379(+0x17165c)[0x55a138b7665c] /usr/bin/redis-server 0.0.0.0:6379(aeProcessEvents+0x1e2)[0x55a138a7c2c2] /usr/bin/redis-server 0.0.0.0:6379(aeMain+0x1d)[0x55a138a7c5fd] /usr/bin/redis-server 0.0.0.0:6379(main+0x360)[0x55a138a73e80] /lib64/libc.so.6(+0x3feb0)[0x7fbef203feb0] /lib64/libc.so.6(__libc_start_main+0x80)[0x7fbef203ff60] /usr/bin/redis-server 0.0.0.0:6379(_start+0x25)[0x55a138a74565]
Comment From: sundb
from the log # Write error saving DB on disk(rdbSaveRio): No space left on device
please check the space of the HDD.
Comment From: noris-spfab
ok, so the bug report is more like byproduct of redis that cannot write to disk and crashing with that messages in the log and not a "real" bug?
Comment From: sundb
this is a panic that Reids throws out, not a bug.
Comment From: noris-spfab
thanks for the clarification.
Comment From: sundb
@noris-spfab you also can disable replica-ignore-disk-write-errors config to prevent Redis to throw this panic when writing logs failed in replica.