### Crash report
We are running redis cluster 5.0.11 with 31 master + 62 slave nodes and 20+ GiB data per node. Running in cluster mode
What is the reason? How do I reproduce and fix it? Thanks a lot!
481472:C 10 Jan 2025 00:20:09.912 # Redis version=5.0.11, bits=64, commit=00000000, modified=0, pid=481472, just started
481473:S 13 Jan 2025 08:26:37.242 * Starting automatic rewriting of AOF on 100% growth
481473:S 13 Jan 2025 08:26:38.095 * Background append only file rewriting started by pid 166120
166120:C 13 Jan 2025 08:27:15.400 # Out Of Memory allocating 5283932191711312175 bytes!
=== REDIS BUG REPORT START: Cut & paste starting from here ===
166120:C 13 Jan 2025 08:27:15.400 # ------------------------------------------------
166120:C 13 Jan 2025 08:27:15.400 # !!! Software Failure. Press left mouse button to continue
166120:C 13 Jan 2025 08:27:15.400 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:4107
166120:C 13 Jan 2025 08:27:15.400 # (forcing SIGSEGV in order to print the stack trace)
166120:C 13 Jan 2025 08:27:15.400 # ------------------------------------------------
166120:C 13 Jan 2025 08:27:15.400 # Redis 5.0.11 crashed by signal: 11
166120:C 13 Jan 2025 08:27:15.400 # Crashed running the instruction at: 0x46fd77
166120:C 13 Jan 2025 08:27:15.400 # Accessing address: 0xffffffffffffffff
166120:C 13 Jan 2025 08:27:15.400 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-aof-rewrite 10.125.112.81:7008 [cluster](_serverPanic+0x117)[0x46fd77]
Backtrace:
redis-aof-rewrite 10.125.112.81:7008 [cluster](logStackTrace+0x29)[0x471a49]
redis-aof-rewrite 10.125.112.81:7008 [cluster](sigsegvHandler+0xac)[0x4720ec]
/lib64/libpthread.so.0(+0x12dd0)[0x7f4e5044bdd0]
redis-aof-rewrite 10.125.112.81:7008 [cluster](_serverPanic+0x117)[0x46fd77]
redis-aof-rewrite 10.125.112.81:7008 [cluster](redisOutOfMemoryHandler+0x2e)[0x42e96e]
redis-aof-rewrite 10.125.112.81:7008 [cluster](zmalloc+0x49)[0x437ee9]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rdbSaveLzfStringObject+0x33)[0x44dd33]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rdbSaveRawString+0x17e)[0x44e0ee]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rdbSaveObject+0x26e)[0x44f25e]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rdbSaveKeyValuePair+0x15a)[0x44f9ca]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rdbSaveRio+0x276)[0x4500a6]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rewriteAppendOnlyFile+0x126)[0x46c9d6]
redis-aof-rewrite 10.125.112.81:7008 [cluster](rewriteAppendOnlyFileBackground+0x144)[0x46d024]
redis-aof-rewrite 10.125.112.81:7008 [cluster](serverCron+0x6e0)[0x431a80]
redis-aof-rewrite 10.125.112.81:7008 [cluster](aeProcessEvents+0x32f)[0x42b28f]
redis-aof-rewrite 10.125.112.81:7008 [cluster](aeMain+0x2b)[0x42b4cb]
redis-aof-rewrite 10.125.112.81:7008 [cluster](main+0x4c0)[0x428330]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f4e5009a6a3]
redis-aof-rewrite 10.125.112.81:7008 [cluster][0x42858a]
------ INFO OUTPUT ------
# Server
redis_version:5.0.11
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:100f2fc458ace29c
redis_mode:cluster
os:Linux 4.19.0-240.23.35.el8_2.bclinux.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:166120
run_id:54391c2cd290ba6b052b720020d8de94d12240d6
tcp_port:7008
uptime_in_seconds:288388
uptime_in_days:3
hz:10
configured_hz:10
lru_clock:8674749
executable:/redis-5.0.11/bin/redis-server
config_file:/redis-5.0.11/conf/7008.conf
# Clients
connected_clients:104
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
# Memory
used_memory:16774105648
used_memory_human:15.62G
used_memory_rss:17698484224
used_memory_rss_human:16.48G
used_memory_peak:17898536344
used_memory_peak_human:16.67G
used_memory_peak_perc:93.72%
used_memory_overhead:537272340
used_memory_startup:1697896
used_memory_dataset:16236833308
used_memory_dataset_perc:96.81%
allocator_allocated:16759881120
allocator_active:17452265472
allocator_resident:17705340928
total_system_memory:540596719616
total_system_memory_human:503.47G
used_memory_lua:41984
used_memory_lua_human:41.00K
used_memory_scripts:424
used_memory_scripts_human:424B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.04
allocator_frag_bytes:692384352
allocator_rss_ratio:1.01
allocator_rss_bytes:253075456
rss_overhead_ratio:1.00
rss_overhead_bytes:-6856704
mem_fragmentation_ratio:1.06
mem_fragmentation_bytes:938978064
mem_not_counted_for_evict:1372
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:1792664
mem_aof_buffer:1372
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:65015702
rdb_bgsave_in_progress:0
rdb_last_save_time:1736439609
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:283
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:256040960
aof_current_size:17967921663
aof_base_size:8983926683
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:450097
total_commands_processed:42774566
instantaneous_ops_per_sec:792
total_net_input_bytes:17950943669
total_net_output_bytes:12971008109
instantaneous_input_kbps:380.24
instantaneous_output_kbps:151.49
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:1
expired_stale_perc:0.25
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1053057
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:slave
master_host:10.253.11.85
master_port:7008
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:1192389247127
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:ec16e74d03968027615e36e57fef78faefef8e1b
master_replid2:abaa1edceb6e178d399fd7ee753a28fd771929a1
master_repl_offset:1192389247127
second_repl_offset:1183494422806
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1192388198552
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:5.847499
used_cpu_user:31.097773
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Commandstats
cmdstat_hset:calls=9286812,usec=105644176,usec_per_call=11.38
cmdstat_zremrangebyscore:calls=259,usec=13887,usec_per_call=53.62
cmdstat_select:calls=5,usec=5,usec_per_call=1.00
cmdstat_config:calls=9619,usec=1004286,usec_per_call=104.41
cmdstat_zadd:calls=287,usec=15295,usec_per_call=53.29
cmdstat_ping:calls=28736,usec=6933,usec_per_call=0.24
cmdstat_set:calls=10585236,usec=74002227,usec_per_call=6.99
cmdstat_del:calls=3091596,usec=117969754,usec_per_call=38.16
cmdstat_command:calls=37,usec=43590,usec_per_call=1178.11
cmdstat_expire:calls=2715461,usec=3653706,usec_per_call=1.35
cmdstat_dbsize:calls=3,usec=3,usec_per_call=1.00
cmdstat_slowlog:calls=19190,usec=97332,usec_per_call=5.07
cmdstat_client:calls=827849,usec=117273912,usec_per_call=141.66
cmdstat_info:calls=9631,usec=1890034,usec_per_call=196.24
cmdstat_scan:calls=2,usec=195,usec_per_call=97.50
cmdstat_cluster:calls=416001,usec=2189933757,usec_per_call=5264.25
cmdstat_hdel:calls=2932,usec=93411,usec_per_call=31.86
cmdstat_exec:calls=2715461,usec=58718835,usec_per_call=21.62
cmdstat_latency:calls=9610,usec=23374,usec_per_call=2.43
cmdstat_auth:calls=429672,usec=892184,usec_per_call=2.08
cmdstat_multi:calls=2715461,usec=804825,usec_per_call=0.30
cmdstat_setnx:calls=312,usec=6459,usec_per_call=20.70
cmdstat_pexpire:calls=6316814,usec=12567249,usec_per_call=1.99
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=9432653,expires=534123,avg_ttl=94314850
------ CLIENT LIST OUTPUT ------
id=450063 addr=10.112.204.27:53946 fd=171 name=lettuce#ClusterTopologyRefresh age=18 idle=18 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450055 addr=10.112.204.7:40716 fd=136 name=lettuce#ClusterTopologyRefresh age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450086 addr=10.112.204.19:44924 fd=18 name=lettuce#ClusterTopologyRefresh age=9 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449998 addr=10.112.204.21:64264 fd=156 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450097 addr=10.112.204.21:64388 fd=175 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450089 addr=10.112.204.27:53958 fd=148 name=lettuce#ClusterTopologyRefresh age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449999 addr=10.112.204.26:45596 fd=159 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450006 addr=10.112.204.22:58978 fd=87 name=lettuce#ClusterTopologyRefresh age=53 idle=53 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450022 addr=10.112.204.12:35718 fd=19 name=lettuce#ClusterTopologyRefresh age=43 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450067 addr=10.112.204.23:39170 fd=39 name=lettuce#ClusterTopologyRefresh age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449997 addr=10.112.204.23:39128 fd=154 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450038 addr=10.112.204.22:59008 fd=28 name=lettuce#ClusterTopologyRefresh age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450059 addr=10.112.204.2:64436 fd=48 name=lettuce#ClusterTopologyRefresh age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450095 addr=10.112.204.22:59106 fd=151 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450096 addr=10.112.204.21:64384 fd=152 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450039 addr=10.112.204.21:64308 fd=124 name=lettuce#ClusterTopologyRefresh age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450088 addr=10.112.204.26:45690 fd=54 name=lettuce#ClusterTopologyRefresh age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450080 addr=10.112.204.23:39184 fd=145 name=lettuce#ClusterTopologyRefresh age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450070 addr=10.112.204.23:39176 fd=45 name=lettuce#ClusterTopologyRefresh age=13 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450084 addr=10.112.204.49:60650 fd=147 name=lettuce#ClusterTopologyRefresh age=10 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450040 addr=10.112.204.21:64324 fd=131 name=lettuce#ClusterTopologyRefresh age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450008 addr=10.112.204.22:58994 fd=88 name=lettuce#ClusterTopologyRefresh age=51 idle=51 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450020 addr=10.112.204.27:53874 fd=114 name=lettuce#ClusterTopologyRefresh age=44 idle=44 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450057 addr=10.112.204.22:59054 fd=21 name=lettuce#ClusterTopologyRefresh age=21 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450058 addr=10.112.204.22:59062 fd=33 name= age=21 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=450007 addr=10.112.204.21:64280 fd=16 name=lettuce#ClusterTopologyRefresh age=52 idle=52 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450030 addr=10.112.204.26:45620 fd=83 name=lettuce#ClusterTopologyRefresh age=39 idle=39 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450001 addr=10.112.204.27:53814 fd=106 name=lettuce#ClusterTopologyRefresh age=59 idle=59 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450005 addr=10.112.204.26:45612 fd=10 name=lettuce#ClusterTopologyRefresh age=55 idle=55 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450003 addr=10.112.204.27:53828 fd=57 name=lettuce#ClusterTopologyRefresh age=58 idle=58 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450065 addr=10.112.204.22:59098 fd=172 name=lettuce#ClusterTopologyRefresh age=17 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450093 addr=10.112.204.23:39202 fd=115 name=lettuce#ClusterTopologyRefresh age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450042 addr=10.112.204.21:64338 fd=168 name=lettuce#ClusterTopologyRefresh age=31 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450079 addr=10.112.204.26:45668 fd=143 name=lettuce#ClusterTopologyRefresh age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450043 addr=10.112.204.26:45650 fd=170 name=lettuce#ClusterTopologyRefresh age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450010 addr=10.112.204.27:53840 fd=22 name=lettuce#ClusterTopologyRefresh age=49 idle=49 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450036 addr=10.112.204.23:39162 fd=123 name=lettuce#ClusterTopologyRefresh age=33 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450019 addr=10.112.204.18:36350 fd=31 name=lettuce#ClusterTopologyRefresh age=44 idle=44 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450100 addr=10.112.204.21:64392 fd=178 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450101 addr=10.112.204.26:45724 fd=179 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450083 addr=10.112.204.26:45678 fd=99 name=lettuce#ClusterTopologyRefresh age=10 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450047 addr=10.112.204.21:64346 fd=133 name=lettuce#ClusterTopologyRefresh age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450011 addr=10.112.204.27:53850 fd=60 name=lettuce#ClusterTopologyRefresh age=47 idle=47 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450012 addr=10.112.204.27:53852 fd=90 name=lettuce#ClusterTopologyRefresh age=47 idle=47 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450013 addr=10.112.204.27:53864 fd=110 name=lettuce#ClusterTopologyRefresh age=47 idle=47 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450004 addr=10.112.204.21:64268 fd=62 name=lettuce#ClusterTopologyRefresh age=57 idle=57 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450034 addr=10.112.204.27:53918 fd=121 name=lettuce#ClusterTopologyRefresh age=38 idle=38 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450028 addr=10.112.204.5:49104 fd=71 name=lettuce#ClusterTopologyRefresh age=40 idle=40 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450023 addr=10.112.204.23:39140 fd=61 name=lettuce#ClusterTopologyRefresh age=42 idle=40 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450099 addr=10.112.204.23:39204 fd=176 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450076 addr=10.112.204.12:35726 fd=47 name=lettuce#ClusterTopologyRefresh age=13 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450015 addr=10.112.204.21:64288 fd=29 name=lettuce#ClusterTopologyRefresh age=46 idle=46 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450016 addr=10.112.204.21:64292 fd=113 name=lettuce#ClusterTopologyRefresh age=46 idle=46 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450029 addr=10.112.204.19:44900 fd=35 name=lettuce#ClusterTopologyRefresh age=39 idle=39 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449996 addr=10.112.204.26:45592 fd=102 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450026 addr=10.112.204.23:39156 fd=164 name=lettuce#ClusterTopologyRefresh age=41 idle=41 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=1269 addr=10.253.112.85:7008 fd=515 name= age=287607 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=pexpire
id=450041 addr=10.112.204.23:39166 fd=167 name=lettuce#ClusterTopologyRefresh age=31 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450053 addr=10.112.204.22:59042 fd=135 name=lettuce#ClusterTopologyRefresh age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449720 addr=10.112.204.23:38684 fd=14 name= age=251 idle=251 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=450018 addr=10.112.204.23:39130 fd=23 name=lettuce#ClusterTopologyRefresh age=44 idle=44 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449894 addr=10.112.204.23:38950 fd=55 name= age=131 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=450033 addr=10.112.204.26:45624 fd=120 name=lettuce#ClusterTopologyRefresh age=38 idle=38 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450021 addr=10.112.204.23:39132 fd=125 name=lettuce#ClusterTopologyRefresh age=43 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450027 addr=10.112.204.27:53902 fd=66 name=lettuce#ClusterTopologyRefresh age=40 idle=40 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450069 addr=10.112.204.27:53954 fd=174 name=lettuce#ClusterTopologyRefresh age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450009 addr=10.112.204.2:63994 fd=13 name=lettuce#ClusterTopologyRefresh age=49 idle=49 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449995 addr=10.112.204.21:64254 fd=101 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450024 addr=10.112.204.27:53886 fd=63 name=lettuce#ClusterTopologyRefresh age=41 idle=41 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450025 addr=10.112.204.23:39142 fd=157 name=lettuce#ClusterTopologyRefresh age=41 idle=41 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449993 addr=10.112.204.22:58962 fd=86 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450056 addr=10.112.204.22:59048 fd=139 name=lettuce#ClusterTopologyRefresh age=21 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450064 addr=10.112.204.21:64376 fd=81 name=lettuce#ClusterTopologyRefresh age=17 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450049 addr=10.112.204.6:57194 fd=30 name=lettuce#ClusterTopologyRefresh age=27 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=449994 addr=10.112.204.21:64238 fd=100 name=lettuce#ClusterTopologyRefresh age=61 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450044 addr=10.112.204.22:59020 fd=15 name=lettuce#ClusterTopologyRefresh age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450045 addr=10.112.204.22:59028 fd=95 name=lettuce#ClusterTopologyRefresh age=30 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450031 addr=10.112.204.27:53910 fd=118 name= age=38 idle=38 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=auth
id=450032 addr=10.112.204.19:44914 fd=41 name=lettuce#ClusterTopologyRefresh age=38 idle=38 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450098 addr=10.112.204.22:59114 fd=85 name=lettuce#ClusterTopologyRefresh age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450094 addr=10.112.204.26:45708 fd=49 name=lettuce#ClusterTopologyRefresh age=2 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450068 addr=10.112.204.18:36352 fd=173 name=lettuce#ClusterTopologyRefresh age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450046 addr=10.112.204.27:53928 fd=132 name=lettuce#ClusterTopologyRefresh age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450051 addr=10.112.204.26:45654 fd=44 name=lettuce#ClusterTopologyRefresh age=25 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450037 addr=10.112.204.26:45636 fd=11 name=lettuce#ClusterTopologyRefresh age=32 idle=32 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450078 addr=10.112.204.23:39182 fd=82 name=lettuce#ClusterTopologyRefresh age=12 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450050 addr=10.112.204.6:57200 fd=42 name=lettuce#ClusterTopologyRefresh age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450087 addr=10.112.204.26:45682 fd=52 name=lettuce#ClusterTopologyRefresh age=9 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450092 addr=10.112.204.26:45692 fd=84 name=lettuce#ClusterTopologyRefresh age=3 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450052 addr=10.112.204.21:64360 fd=65 name=lettuce#ClusterTopologyRefresh age=24 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450090 addr=10.112.204.27:53968 fd=68 name= age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=450091 addr=10.112.204.27:53982 fd=69 name=lettuce#ClusterTopologyRefresh age=8 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450014 addr=10.112.204.22:59004 fd=111 name=lettuce#ClusterTopologyRefresh age=47 idle=46 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450048 addr=10.112.204.21:64358 fd=17 name=lettuce#ClusterTopologyRefresh age=27 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450002 addr=10.112.204.21:64266 fd=160 name=lettuce#ClusterTopologyRefresh age=58 idle=58 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450035 addr=10.112.204.26:45634 fd=32 name=lettuce#ClusterTopologyRefresh age=33 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450081 addr=10.112.204.23:39198 fd=91 name=lettuce#ClusterTopologyRefresh age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450082 addr=10.112.204.49:60638 fd=146 name=lettuce#ClusterTopologyRefresh age=11 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450085 addr=10.112.204.5:46522 fd=158 name=lettuce#ClusterTopologyRefresh age=10 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450000 addr=10.112.204.26:45600 fd=105 name=lettuce#ClusterTopologyRefresh age=60 idle=60 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450054 addr=10.112.204.7:40704 fd=70 name=lettuce#ClusterTopologyRefresh age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450060 addr=10.112.204.22:59070 fd=73 name=lettuce#ClusterTopologyRefresh age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450061 addr=10.112.204.22:59084 fd=80 name=lettuce#ClusterTopologyRefresh age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
id=450062 addr=10.112.204.27:53940 fd=142 name=lettuce#ClusterTopologyRefresh age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
------ REGISTERS ------
166120:C 13 Jan 2025 08:27:15.402 #
RAX:0000000000000000 RBX:000000000052e798
RCX:0000000001856010 RDX:0000000000000000
RDI:0000000001865600 RSI:0000000000000001
RBP:000000000000100b RSP:00007ffeb7f77af0
R8 :000000000185604a R9 :0000000000000000
R10:0000000000000000 R11:0000000000000206
R12:00007f4e407b9a61 R13:00007ffeb7f780d0
R14:49544c553834392e R15:000000000005eb8b
RIP:000000000046fd77 EFL:0000000000010202
CSGSFS:002b000000000033
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77aff) -> 00007f4e50163b74
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77afe) -> 0000000000000d68
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77afd) -> 0000000001858c10
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77afc) -> 5000000000001000
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77afb) -> 0000000000001000
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77afa) -> 0000000000000000
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af9) -> 00007ffeb7f77bf0
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af8) -> 00007ffeb7f77c00
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af7) -> 59524f4d454d2046
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af6) -> 4f2054554f20726f
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af5) -> 6620676e6974726f
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af4) -> 6261207369646552
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af3) -> 00007ffeb7f77c10
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af2) -> 00007ffeb7f77ce0
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af1) -> 0000003000000018
166120:C 13 Jan 2025 08:27:15.402 # (00007ffeb7f77af0) -> 0000000000000002
------ FAST MEMORY TEST ------
*** Preparing to test memory region 790000 (2252800 bytes)
*** Preparing to test memory region 1856000 (135168 bytes)
*** Preparing to test memory region 7f49c2e00000 (2097152 bytes)
*** Preparing to test memory region 7f49c3661000 (8192 bytes)
*** Preparing to test memory region 7f49c386a000 (4096 bytes)
*** Preparing to test memory region 7f49c3abc000 (4096 bytes)
*** Preparing to test memory region 7f49c3f2e000 (4096 bytes)
*** Preparing to test memory region 7f49c43e5000 (8192 bytes)
*** Preparing to test memory region 7f49c47fa000 (24576 bytes)
*** Preparing to test memory region 7f49c4800000 (19096666112 bytes)
*** Preparing to test memory region 7f4e36d49000 (167772160 bytes)
*** Preparing to test memory region 7f4e40d4a000 (8388608 bytes)
*** Preparing to test memory region 7f4e4154b000 (8388608 bytes)
*** Preparing to test memory region 7f4e41d4c000 (8388608 bytes)
*** Preparing to test memory region 7f4e4254c000 (3145728 bytes)
*** Preparing to test memory region 7f4e4f800000 (8388608 bytes)
*** Preparing to test memory region 7f4e50435000 (16384 bytes)
*** Preparing to test memory region 7f4e50655000 (16384 bytes)
*** Preparing to test memory region 7f4e50fff000 (32768 bytes)
*** Preparing to test memory region 7f4e51012000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: _serverPanic (base: 0x46fc60)
Module: redis-aof-rewrite 10.253.112.81:7008 [cluster] (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x46fc60 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
166120:C 13 Jan 2025 08:29:13.574 # dump of function (hexdump of 407 bytes):
5589f5534889fb4881ecd801000084c048898c24380100004c898424400100004c898c244801000074400f298424500100000f298c24600100000f299424700100000f299c24800100000f29a424900100000f29ac24a00100f29b424b00100000f29bc24c0010000488d8424f0010000488d4c2408488d7c2420be000100004889442410488d84242000c744240818000000c744240c300000004889442418e8585dfbff8b05a625320085c07505e8f9f8ffffbe50db5300bf0300000031c0e838ebfbffbe88db5300bf0300000031c0e827ebfbff488d5424204189e84889d9be58ec5300bf0300000031c0e80bebfbffbec8db5300bf0300000031c0e8faeafbffbe50db5300bf0300000031c0e8e9eafbffc60425ffffffff784881c4d80100005b5dc30f1f80000000004157415641554989f541544989cc554889fd534889d3ba040000004889df4881ece80200000fb601488d74241483e00f0fc889442414e865f9ffff4889ef4c89eee8da67fdff410fb614244889c589d083e00f0f84d70000003c010f844f0200003c02
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
481473:S 13 Jan 2025 08:29:16.447 # Background AOF rewrite terminated by signal 11
481473:S 13 Jan 2025 08:29:16.547 * Starting automatic rewriting of AOF on 100% growth
481473:S 13 Jan 2025 08:29:17.235 * Background append only file rewriting started by pid 166659
166659:C 13 Jan 2025 08:29:54.585 # Out Of Memory allocating 5283932191711312175 bytes!
Repeat above...
### Redis config file:
bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 300
daemonize yes
dir /data
supervised no
loglevel notice
logfile "/data/redis.log"
databases 16
always-show-logo yes
save ""
stop-writes-on-bgsave-error no
**rdbcompression yes**
rdbchecksum yes
dbfilename dump.rdb
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
masterauth xxxxxxx
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
**aof-use-rdb-preamble yes**
lua-time-limit 5000
cluster-enabled yes
cluster-config-file /redis/etc/nodes.conf
cluster-node-timeout 15000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
### Additional information
- os: Linux 4.19.0-240.23.35.el8_2.bclinux.x86_64 x86_64
- Steps to reproduce (if any)
Comment From: ShooterIT
you mean your machine has enough memory, but redis crashed with OOM error? could you show the log when redis started, does it have this log?
overcommit_memory is set to 0! Background save may fail under low memory condition.
i suppose it is similar with https://github.com/redis/redis/issues/10234
Comment From: zhuleiandy888
@ShooterIT OS core config： sysctl vm.max_map_count = 655360 sysctl vm.overcommit_memory = 1 sysctl vm.swappiness = 5 /sys/kernel/mm/transparent_hugepage/enabled never
Please note this extremely large number: 5283932191711312175 bytes = 5283932 TB
481473:S 13 Jan 2025 08:26:38.095 * Background append only file **rewriting** started by pid 166120
166120:C 13 Jan 2025 08:27:15.400 # Out Of Memory allocating **5283932191711312175** bytes!
it is similar with : #13649
Comment From: ShooterIT
oh, i missed, your issue is to allocate extremely large member, i think there may be overflow in some areas, like https://github.com/redis/redis/issues/8299. since 5.0.5 is too old to maintain, maybe you can try new version
Comment From: zhuleiandy888
@ShooterIT Thank for your suggestion! i think the problem is like this：13832 i try to reproduce it.