We are running redis cluster, often we see redis server crash with following error.
Crash report
redis-server[2514]: === REDIS BUG REPORT START: Cut & paste starting from here ===
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.150 # Redis 5.0.5 crashed by signal: 11
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.150 # Crashed running the instruction at: 0x4aa80e
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.150 # Accessing address: (nil)
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.150 # Failed assertion: <no assertion failed> (<no file>:0)
redis-server[2514]: ------ STACK TRACE ------
redis-server[2514]: EIP:
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0xe)[0x4aa80e]
redis-server[2514]: Backtrace:
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](logStackTrace+0x45)[0x473ed5]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](sigsegvHandler+0xb9)[0x474699]
redis-server[2514]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2825882390]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0xe)[0x4aa80e]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxRecursiveFree+0x62)[0x4aa862]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](raxFreeWithCallback+0xf)[0x4aa8cf]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](slotToKeyFlush+0x14)[0x4488a4]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](emptyDb+0xf5)[0x4489d5]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](readSyncBulkPayload+0x298)[0x44aa98]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](aeProcessEvents+0x36d)[0x429cdd]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](aeMain+0x2b)[0x429f0b]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](main+0x4ae)[0x426abe]
redis-server[2514]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f28254c7840]
redis-server[2514]: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster](_start+0x29)[0x426d29]
redis-server[2514]: ------ INFO OUTPUT ------
redis-server[2514]: # Server
redis-server[2514]: redis_version:5.0.5
redis-server[2514]: redis_git_sha1:00000000
redis-server[2514]: redis_git_dirty:0
redis-server[2514]: redis_build_id:3412dad92d914679
redis-server[2514]: redis_mode:cluster
redis-server[2514]: os:Linux 4.15.0-1087-gcp x86_64
redis-server[2514]: arch_bits:64
redis-server[2514]: multiplexing_api:epoll
redis-server[2514]: atomicvar_api:atomic-builtin
redis-server[2514]: gcc_version:5.4.0
redis-server[2514]: process_id:2514
redis-server[2514]: run_id:93ed37bd0a754eb9060519cec12d7e94912da993
redis-server[2514]: tcp_port:6379
redis-server[2514]: uptime_in_seconds:564
redis-server[2514]: uptime_in_days:0
redis-server[2514]: hz:10
redis-server[2514]: configured_hz:10
redis-server[2514]: lru_clock:873536
redis-server[2514]: executable:/usr/local/bin/redis-server
redis-server[2514]: config_file:/etc/redis/redis.conf
redis-server[2514]: [1B blob data]
redis-server[2514]: # Clients
redis-server[2514]: connected_clients:1
redis-server[2514]: client_recent_max_input_buffer:2
redis-server[2514]: client_recent_max_output_buffer:0
redis-server[2514]: blocked_clients:0
redis-server[2514]: [1B blob data]
redis-server[2514]: # Memory
redis-server[2514]: used_memory:47935600
redis-server[2514]: used_memory_human:45.71M
redis-server[2514]: used_memory_rss:10544611328
redis-server[2514]: used_memory_rss_human:9.82G
redis-server[2514]: used_memory_peak:10360213920
redis-server[2514]: used_memory_peak_human:9.65G
redis-server[2514]: used_memory_peak_perc:0.46%
redis-server[2514]: used_memory_overhead:1485554
redis-server[2514]: used_memory_startup:1468416
redis-server[2514]: used_memory_dataset:46450046
redis-server[2514]: used_memory_dataset_perc:99.96%
redis-server[2514]: allocator_allocated:10359168424
redis-server[2514]: allocator_active:10360307712
redis-server[2514]: allocator_resident:10545127424
redis-server[2514]: total_system_memory:13649055744
redis-server[2514]: total_system_memory_human:12.71G
redis-server[2514]: used_memory_lua:40960
redis-server[2514]: used_memory_lua_human:40.00K
redis-server[2514]: used_memory_scripts:216
redis-server[2514]: used_memory_scripts_human:216B
redis-server[2514]: number_of_cached_scripts:1
redis-server[2514]: maxmemory:10400396288
redis-server[2514]: maxmemory_human:9.69G
redis-server[2514]: maxmemory_policy:allkeys-lru
redis-server[2514]: allocator_frag_ratio:1.00
redis-server[2514]: allocator_frag_bytes:1139288
redis-server[2514]: allocator_rss_ratio:1.02
redis-server[2514]: allocator_rss_bytes:184819712
redis-server[2514]: rss_overhead_ratio:1.00
redis-server[2514]: rss_overhead_bytes:-516096
redis-server[2514]: mem_fragmentation_ratio:1.02
redis-server[2514]: mem_fragmentation_bytes:185508808
redis-server[2514]: mem_not_counted_for_evict:0
redis-server[2514]: mem_replication_backlog:0
redis-server[2514]: mem_clients_slaves:0
redis-server[2514]: mem_clients_normal:16922
redis-server[2514]: mem_aof_buffer:0
redis-server[2514]: mem_allocator:jemalloc-5.1.0
redis-server[2514]: active_defrag_running:0
redis-server[2514]: lazyfree_pending_objects:0
redis-server[2514]: [1B blob data]
redis-server[2514]: # Persistence
redis-server[2514]: loading:0
redis-server[2514]: rdb_changes_since_last_save:0
redis-server[2514]: rdb_bgsave_in_progress:0
redis-server[2514]: rdb_last_save_time:1611485708
redis-server[2514]: rdb_last_bgsave_status:ok
redis-server[2514]: rdb_last_bgsave_time_sec:-1
redis-server[2514]: rdb_current_bgsave_time_sec:-1
redis-server[2514]: rdb_last_cow_size:0
redis-server[2514]: aof_enabled:0
redis-server[2514]: aof_rewrite_in_progress:0
redis-server[2514]: aof_rewrite_scheduled:0
redis-server[2514]: aof_last_rewrite_time_sec:-1
redis-server[2514]: aof_current_rewrite_time_sec:-1
redis-server[2514]: aof_last_bgrewrite_status:ok
redis-server[2514]: aof_last_write_status:ok
redis-server[2514]: aof_last_cow_size:0
redis-server[2514]: [1B blob data]
redis-server[2514]: # Stats
redis-server[2514]: total_connections_received:7
redis-server[2514]: total_commands_processed:58
redis-server[2514]: instantaneous_ops_per_sec:0
redis-server[2514]: total_net_input_bytes:3642056396
redis-server[2514]: total_net_output_bytes:214894
redis-server[2514]: instantaneous_input_kbps:123085.17
redis-server[2514]: instantaneous_output_kbps:2.31
redis-server[2514]: rejected_connections:0
redis-server[2514]: sync_full:0
redis-server[2514]: sync_partial_ok:0
redis-server[2514]: sync_partial_err:0
redis-server[2514]: expired_keys:0
redis-server[2514]: expired_stale_perc:0.00
redis-server[2514]: expired_time_cap_reached_count:0
redis-server[2514]: evicted_keys:0
redis-server[2514]: keyspace_hits:0
redis-server[2514]: keyspace_misses:0
redis-server[2514]: pubsub_channels:0
redis-server[2514]: pubsub_patterns:0
redis-server[2514]: latest_fork_usec:0
redis-server[2514]: migrate_cached_sockets:0
redis-server[2514]: slave_expires_tracked_keys:0
redis-server[2514]: active_defrag_hits:0
redis-server[2514]: active_defrag_misses:0
redis-server[2514]: active_defrag_key_hits:0
redis-server[2514]: active_defrag_key_misses:0
redis-server[2514]: [1B blob data]
redis-server[2514]: # Replication
redis-server[2514]: role:slave
redis-server[2514]: master_host:xx.xxx..83.166
redis-server[2514]: master_port:6379
redis-server[2514]: master_link_status:down
redis-server[2514]: master_last_io_seconds_ago:-1
redis-server[2514]: master_sync_in_progress:1
redis-server[2514]: slave_repl_offset:1
redis-server[2514]: master_sync_left_bytes:0
redis-server[2514]: master_sync_last_io_seconds_ago:0
redis-server[2514]: master_link_down_since_seconds:150
redis-server[2514]: slave_priority:100
redis-server[2514]: slave_read_only:1
redis-server[2514]: connected_slaves:0
redis-server[2514]: master_replid:a99028ee574759c8d49dabba6d0f851257dcbd84
redis-server[2514]: master_replid2:0000000000000000000000000000000000000000
redis-server[2514]: master_repl_offset:2771796122601
redis-server[2514]: second_repl_offset:-1
redis-server[2514]: repl_backlog_active:0
redis-server[2514]: repl_backlog_size:1048576
redis-server[2514]: repl_backlog_first_byte_offset:2771796122602
redis-server[2514]: repl_backlog_histlen:0
redis-server[2514]: [1B blob data]
redis-server[2514]: # CPU
redis-server[2514]: used_cpu_sys:29.630961
redis-server[2514]: used_cpu_user:208.543581
redis-server[2514]: used_cpu_sys_children:0.000000
redis-server[2514]: used_cpu_user_children:0.000000
redis-server[2514]: [1B blob data]
redis-server[2514]: # Commandstats
redis-server[2514]: cmdstat_readonly:calls=1,usec=0,usec_per_call=0.00
redis-server[2514]: cmdstat_info:calls=57,usec=6127,usec_per_call=107.49
redis-server[2514]: [1B blob data]
redis-server[2514]: # Cluster
redis-server[2514]: cluster_enabled:1
redis-server[2514]: [1B blob data]
redis-server[2514]: # Keyspace
redis-server[2514]: ------ CLIENT LIST OUTPUT ------
redis-server[2514]: id=11 addr=xx.xxx..83.168:39310 fd=19 name= age=280 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
redis-server[2514]: ------ REGISTERS ------
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 #
redis-server[2514]: RAX:00007f2582832258 RBX:00007f2582832250
redis-server[2514]: RCX:0000000000000001 RDX:0000000000000000
redis-server[2514]: RDI:00007f2825028678 RSI:0000000000000000
redis-server[2514]: RBP:00007f2825028678 RSP:00007ffe75074ba0
redis-server[2514]: R8 :00000000000000a1 R9 :0000000000000048
redis-server[2514]: R10:0000000000000009 R11:0000000000000003
redis-server[2514]: R12:0000000000000000 R13:00007f2582832250
redis-server[2514]: R14:0000000000000000 R15:0000000000000400
redis-server[2514]: RIP:00000000004aa80e EFL:0000000000010206
redis-server[2514]: CSGSFS:002b000000000033
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074baf) -> 00007f257112d3a0
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074bae) -> 0000000000000000
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074bad) -> 00007f2825028678
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074bac) -> 00007f257112d3a8
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074bab) -> 00000000004aa862
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074baa) -> 00007f26e4894918
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba9) -> 00007f26e4894918
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba8) -> 0000000000000000
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba7) -> 00007f2825028678
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba6) -> 00007f26e4894918
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba5) -> 00000000004aa862
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba4) -> 00007f25747575a0
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba3) -> 00007f2582832250
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba2) -> 0000000000000000
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba1) -> 00007f2825028678
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # (00007ffe75074ba0) -> 00007f2582832250
redis-server[2514]: ------ FAST MEMORY TEST ------
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # Bio thread for job type #0 terminated
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # Bio thread for job type #1 terminated
redis-server[2514]: 2514:S 24 Jan 2021 18:05:09.194 # Bio thread for job type #2 terminated
redis-server[2514]: *** Preparing to test memory region 7ae000 (2248704 bytes)
redis-server[2514]: *** Preparing to test memory region 1e13000 (135168 bytes)
redis-server[2514]: *** Preparing to test memory region 7f255ee00000 (11553210368 bytes)
redis-server[2514]: *** Preparing to test memory region 7f280f8fd000 (327155712 bytes)
redis-server[2514]: *** Preparing to test memory region 7f28230fe000 (8388608 bytes)
redis-server[2514]: *** Preparing to test memory region 7f28238ff000 (8388608 bytes)
redis-server[2514]: *** Preparing to test memory region 7f2824100000 (8388608 bytes)
redis-server[2514]: *** Preparing to test memory region 7f2824900000 (11534336 bytes)
redis-server[2514]: *** Preparing to test memory region 7f282586d000 (16384 bytes)
redis-server[2514]: *** Preparing to test memory region 7f2825a8a000 (16384 bytes)
redis-server[2514]: *** Preparing to test memory region 7f28263bb000 (24576 bytes)
redis-server[2514]: *** Preparing to test memory region 7f28263ca000 (4096 bytes)
redis-server[2514]: .O.O.O.O.O.O.O.O.O.O.O.O
redis-server[2514]: Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
redis-server[2514]: ------ DUMPING CODE AROUND EIP ------
redis-server[2514]: Symbol: raxRecursiveFree (base: 0x4aa800)
redis-server[2514]: Module: /usr/local/bin/redis-server xx.xxx..83.168:6379 [cluster] (base 0x400000)
redis-server[2514]: $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
redis-server[2514]: $ objdump --adjust-vma=0x4aa800 -D -b binary -m i386:x86-64 /tmp/dump.bin
redis-server[2514]: ------
redis-server[2514]: 2514:S 24 Jan 2021 18:05:53.221 # dump of function (hexdump of 142 bytes):
redis-server[2514]: 415641554989f64154554989d453f606044889fd757a8b0ec1e9038d410489ce8d51ff48f7d883e0074801f048c1e603488d5c06fc4c01f385c9742b48c1e2034889d84829d04c8d68f8660f1f440000488b334c89e24889ef4883eb08e89effffff4939dd75e94d85e4740b410fb60683e0033c0174394c89f7e891ccf8ff48836d10015b5d415c415d415ec30f
redis-server[2514]: Function at 0x4aa800 is raxRecursiveFree
redis-server[2514]: Function at 0x437510 is zfree
redis-server[2514]: === REDIS BUG REPORT END. Make sure to include from START to END. ===
redis-server[2514]: Please report the crash by opening an issue on github:
redis-server[2514]: http://github.com/antirez/redis/issues
redis-server[2514]: Suspect RAM error? Use redis-server --test-memory to verify it.
systemd[1]: redis-server.service: Main process exited, code=dumped, status=11/SEGV
systemd[1]: redis-server.service: Unit entered failed state.
systemd[1]: redis-server.service: Failed with result 'core-dump'.
Aditional information
- OS distribution and version Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-1087-gcp x86_64)
- Steps to reproduce (if any) None
Comment From: itamarhaber
Appears related to #7719
Comment From: codetodya
@itamarhaber any pointers how I can reproduce this issue?
Comment From: itamarhaber
@codetodya regrettably none yet. Would it be possible for you to attach the log lines before the crash from both the replica and its master?