Redis Redis 5.0.4 crashed by signal: 11

=== REDIS BUG REPORT START: Cut & paste starting from here ===
24734:S 03 Jun 2020 03:32:05.013 # Redis 5.0.4 crashed by signal: 11
24734:S 03 Jun 2020 03:32:05.013 # Crashed running the instruction at: 0x4a4bb5
24734:S 03 Jun 2020 03:32:05.013 # Accessing address: (nil)
24734:S 03 Jun 2020 03:32:05.013 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x15)[0x4a4bb5]

Backtrace:
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](logStackTrace+0x29)[0x470b89]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](sigsegvHandler+0xac)[0x47122c]
/lib64/libpthread.so.0(+0xf5d0)[0x7fce7e5135d0]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x15)[0x4a4bb5]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxRecursiveFree+0x92)[0x4a4c32]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](raxFreeWithCallback+0xf)[0x4a4caf]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](slotToKeyFlush+0x14)[0x447a84]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](emptyDb+0xfd)[0x447bbd]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](readSyncBulkPayload+0x26f)[0x4493af]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x2a0)[0x42abf0]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](aeMain+0x2b)[0x42aebb]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster](main+0x4af)[0x427d6f]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fce7e1593d5]
/icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster][0x427fba]

------ INFO OUTPUT ------
# Server
redis_version:5.0.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:b1ab97271a1ec5c3
redis_mode:cluster
os:Linux 3.10.0-957.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:24734
run_id:b5abac4e5b19eec64647dda35f49af60fb0074b3
tcp_port:6379
uptime_in_seconds:3825
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:14064282
executable:/icp/redis-5.0.4/src/redis-server
config_file:/icp/redis-5.0.4/redis.conf

# Clients
connected_clients:65
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

# Memory
used_memory:172478016
used_memory_human:164.49M
used_memory_rss:110706331648
used_memory_rss_human:103.10G
used_memory_peak:120550160800
used_memory_peak_human:112.27G
used_memory_peak_perc:0.14%
used_memory_overhead:2571010
used_memory_startup:1471080
used_memory_dataset:169907006
used_memory_dataset_perc:99.36%
allocator_allocated:109810260312
allocator_active:110502969344
allocator_resident:111414452224
total_system_memory:540708544512
total_system_memory_human:503.57G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:472446402560
maxmemory_human:440.00G
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:692709032
allocator_rss_ratio:1.01
allocator_rss_bytes:911482880
rss_overhead_ratio:0.99
rss_overhead_bytes:-708120576
mem_fragmentation_ratio:1.01
mem_fragmentation_bytes:896204280
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1099930
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:6412386
rdb_bgsave_in_progress:0
rdb_last_save_time:1591118761
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:754
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:36528128

# Stats
total_connections_received:9802
total_commands_processed:22496
instantaneous_ops_per_sec:0
total_net_input_bytes:52027392194
total_net_output_bytes:2796585
instantaneous_input_kbps:92027.19
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1646385
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:slave
master_host:172.16.237.172
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
slave_repl_offset:1
master_sync_left_bytes:0
master_sync_last_io_seconds_ago:0
master_link_down_since_seconds:1094
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:6976896842716a59e37be453be617aa1735203ab
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:14755739674948
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10737418240
repl_backlog_first_byte_offset:14755739674949
repl_backlog_histlen:0

# CPU
used_cpu_sys:485.246212
used_cpu_user:995.350596
used_cpu_sys_children:198.382939
used_cpu_user_children:552.006059

# Commandstats
cmdstat_ping:calls=8608,usec=5179,usec_per_call=0.60
cmdstat_cluster:calls=1955,usec=882351,usec_per_call=451.33
cmdstat_auth:calls=9802,usec=17043,usec_per_call=1.74
cmdstat_config:calls=176,usec=4168,usec_per_call=23.68
cmdstat_dbsize:calls=1955,usec=2966,usec_per_call=1.52

# Cluster
cluster_enabled:1

# Keyspace

------ CLIENT LIST OUTPUT ------
id=9754 addr=?:0 fd=50 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9755 addr=?:0 fd=52 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9756 addr=?:0 fd=56 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9757 addr=?:0 fd=58 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9758 addr=?:0 fd=63 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9742 addr=172.16.237.101:45144 fd=79 name= age=51 idle=51 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=9783 addr=?:0 fd=11 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9784 addr=?:0 fd=30 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9785 addr=?:0 fd=43 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9786 addr=?:0 fd=44 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9787 addr=?:0 fd=47 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9788 addr=?:0 fd=48 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9789 addr=?:0 fd=49 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9799 addr=?:0 fd=31 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9800 addr=?:0 fd=33 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9801 addr=?:0 fd=35 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9802 addr=?:0 fd=36 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9803 addr=?:0 fd=37 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9804 addr=?:0 fd=38 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9805 addr=?:0 fd=40 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9806 addr=?:0 fd=41 name= age=16 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9790 addr=?:0 fd=51 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9791 addr=?:0 fd=53 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9792 addr=?:0 fd=54 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9793 addr=?:0 fd=55 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9794 addr=?:0 fd=57 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9795 addr=?:0 fd=59 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9796 addr=?:0 fd=60 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9797 addr=?:0 fd=61 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9798 addr=?:0 fd=62 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9743 addr=?:0 fd=12 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9744 addr=?:0 fd=13 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9745 addr=?:0 fd=14 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9746 addr=?:0 fd=15 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9747 addr=?:0 fd=17 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9748 addr=?:0 fd=25 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9749 addr=?:0 fd=28 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9750 addr=?:0 fd=32 name= age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9759 addr=?:0 fd=16 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9760 addr=?:0 fd=18 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9761 addr=?:0 fd=19 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9762 addr=?:0 fd=20 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9763 addr=?:0 fd=21 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9764 addr=?:0 fd=22 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9765 addr=?:0 fd=23 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9766 addr=?:0 fd=24 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9767 addr=?:0 fd=26 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9775 addr=?:0 fd=39 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9776 addr=?:0 fd=42 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9777 addr=?:0 fd=65 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9778 addr=?:0 fd=66 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9779 addr=?:0 fd=67 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9780 addr=?:0 fd=68 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9781 addr=?:0 fd=69 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9782 addr=?:0 fd=74 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9768 addr=?:0 fd=27 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9769 addr=?:0 fd=29 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9770 addr=?:0 fd=64 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9771 addr=?:0 fd=70 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9772 addr=?:0 fd=71 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9773 addr=?:0 fd=72 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9774 addr=?:0 fd=73 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9751 addr=?:0 fd=34 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9752 addr=?:0 fd=45 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9753 addr=?:0 fd=46 name= age=22 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping

------ REGISTERS ------
24734:S 03 Jun 2020 03:32:06.854 # 
RAX:0000000000000000 RBX:00007faf8d06c090
RCX:0000000000000008 RDX:0000000000000000
RDI:00007fce7dc28588 RSI:0000000000000000
RBP:00007fce7dc28588 RSP:00007fff361f5e70
R8 :0000000000000000 R9 :00007fce7de00900
R10:00007fc55671e940 R11:00007fca40000000
R12:0000000000000000 R13:00007faf8d06c090
R14:0000000000000000 R15:0000000000000010
RIP:00000000004a4bb5 EFL:0000000000010202
CSGSFS:0000000000000033
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7f) -> 00000000004a4c32
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7e) -> 00007fb93b99fd00
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7d) -> 00007fb93b99fd08
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7c) -> 0000000000000000
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7b) -> 00007fce7dc28588
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e7a) -> 00007fb93b99fd28
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e79) -> 0000000000000000
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e78) -> 00007fce7f03d930
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e77) -> 00000000004a4c32
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e76) -> 00007faf7ef913e0
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e75) -> 00007faf8d06c090
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e74) -> 0000000000000000
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e73) -> 00007fce7dc28588
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e72) -> 00007faf8d06c090
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e71) -> 00007fc42978c830
24734:S 03 Jun 2020 03:32:06.854 # (00007fff361f5e70) -> 00007fcd7c212e40

------ FAST MEMORY TEST ------
24734:S 03 Jun 2020 03:32:06.855 # Bio thread for job type #0 terminated
24734:S 03 Jun 2020 03:32:06.855 # Bio thread for job type #1 terminated
24734:S 03 Jun 2020 03:32:06.855 # Bio thread for job type #2 terminated
*** Preparing to test memory region 78e000 (2248704 bytes)
*** Preparing to test memory region a19000 (135168 bytes)
*** Preparing to test memory region 7faf6d800000 (132068147200 bytes)
*** Preparing to test memory region 7fce2d7d3000 (1207959552 bytes)
*** Preparing to test memory region 7fce757d4000 (8388608 bytes)
*** Preparing to test memory region 7fce75fd5000 (8388608 bytes)
*** Preparing to test memory region 7fce767d6000 (8388608 bytes)
*** Preparing to test memory region 7fce76fd6000 (3145728 bytes)
*** Preparing to test memory region 7fce7d800000 (8388608 bytes)
*** Preparing to test memory region 7fce7e4ff000 (20480 bytes)
*** Preparing to test memory region 7fce7e71c000 (16384 bytes)
*** Preparing to test memory region 7fce7f03d000 (20480 bytes)
*** Preparing to test memory region 7fce7f04d000 (4096 bytes)
*** Preparing to test memory region 7fce7f04e000 (4096 bytes)
*** Preparing to test memory region 7fce7f051000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: raxRecursiveFree (base: 0x4a4ba0)
Module: /icp/redis-5.0.4/src/redis-server 0.0.0.0:6379 [cluster] (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x4a4ba0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
24734:S 03 Jun 2020 03:41:26.653 # dump of function (hexdump of 149 bytes):
41564989f6415541544989d4554889fd534883ec100fb6068b16a8040f85a6000000c1ea038d720489d1448d42ff48f7de83e6074801ce48c1e10383e0033c010f94c00fb6f848c1e03f488d5cf9fc48c1f83f83e0084801f34829c34c01f385d2743449c1e0034889d84c29c04c8d68f80f1f8000000000488b034c89e24889ef4883eb084889c64889442408e86effffff4c39eb

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Comment From: hyunjong-yoo

above situation occurred after redis shutdown and start. it occurred abount 5 min after completion (rdb, aof data loading) after restarting, currently service is normail.

check issue. thanks.

Comment From: oranagra

@hyunjong-yoo can you please upload the redis-server binary you were using?

Comment From: hyunjong-yoo

Attached. This binary run on CENTOS 7.4. redis-server.tar.gz

Comment From: oranagra

translated stack trace:

EIP:
0x4a4bb5 is in raxRecursiveFree (rax.c:1223).

Backtrace:
0x470b89 is in logStackTrace (debug.c:1098).
0x47122c is in sigsegvHandler (debug.c:1273).
/lib64/libpthread.so.0(+0xf5d0)[0x7fce7e5135d0]
0x4a4bb5 is in raxRecursiveFree (rax.c:1223).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4c32 is in raxRecursiveFree (rax.c:1225).
0x4a4caf is in raxFreeWithCallback (rax.c:1242).
0x447a84 is in slotToKeyFlush (db.c:1499).
0x447bbd is in emptyDb (db.c:375).
0x4493af is in readSyncBulkPayload (replication.c:1278).
0x42abf0 is in aeProcessEvents (ae.c:448).
0x42aebb is in aeMain (ae.c:498).
0x427d6f is in main (server.c:4198).
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fce7e1593d5]

rax.c:1223

    int numchildren = n->iscompr ? 1 : n->size;

for the record, no differences in rax.c between 5.0.4 and 5.0.9, and the ones in 6.0 do not explain it.

@antirez are you aware of any recent change that might have already fixed this?

Comment From: antirez

@oranagra nope unfortunately I'm not aware of any Rax issue that could explain this crash. There is always the problem of any other random corruption showing this result. Rax is very well tested, however we lack a test with such big amounts of memory, the user is using 100GB... Maybe there are certain things happening only with very large trees? I'm not sure however where to find an instance right now with this amount of memory.

Comment From: antirez

@oranagra this box has 500G of RAM...

Comment From: oranagra

i'm not impressed by numbers (they all look alike to me). maybe NUMA makes an actual difference, or other boolean shifts, like crossing 32bit boundary (more than 4GB), but i don't see why 500GB is different than 32GB.

i'm assuming this is a bug. but without being able to reproduce it, i don't know what to do.

Comment From: hyunjong-yoo

I checked the status. I will talk about the issue again if the upgrade or occurs again later.

Thanks for the answer.