Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
24608:C 18 Oct 2020 12:59:06.038 # Redis 6.0.6 crashed by signal: 11
24608:C 18 Oct 2020 12:59:06.038 # Crashed running the instruction at: 0x561cbd1c46f8
24608:C 18 Oct 2020 12:59:06.038 # Accessing address: 0x1f2cf8
24608:C 18 Oct 2020 12:59:06.039 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-rdb-bgsave 0.0.0.0:6379 [cluster](je_malloc_usable_size+0x78)[0x561cbd1c46f8]
Backtrace:
redis-rdb-bgsave 0.0.0.0:6379 [cluster](logStackTrace+0x5a)[0x561cbd143daa]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](sigsegvHandler+0xb1)[0x561cbd144561]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0)[0x7f63b0dce8a0]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](je_malloc_usable_size+0x78)[0x561cbd1c46f8]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](+0x4e009)[0x561cbd105009]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](rdbSave+0x2a1)[0x561cbd124f41]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](rdbSaveBackground+0xc3)[0x561cbd125013]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](serverCron+0x795)[0x561cbd0fecc5]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](aeProcessEvents+0x33b)[0x561cbd0f64db]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](aeMain+0x1d)[0x561cbd0f67cd]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](main+0x51b)[0x561cbd0f2dcb]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f63b09ecb97]
redis-rdb-bgsave 0.0.0.0:6379 [cluster](_start+0x2a)[0x561cbd0f307a]
------ INFO OUTPUT ------
# Server
redis_version:6.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:d463d609620685c0
redis_mode:cluster
os:Linux 4.15.0-70-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.5.0
process_id:24608
run_id:9a02f0a7f9f78c565a55e0486072cc835547b757
tcp_port:6379
uptime_in_seconds:56892
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:9190426
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
# Clients
connected_clients:104
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:6157456
used_memory_human:5.87M
used_memory_rss:9527296
used_memory_rss_human:9.09M
used_memory_peak:7151256
used_memory_peak_human:6.82M
used_memory_peak_perc:86.10%
used_memory_overhead:4320980
used_memory_startup:1474800
used_memory_dataset:1836476
used_memory_dataset_perc:39.22%
allocator_allocated:6428416
allocator_active:7286784
allocator_resident:10227712
total_system_memory:16819585024
total_system_memory_human:15.66G
used_memory_lua:47104
used_memory_lua_human:46.00K
used_memory_scripts:880
used_memory_scripts_human:880B
number_of_cached_scripts:2
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.13
allocator_frag_bytes:858368
allocator_rss_ratio:1.40
allocator_rss_bytes:2940928
rss_overhead_ratio:0.93
rss_overhead_bytes:-700416
mem_fragmentation_ratio:1.55
mem_fragmentation_bytes:3371376
mem_not_counted_for_evict:3160
mem_replication_backlog:1048576
mem_clients_slaves:16986
mem_clients_normal:1766626
mem_aof_buffer:3160
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:2430
rdb_bgsave_in_progress:0
rdb_last_save_time:1603025645
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:942080
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:0
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:733184
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:9772323
aof_base_size:466048
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:9
# Stats
total_connections_received:11766
total_commands_processed:702730
instantaneous_ops_per_sec:5
total_net_input_bytes:220479722
total_net_output_bytes:1288979139
instantaneous_input_kbps:3.72
instantaneous_output_kbps:3.71
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:314
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:3140
evicted_keys:0
keyspace_hits:86609
keyspace_misses:41
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:956
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.21.2,port=6380,state=online,offset=209398607,lag=1
master_replid:acf424eb1154dc7834078327f258a53d03a71e02
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:209402685
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:208354110
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:0.004260
used_cpu_user:0.000000
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Modules
# Commandstats
cmdstat_hexists:calls=20,usec=95,usec_per_call=4.75
cmdstat_llen:calls=12598,usec=54538,usec_per_call=4.33
cmdstat_replconf:calls=56798,usec=150178,usec_per_call=2.64
cmdstat_hlen:calls=6,usec=19,usec_per_call=3.17
cmdstat_cluster:calls=11644,usec=2559212,usec_per_call=219.79
cmdstat_eval:calls=40,usec=7763,usec_per_call=194.07
cmdstat_rpop:calls=8138,usec=167316,usec_per_call=20.56
cmdstat_hdel:calls=109227,usec=1111495,usec_per_call=10.18
cmdstat_publish:calls=20,usec=1241,usec_per_call=62.05
cmdstat_hset:calls=124109,usec=1256575,usec_per_call=10.12
cmdstat_del:calls=39445,usec=233593,usec_per_call=5.92
cmdstat_hkeys:calls=1,usec=5,usec_per_call=5.00
cmdstat_hget:calls=15718,usec=107552,usec_per_call=6.84
cmdstat_set:calls=193227,usec=1792203,usec_per_call=9.28
cmdstat_lpop:calls=3796,usec=85776,usec_per_call=22.60
cmdstat_hgetall:calls=17066,usec=3494712,usec_per_call=204.78
cmdstat_hincrby:calls=283,usec=3129,usec_per_call=11.06
cmdstat_lpush:calls=7489,usec=137375,usec_per_call=18.34
cmdstat_wait:calls=40,usec=142,usec_per_call=3.55
cmdstat_ping:calls=2,usec=3,usec_per_call=1.50
cmdstat_slaveof:calls=15,usec=49,usec_per_call=3.27
cmdstat_rpush:calls=5113,usec=53219,usec_per_call=10.41
cmdstat_pexpire:calls=546,usec=3797,usec_per_call=6.95
cmdstat_lset:calls=39382,usec=441018,usec_per_call=11.20
cmdstat_client:calls=11470,usec=56260,usec_per_call=4.90
cmdstat_config:calls=2,usec=88,usec_per_call=44.00
cmdstat_psync:calls=1,usec=1108,usec_per_call=1108.00
cmdstat_exists:calls=20,usec=84,usec_per_call=4.20
cmdstat_lindex:calls=41221,usec=465213,usec_per_call=11.29
cmdstat_info:calls=182,usec=39644,usec_per_call=217.82
cmdstat_incrby:calls=327,usec=5675,usec_per_call=17.35
cmdstat_setex:calls=4784,usec=131158,usec_per_call=27.42
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=184,expires=12,avg_ttl=24240086
------ CLIENT LIST OUTPUT ------
id=15 addr=192.168.21.5:44876 fd=24 name= age=56892 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=16 addr=192.168.21.5:44878 fd=25 name= age=56892 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=17 addr=192.168.21.5:44880 fd=26 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=18 addr=192.168.21.5:44882 fd=27 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=11765 addr=192.168.21.7:60562 fd=122 name= age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=cluster user=default
id=19 addr=192.168.21.5:44884 fd=28 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=20 addr=192.168.21.5:44886 fd=29 name= age=56892 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=21 addr=192.168.21.5:44888 fd=30 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=22 addr=192.168.21.5:44890 fd=31 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=23 addr=192.168.21.5:44892 fd=32 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=24 addr=192.168.21.5:44894 fd=33 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=25 addr=192.168.21.5:44896 fd=34 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=26 addr=192.168.21.5:44898 fd=35 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=27 addr=192.168.21.5:44900 fd=36 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=28 addr=192.168.21.5:44902 fd=37 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=29 addr=192.168.21.5:44904 fd=38 name= age=56892 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=30 addr=192.168.21.5:44908 fd=39 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=31 addr=192.168.21.5:44906 fd=40 name= age=56892 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=32 addr=192.168.21.8:50100 fd=41 name= age=56892 idle=714 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=33 addr=192.168.21.8:50102 fd=42 name= age=56892 idle=546 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=34 addr=192.168.21.8:50106 fd=43 name= age=56892 idle=1547 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=106 addr=192.168.21.2:40789 fd=123 name= age=56891 idle=1 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=replconf user=default
id=35 addr=192.168.21.8:50108 fd=44 name= age=56892 idle=1551 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=36 addr=192.168.21.8:50110 fd=45 name= age=56892 idle=1552 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=37 addr=192.168.21.8:50112 fd=46 name= age=56892 idle=2015 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=38 addr=192.168.21.8:50114 fd=47 name= age=56892 idle=723 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=39 addr=192.168.21.8:50116 fd=48 name= age=56892 idle=1552 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=40 addr=192.168.21.8:50118 fd=49 name= age=56892 idle=1545 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=41 addr=192.168.21.8:50120 fd=50 name= age=56892 idle=1545 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=42 addr=192.168.21.8:50122 fd=51 name= age=56892 idle=546 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=43 addr=192.168.21.8:50124 fd=52 name= age=56892 idle=2015 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=44 addr=192.168.21.8:50126 fd=53 name= age=56892 idle=1547 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=45 addr=192.168.21.8:50128 fd=54 name= age=56892 idle=721 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=46 addr=192.168.21.8:50130 fd=55 name= age=56892 idle=714 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=47 addr=192.168.21.8:50132 fd=56 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=48 addr=192.168.21.8:50134 fd=57 name= age=56892 idle=714 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=49 addr=192.168.21.8:50138 fd=58 name= age=56892 idle=1545 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=50 addr=192.168.21.8:50136 fd=59 name= age=56892 idle=1549 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=51 addr=192.168.21.6:59036 fd=60 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=110 addr=192.168.21.7:60702 fd=125 name= age=56885 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall user=default
id=52 addr=192.168.21.8:50142 fd=61 name= age=56892 idle=1545 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=53 addr=192.168.21.8:50144 fd=62 name= age=56892 idle=720 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=54 addr=192.168.21.8:50146 fd=63 name= age=56892 idle=717 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=55 addr=192.168.21.8:50148 fd=64 name= age=56892 idle=720 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=56 addr=192.168.21.8:50150 fd=65 name= age=56892 idle=723 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=57 addr=192.168.21.8:50152 fd=66 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=58 addr=192.168.21.7:60578 fd=67 name= age=56892 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lset user=default
id=109 addr=192.168.21.7:60654 fd=124 name= age=56889 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall user=default
id=59 addr=192.168.21.6:59038 fd=68 name= age=56892 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=60 addr=192.168.21.7:60580 fd=69 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=61 addr=192.168.21.8:50154 fd=70 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=62 addr=192.168.21.8:50156 fd=71 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=63 addr=192.168.21.8:50158 fd=72 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=64 addr=192.168.21.8:50160 fd=73 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=65 addr=192.168.21.8:50162 fd=74 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=66 addr=192.168.21.8:50164 fd=75 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=67 addr=192.168.21.8:50166 fd=76 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=68 addr=192.168.21.8:50168 fd=77 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=69 addr=192.168.21.8:50170 fd=78 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=70 addr=192.168.21.8:50172 fd=79 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=71 addr=192.168.21.8:50174 fd=80 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=72 addr=192.168.21.8:50176 fd=81 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=73 addr=192.168.21.8:50178 fd=82 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=74 addr=192.168.21.8:50180 fd=83 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=75 addr=192.168.21.8:50182 fd=84 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=76 addr=192.168.21.8:50184 fd=85 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=77 addr=192.168.21.8:50186 fd=86 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=78 addr=192.168.21.8:50188 fd=87 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=79 addr=192.168.21.8:50190 fd=88 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=80 addr=192.168.21.8:50192 fd=89 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=81 addr=192.168.21.8:50194 fd=90 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=82 addr=192.168.21.8:50196 fd=91 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=83 addr=192.168.21.8:50198 fd=92 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=84 addr=192.168.21.8:50200 fd=93 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=85 addr=192.168.21.8:50202 fd=94 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=86 addr=192.168.21.8:50204 fd=95 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=87 addr=192.168.21.8:50206 fd=96 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=88 addr=192.168.21.8:50208 fd=97 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=89 addr=192.168.21.8:50210 fd=98 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=90 addr=192.168.21.8:50212 fd=99 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=91 addr=192.168.21.8:50214 fd=100 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=92 addr=192.168.21.8:50216 fd=101 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=93 addr=192.168.21.8:50218 fd=102 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=94 addr=192.168.21.8:50220 fd=103 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=95 addr=192.168.21.8:50222 fd=104 name= age=56892 idle=1547 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=96 addr=192.168.21.8:50224 fd=105 name= age=56892 idle=723 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=97 addr=192.168.21.8:50226 fd=106 name= age=56892 idle=717 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=98 addr=192.168.21.8:50228 fd=107 name= age=56892 idle=1551 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=99 addr=192.168.21.8:50230 fd=108 name= age=56892 idle=714 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=100 addr=192.168.21.8:50232 fd=109 name= age=56892 idle=721 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=101 addr=192.168.21.8:50234 fd=110 name= age=56892 idle=1549 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=incrby user=default
id=102 addr=192.168.21.8:50236 fd=111 name= age=56892 idle=723 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=103 addr=192.168.21.8:50238 fd=112 name= age=56892 idle=1547 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=pexpire user=default
id=104 addr=192.168.21.7:60584 fd=113 name= age=56892 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall user=default
id=4 addr=192.168.21.8:50096 fd=13 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=5 addr=192.168.21.8:50098 fd=14 name= age=56892 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset user=default
id=6 addr=192.168.21.5:44858 fd=15 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=7 addr=192.168.21.5:44860 fd=16 name= age=56892 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=8 addr=192.168.21.5:44862 fd=17 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=9 addr=192.168.21.5:44864 fd=18 name= age=56892 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=10 addr=192.168.21.5:44866 fd=19 name= age=56892 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=11 addr=192.168.21.5:44868 fd=20 name= age=56892 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
id=12 addr=192.168.21.5:44870 fd=21 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=13 addr=192.168.21.5:44872 fd=22 name= age=56892 idle=56892 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL user=default
id=14 addr=192.168.21.5:44874 fd=23 name= age=56892 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set user=default
------ REGISTERS ------
24608:C 18 Oct 2020 12:59:06.045 #
RAX:00000000001f2cf8 RBX:0000561cbe59fa50
RCX:00007f63b20f4778 RDX:0000000000000001
RDI:00007f63b20f4868 RSI:0000000000000000
RBP:00007f63b20f4758 RSP:00007ffeb15fda40
R8 :0000561c80000000 R9 :0000000000000000
R10:0000000000000006 R11:0000000000000206
R12:00007f63b049a4c8 R13:00007f63b049a4c8
R14:00007ffeb15fde10 R15:0000000000000001
RIP:0000561cbd1c46f8 EFL:0000000000010202
CSGSFS:002b000000000033
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4f) -> 0000000000000000
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4e) -> 00007ffeb15fdae8
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4d) -> 0000000000000000
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4c) -> 0000000000000000
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4b) -> 0000001eb049a4c8
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda4a) -> 00007f63b0b80bce
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda49) -> 0000000000000004
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda48) -> 0000000000000000
24608:C 18 Oct 2020 12:59:06.045 # (00007ffeb15fda47) -> 70509aa598dc5200
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda46) -> 00007ffeb15fdccf
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda45) -> 00007f63b09fd58e
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda44) -> 00007ffeb15fdb70
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda43) -> 000000000000001e
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda42) -> 00007ffe006f6d2e
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda41) -> 6362696c2f534547
24608:C 18 Oct 2020 12:59:06.046 # (00007ffeb15fda40) -> 415353454d5f434c
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
*** Preparing to test memory region 561cbd490000 (2277376 bytes)
*** Preparing to test memory region 561cbe56b000 (270336 bytes)
*** Preparing to test memory region 7f63a7c00000 (4329472 bytes)
*** Preparing to test memory region 7f63ac200000 (8388608 bytes)
*** Preparing to test memory region 7f63aca00000 (2097152 bytes)
*** Preparing to test memory region 7f63ace00000 (8388608 bytes)
*** Preparing to test memory region 7f63ad600000 (2097152 bytes)
*** Preparing to test memory region 7f63ad97c000 (3670016 bytes)
*** Preparing to test memory region 7f63adcfd000 (8388608 bytes)
*** Preparing to test memory region 7f63ae4fe000 (8388608 bytes)
*** Preparing to test memory region 7f63aecff000 (8388608 bytes)
*** Preparing to test memory region 7f63af500000 (8388608 bytes)
*** Preparing to test memory region 7f63afd00000 (3145728 bytes)
*** Preparing to test memory region 7f63b0000000 (8388608 bytes)
*** Preparing to test memory region 7f63b0db8000 (16384 bytes)
*** Preparing to test memory region 7f63b0fd7000 (16384 bytes)
*** Preparing to test memory region 7f63b14a3000 (12288 bytes)
*** Preparing to test memory region 7f63b20f4000 (32768 bytes)
*** Preparing to test memory region 7f63b2106000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: je_malloc_usable_size (base: 0x561cbd1c4680)
Module: redis-rdb-bgsave 0.0.0.0:6379 [cluster] (base 0x561cbd0b7000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x561cbd1c4680 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
24608:C 18 Oct 2020 12:59:06.729 # dump of function (hexdump of 248 bytes):
55534889fb4881ec9801000064488b042528000000488984248801000031c0803db2132d00000f85840000004885ff0f84bb0000004889e54889efe840d5030031c04889de4889da48c1ee1a4881e2000000c081e6f00000004801ee488b0e4839ca0f85900000004889d848c1e80925f8ff1f0048034608488b18488d153e72080048c1eb30488b04da488bbc24880100006448333c25280000000f85920100004881c4980100005b5dc30f1f440000488b05e1172c0064803800751b4885ff742e644803042500000000488d6828e96effffff0f1f400064480304250000000031f64889c7e855b805004885db754831c0eb960f1f4000
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Aditional information
- OS distribution and version Linux
6.0.6
- Steps to reproduce (if any)
Comment From: oranagra
@guotie can you please upload the executable you used (hopefully it can provide more details on the stack trace of the crash)
Comment From: guotie
Comment From: oranagra
@guotie any chance you have the same build with the debug symbols (in case they were stripped after compilation)?
some thing don't make sense. i don't see how rdbSave can lead to je_malloc_usable_size
Comment From: guotie
I use apt install it
Comment From: oranagra
@guotie can you please check what you have in the log file just above the crash log? i'm suspecting it's "Failed opening the RDB file"... maybe your disk became non-writable? still trying to figure out why it cased a crash.
Comment From: oranagra
@guotie please specify your apt source (from the disassembly it doesn't like like clean redis 6.0.6)
for the record: it looks like rdbSave is calling zfree after logging "Failed opening the RDB file"
Comment From: guotie
sudo apt-get install software-properties-common -y
sudo add-apt-repository ppa:chris-lea/redis-server -y
sudo apt-get update && sudo apt-get install redis-server -y
sudo systemctl enable redis-server
sudo systemctl start redis-server
Comment From: oranagra
it turns out there's a bug in the PPA code. but it's only triggered in this error condition. for some reason your disk wasn't writable and that crashed the child child process, if you'll solve that you'll be ok until the bug is fixed.
Comment From: yossigo
@guotie This has been a bug in the debian-specific patches, and it has been resolved in the new official PPA we maintain. If you want to use it, you'll have to use sudo add-apt-repository ppa:redislabs/redis -y.
Comment From: yossigo
Pinging @chrislea and @lamby, as the source of this crash is the Debian 0010-Use-get_current_dir_name-over-PATHMAX-etc.patch which seems to exhibit two issues:
1. It uses zfree() to free an allocation done by get_current_dir_name().
2. In another occasion it uses zfree() on a stack pointer.
Both cases only involve error handling/reporting code flows when failing to write local files so they are not that common, but if reached crashes Redis.
Comment From: lamby
Thanks @yossigo. Can you share your patch that does not have those two issues so we can include them in the Debian package?
Comment From: yossigo
@lamby Sure I'll do that.
Comment From: lamby
@yossigo Can you share your patch...? Looks like you closed this without sending it on.
Comment From: yossigo
@lamby Looks like I did, and it's merged: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=972683
Comment From: lamby
Ah! Thanks! :