Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 22 Jan 2022 21:37:12.022 # Redis 6.2.5 crashed by signal: 11, si_code: 1
1:M 22 Jan 2022 21:37:12.022 # Accessing address: 0x4000166a
1:M 22 Jan 2022 21:37:12.022 # Crashed running the instruction at: 0x400004b644
------ STACK TRACE ------
EIP:
/usr/local/bin/redis-server *:6379(dictSdsKeyCompare+0x34)[0x400004b644]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x4002158140]
/usr/local/bin/redis-server *:6379(dictSdsKeyCompare+0x34)[0x400004b644]
/usr/local/bin/redis-server *:6379(dictAddRaw+0x155)[0x4000049af5]
/usr/local/bin/redis-server *:6379(dictAdd+0x11)[0x4000049c81]
/usr/local/bin/redis-server *:6379(dbAdd+0x25)[0x400006fbe5]
/usr/local/bin/redis-server *:6379(hashTypeLookupWriteOrCreate+0x67)[0x4000092637]
/usr/local/bin/redis-server *:6379(hsetCommand+0x30)[0x4000092bf0]
/usr/local/bin/redis-server *:6379(call+0xa1)[0x400004f161]
/usr/local/bin/redis-server *:6379(luaRedisGenericCommand+0x389)[0x40000b5ef9]
/usr/local/bin/redis-server *:6379(+0x10d378)[0x400010d378]
/usr/local/bin/redis-server *:6379(+0x1175b0)[0x40001175b0]
/usr/local/bin/redis-server *:6379(+0x10da15)[0x400010da15]
/usr/local/bin/redis-server *:6379(+0x10cd48)[0x400010cd48]
/usr/local/bin/redis-server *:6379(+0x10dbc0)[0x400010dbc0]
/usr/local/bin/redis-server *:6379(lua_pcall+0x48)[0x400010b0a8]
/usr/local/bin/redis-server *:6379(evalGenericCommand+0x1f0)[0x40000b3870]
/usr/local/bin/redis-server *:6379(call+0xa1)[0x400004f161]
/usr/local/bin/redis-server *:6379(processCommand+0x5b3)[0x4000050d03]
/usr/local/bin/redis-server *:6379(processInputBuffer+0xf8)[0x4000063c28]
/usr/local/bin/redis-server *:6379(+0xfac98)[0x40000fac98]
/usr/local/bin/redis-server *:6379(aeProcessEvents+0x292)[0x4000047d82]
/usr/local/bin/redis-server *:6379(aeMain+0x1d)[0x4000047fed]
/usr/local/bin/redis-server *:6379(main+0x316)[0x4000044286]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x400218cd0a]
/usr/local/bin/redis-server *:6379(_start+0x2a)[0x400004475a]
------ REGISTERS ------
1:M 22 Jan 2022 21:37:12.030 #
RAX:000000400004b6e8 RBX:000000400293c430
RCX:000000400019b220 RDX:0000000000000044
RDI:000000400629c283 RSI:400000004000166b
RBP:0000000000000000 RSP:0000004001c3f4c8
R8 :0000000000000000 R9 :00000040026000c0
R10:0000000000000005 R11:000000400293a000
R12:0000004001c3e399 R13:000000400293c450
R14:000000400293c420 R15:000000400629c283
RIP:000000400004b644 EFL:0000000000000207
CSGSFS:002b000000000033
1:M 22 Jan 2022 21:37:12.030 # (0000004001c3f4d7) -> 0000004002a35000
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d6) -> 00000040061f7cf0
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d5) -> 00000040061f7d50
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d4) -> 0000004000049c81
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d3) -> 0000004000209350
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d2) -> 00000012f1780fe9
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d1) -> 0000004002a42540
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4d0) -> 00000040061f7d50
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cf) -> 00000040061f7cf0
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4ce) -> 000000400293c420
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cd) -> 0000000000000001
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cc) -> 000000000000017b
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4cb) -> 000000400293c470
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4ca) -> 0000000000000bd8
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4c9) -> 5f6518bc5329297b
1:M 22 Jan 2022 21:37:12.031 # (0000004001c3f4c8) -> 0000004000049af5
------ INFO OUTPUT ------
# Server
redis_version:6.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3afff201ac09d18c
redis_mode:standalone
os:Linux 5.10.76-linuxkit x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:9db7c01289f160a850ea0596de1a8c78dc7ea7ef
tcp_port:6379
server_time_usec:1642887432021178
uptime_in_seconds:104
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:15497479
executable:/usr/local/bin/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:22
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:731
client_recent_max_output_buffer:0
blocked_clients:4
tracking_clients:0
clients_in_timeout_table:4
# Memory
used_memory:3790216
used_memory_human:3.61M
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:4012296
used_memory_peak_human:3.83M
used_memory_peak_perc:94.47%
used_memory_overhead:1443336
used_memory_startup:904800
used_memory_dataset:2346880
used_memory_dataset_perc:81.34%
allocator_allocated:3801080
allocator_active:4284416
allocator_resident:7626752
total_system_memory:2085158912
total_system_memory_human:1.94G
used_memory_lua:65536
used_memory_lua_human:64.00K
used_memory_scripts:19728
used_memory_scripts_human:19.27K
number_of_cached_scripts:6
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.13
allocator_frag_bytes:483336
allocator_rss_ratio:1.78
allocator_rss_bytes:3342336
rss_overhead_ratio:0.00
rss_overhead_bytes:-7626752
mem_fragmentation_ratio:0.00
mem_fragmentation_bytes:-3729032
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:452512
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:502
rdb_bgsave_in_progress:0
rdb_last_save_time:1642887328
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:22
total_commands_processed:1403
instantaneous_ops_per_sec:29
total_net_input_bytes:315081
total_net_output_bytes:563246
instantaneous_input_kbps:2.13
instantaneous_output_kbps:5.87
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:24
evicted_keys:0
keyspace_hits:1271
keyspace_misses:78
pubsub_channels:8
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:41
dump_payload_sanitizations:0
total_reads_processed:387
total_writes_processed:764
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:a740067518cdb3377a576b9b36354c391a907c9a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.842743
used_cpu_user:1.708462
used_cpu_sys_children:0.011343
used_cpu_user_children:0.049196
used_cpu_sys_main_thread:0.832938
used_cpu_user_main_thread:1.685633
# Modules
module:name=ReJSON,ver=20004,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20205,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
# Commandstats
cmdstat_get:calls=22,usec=234,usec_per_call=10.64,rejected_calls=0,failed_calls=0
cmdstat_rpoplpush:calls=43,usec=207,usec_per_call=4.81,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=1,usec=249,usec_per_call=249.00,rejected_calls=0,failed_calls=0
cmdstat_monitor:calls=1,usec=126,usec_per_call=126.00,rejected_calls=0,failed_calls=0
cmdstat_zrangebyscore:calls=57,usec=5103,usec_per_call=89.53,rejected_calls=0,failed_calls=0
cmdstat_evalsha:calls=157,usec=244241,usec_per_call=1555.68,rejected_calls=0,failed_calls=20
cmdstat_zscore:calls=22,usec=1428,usec_per_call=64.91,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=22,usec=86,usec_per_call=3.91,rejected_calls=0,failed_calls=0
cmdstat_zrem:calls=44,usec=1538,usec_per_call=34.95,rejected_calls=0,failed_calls=0
cmdstat_del:calls=22,usec=484,usec_per_call=22.00,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=80,usec=555,usec_per_call=6.94,rejected_calls=0,failed_calls=0
cmdstat_lrem:calls=44,usec=1272,usec_per_call=28.91,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=21,usec=334,usec_per_call=15.90,rejected_calls=0,failed_calls=21
cmdstat_hmset:calls=44,usec=1688,usec_per_call=38.36,rejected_calls=0,failed_calls=0
cmdstat_command:calls=1,usec=10173,usec_per_call=10173.00,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=45,usec=1409,usec_per_call=31.31,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=1,usec=366,usec_per_call=366.00,rejected_calls=0,failed_calls=0
cmdstat_smembers:calls=4,usec=63,usec_per_call=15.75,rejected_calls=0,failed_calls=0
cmdstat_info:calls=14,usec=7414,usec_per_call=529.57,rejected_calls=0,failed_calls=0
cmdstat_json.set:calls=68,usec=21506,usec_per_call=316.26,rejected_calls=0,failed_calls=0
cmdstat_zrange:calls=78,usec=3810,usec_per_call=48.85,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=65,usec=4267,usec_per_call=65.65,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=23,usec=1594,usec_per_call=69.30,rejected_calls=0,failed_calls=0
cmdstat_publish:calls=224,usec=2361,usec_per_call=10.54,rejected_calls=0,failed_calls=0
cmdstat_lrange:calls=4,usec=40,usec_per_call=10.00,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=23,usec=538,usec_per_call=23.39,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=22,usec=4246,usec_per_call=193.00,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=65,usec=1886,usec_per_call=29.02,rejected_calls=0,failed_calls=0
cmdstat_srem:calls=44,usec=319,usec_per_call=7.25,rejected_calls=0,failed_calls=0
cmdstat_brpoplpush:calls=35,usec=6332,usec_per_call=180.91,rejected_calls=0,failed_calls=0
cmdstat_client:calls=26,usec=228,usec_per_call=8.77,rejected_calls=0,failed_calls=0
cmdstat_set:calls=30,usec=3504,usec_per_call=116.80,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=8,usec=378,usec_per_call=47.25,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=20,usec=86805,usec_per_call=4340.25,rejected_calls=0,failed_calls=0
cmdstat_incr:calls=23,usec=787,usec_per_call=34.22,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_ERR:count=21
errorstat_NOSCRIPT:count=20
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=1024,expires=21,avg_ttl=24087363
------ CLIENT LIST OUTPUT ------
id=9 addr=172.17.0.1:59028 laddr=172.17.0.3:6379 fd=10 name=local:bull:Y2Fk age=20 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=729 qbuf-free=40225 argv-mem=589 obl=0 oll=0 omem=0 tot-mem=62189 events=r cmd=evalsha user=default redir=-1
id=10 addr=172.17.0.1:59030 laddr=172.17.0.3:6379 fd=11 name=local:bull:Y2FkLWNhY2hl age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1
id=11 addr=172.17.0.1:59032 laddr=172.17.0.3:6379 fd=12 name=local:bull:ZGlzY29yZA== age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1
id=12 addr=172.17.0.1:59034 laddr=172.17.0.3:6379 fd=13 name= age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
id=13 addr=172.17.0.1:59026 laddr=172.17.0.3:6379 fd=14 name=local:bull:Y2Fk age=20 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=14 addr=172.17.0.1:59038 laddr=172.17.0.3:6379 fd=15 name= age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1
id=15 addr=172.17.0.1:59036 laddr=172.17.0.3:6379 fd=16 name=local:bull:cGF0cm9scw== age=20 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20584 events=r cmd=evalsha user=default redir=-1
id=7 addr=172.17.0.1:59022 laddr=172.17.0.3:6379 fd=8 name= age=30 idle=0 flags=O db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=879 oll=0 omem=0 tot-mem=20496 events=r cmd=monitor user=default redir=-1
id=8 addr=172.17.0.1:59024 laddr=172.17.0.3:6379 fd=9 name=local:bull:ZGlzY29yZC1ub3RpZg== age=20 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1
id=16 addr=172.17.0.1:59044 laddr=172.17.0.3:6379 fd=17 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1
id=17 addr=172.17.0.1:59046 laddr=172.17.0.3:6379 fd=18 name= age=19 idle=15 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=18 addr=172.17.0.1:59050 laddr=172.17.0.3:6379 fd=19 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default redir=-1
id=19 addr=172.17.0.1:59052 laddr=172.17.0.3:6379 fd=20 name= age=19 idle=19 flags=P db=0 sub=4 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=25 addr=172.17.0.1:59064 laddr=172.17.0.3:6379 fd=26 name=local:bull:cGF0cm9scw== age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=59 obl=0 oll=0 omem=0 tot-mem=20579 events=r cmd=brpoplpush user=default redir=-1
id=26 addr=172.17.0.1:59066 laddr=172.17.0.3:6379 fd=27 name=local:bull:ZGlzY29yZA== age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=540 qbuf-free=0 argv-mem=59 obl=0 oll=0 omem=0 tot-mem=21211 events=r cmd=brpoplpush user=default redir=-1
id=27 addr=172.17.0.1:59068 laddr=172.17.0.3:6379 fd=28 name=local:bull:Y2FkLWNhY2hl age=19 idle=4 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=94 qbuf-free=0 argv-mem=63 obl=0 oll=0 omem=0 tot-mem=20687 events=r cmd=brpoplpush user=default redir=-1
id=28 addr=172.17.0.1:59070 laddr=172.17.0.3:6379 fd=29 name=local:bull:Y2Fk age=19 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=82 argv-mem=51 obl=0 oll=0 omem=0 tot-mem=20659 events=r cmd=brpoplpush user=default redir=-1
id=20 addr=172.17.0.1:59048 laddr=172.17.0.3:6379 fd=21 name= age=19 idle=19 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=21 addr=172.17.0.1:59056 laddr=172.17.0.3:6379 fd=22 name=local:bull:Y2FkLWNhY2hl age=19 idle=19 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=22 addr=172.17.0.1:59058 laddr=172.17.0.3:6379 fd=23 name=local:bull:Y2Fk age=19 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=23 addr=172.17.0.1:59062 laddr=172.17.0.3:6379 fd=24 name=local:bull:ZGlzY29yZA== age=19 idle=19 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=24 addr=172.17.0.1:59060 laddr=172.17.0.3:6379 fd=25 name=local:bull:cGF0cm9scw== age=19 idle=4 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
------ CURRENT CLIENT INFO ------
id=9 addr=172.17.0.1:59028 laddr=172.17.0.3:6379 fd=10 name=local:bull:Y2Fk age=20 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=729 qbuf-free=40225 argv-mem=589 obl=0 oll=0 omem=0 tot-mem=62189 events=r cmd=evalsha user=default redir=-1
argv[0]: 'evalsha'
argv[1]: '8f55ae4a3be429c6d38c5d5db3e80edf89197b64'
argv[2]: '6'
argv[3]: 'local:bull:cad:wait'
argv[4]: 'local:bull:cad:paused'
argv[5]: 'local:bull:cad:meta-paused'
argv[6]: 'local:bull:cad:id'
argv[7]: 'local:bull:cad:delayed'
argv[8]: 'local:bull:cad:priority'
argv[9]: 'local:bull:cad:'
argv[10]: 'repeat:757805924c3d6f0c532f0e36651e4769:1642887433000'
argv[11]: 'warn inactivity'
argv[12]: '{}'
argv[13]: '{"repeat":{"count":970,"key":"warn inactivity:inactivity-warning::1000","every":1000,"jobId":"inactivity-warning"},"jobId":"repeat:757805924c3d6f0c532f0e36651e4769:1642887433000","delay":980,"timestamp":1642887432020,"prevMillis":1642887433000,"attempts":1}'
argv[14]: '1642887432020'
argv[15]: '980'
argv[16]: '1642887433000'
argv[17]: '0'
argv[18]: 'LPUSH'
argv[19]: 'e9288e3f-961b-4a08-a6aa-19fb873b5ab2'
------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
1: modulesCollectInfo
at /usr/src/redis/src/module.c:7076:9
2: logModulesInfo
at /usr/src/redis/src/debug.c:1598:22
3: printCrashReport
at /usr/src/redis/src/debug.c:1849:5
sigsegvHandler
at /usr/src/redis/src/debug.c:1831:5
4: <unknown>
5: dictSdsKeyCompare
at /usr/src/redis/src/server.c:1259:8
6: _dictKeyIndex
at /usr/src/redis/src/dict.c:1038:33
dictAddRaw
at /usr/src/redis/src/dict.c:325:18
7: dictAdd
at /usr/src/redis/src/dict.c:290:24
8: dbAdd
at /usr/src/redis/src/db.c:194:18
9: hashTypeLookupWriteOrCreate
at /usr/src/redis/src/t_hash.c:454:9
hashTypeLookupWriteOrCreate
at /usr/src/redis/src/t_hash.c:448:7
10: hsetCommand
at /usr/src/redis/src/t_hash.c:663:14
11: call
at /usr/src/redis/src/server.c:3717:5
12: luaRedisGenericCommand
at /usr/src/redis/src/scripting.c:736:5
13: luaD_precall
14: luaV_execute
15: luaD_call
16: luaD_rawrunprotected
17: luaD_pcall
18: lua_pcall
19: evalGenericCommand
at /usr/src/redis/src/scripting.c:1598:11
20: call
at /usr/src/redis/src/server.c:3717:5
21: processCommand
at /usr/src/redis/src/server.c:4242:9
22: processCommandAndResetClient
at /usr/src/redis/src/networking.c:2018:9
processInputBuffer
at /usr/src/redis/src/networking.c:2119:17
23: callHandler
at /usr/src/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /usr/src/redis/src/connection.c:295:14
24: aeProcessEvents
at /usr/src/redis/src/ae.c:427:17
25: aeMain
at /usr/src/redis/src/ae.c:487:9
26: main
at /usr/src/redis/src/server.c:6396:5
27: __libc_start_main
28: _start
------ FAST MEMORY TEST ------
1:M 22 Jan 2022 21:37:12.094 # Bio thread for job type #0 terminated
1:M 22 Jan 2022 21:37:12.095 # Bio thread for job type #1 terminated
1:M 22 Jan 2022 21:37:12.095 # Bio thread for job type #2 terminated
*** Preparing to test memory region 4000213000 (2682880 bytes)
*** Preparing to test memory region 4001c6c000 (12288 bytes)
*** Preparing to test memory region 4001db5000 (8192 bytes)
*** Preparing to test memory region 4002140000 (16384 bytes)
*** Preparing to test memory region 4002162000 (16384 bytes)
*** Preparing to test memory region 4002327000 (40960 bytes)
*** Preparing to test memory region 4002600000 (2097152 bytes)
*** Preparing to test memory region 4002931000 (2097152 bytes)
*** Preparing to test memory region 4003000000 (4194304 bytes)
*** Preparing to test memory region 4003938000 (12288 bytes)
*** Preparing to test memory region 400393c000 (8388608 bytes)
*** Preparing to test memory region 400413d000 (8388608 bytes)
*** Preparing to test memory region 400493e000 (8388608 bytes)
*** Preparing to test memory region 400513f000 (8388608 bytes)
*** Preparing to test memory region 4005940000 (8388608 bytes)
*** Preparing to test memory region 4006140000 (5767168 bytes)
*** Preparing to test memory region 400740b000 (3670016 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: dictSdsKeyCompare (base: 0x400004b610)
Module: /usr/local/bin/redis-server *:6379 (base 0x4000000000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x400004b610 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 22 Jan 2022 21:37:12.394 # dump of function (hexdump of 180 bytes):
4889f74889d60fb657ff89d083e0073c040f879f7bffff488d0df2fb14000fb6c0486304814801c8ffe0660f1f440000486357f70fb64eff89c883e0073c040f876a7bffff4c8d05d8fb14000fb6c0496304804c01c0ffe00f1f8400000000008b4ef731c039ca75214883ec08e8ee6cffff85c00f94c04883c4080fb6c0c3900fb74efb31c039ca74dfc30f1f4400000fb64efdebcd662e0f1f840000000000c0e9030fb6c9ebbb0f1f8400000000008b4eefeb
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
This from running a stock ReJSON docker container, with no other modules. Most of the redis interaction is from the Bull queuing library
Comment From: oranagra
@MeirShpilraien can you please take a look?
Comment From: MeirShpilraien
@codekrafter any chance you can share the script with this sha: 8f55ae4a3be429c6d38c5d5db3e80edf89197b64? Can help reproduce the issue.
Comment From: MeirShpilraien
Also notice that RediSearch and RedisJson is also loaded, we need to also considered that, might be a module that causes memory corruption ...
Comment From: codekrafter
@MeirShpilraien it should be this script: https://github.com/OptimalBits/bull/blob/v4.2.1/lib/commands/addJob-6.lua This is a crash from my local development environment (an M1 mac if that is of interest), but I know my production redis instance has been crashing every so often, I haven't gotten a chance to pull the logs from that yet but I would bet it is the same problem.
Comment From: PingXie
Well the crash log you posted is about a Redis server running on x64 Linux.
Based on the instruction where this crash occurred, the invalid memory pointer belongs to a key string that is already in the database. Assuming the physical memory module in your system is not flaky, this sounds like a use-after-free issue to me. If it reproduces relatively consistently on your local dev env, maybe try a few more repros with Valgrind?
0: 48 89 f7 mov rdi,rsi 3: 48 89 d6 mov rsi,rdx 6: 0f b6 57 ff movzx edx,BYTE PTR [rdi-0x1] a: 89 d0 mov eax,edx c: 83 e0 07 and eax,0x7 f: 3c 04 cmp al,0x4 11: 0f 87 9f 7b ff ff ja 0xffffffffffff7bb6 17: 48 8d 0d f2 fb 14 00 lea rcx,[rip+0x14fbf2] # 0x14fc10 1e: 0f b6 c0 movzx eax,al 21: 48 63 04 81 movsxd rax,DWORD PTR [rcx+rax4] 25: 48 01 c8 add rax,rcx 28: ff e0 jmp rax 2a: 66 0f 1f 44 00 00 nop WORD PTR [rax+rax1+0x0] 30: 48 63 57 f7 movsxd rdx,DWORD PTR [rdi-0x9] 34: 0f b6 4e ff movzx ecx,BYTE PTR [rsi-0x1] // <--- here %rsi is key2 in dictSdsKeyCompare,
%rsi = 0x4000166b
static int dictSdsKeyCompare(void privdata, const void key1, const void *key2)
which is invoked by
static long _dictKeyIndex(dict d, const void key, uint64_t hash, dictEntry existing) { ... if (key==he->key || dictCompareKeys(d, key, he->key**)) { // <-- here he->key is key2 ... }
Comment From: MeirShpilraien
@codekrafter I do not see anything suspicious in the Lua script. I suspects its the modules fault. Is it possible to run without the module (just to try to prove/disprove this theory)?
Comment From: codekrafter
@MeirShpilraien most of my app's interaction with redis is through module commands, which makes it hard to test the application without the module.
Comment From: MeirShpilraien
@codekrafter any chance you have a deterministic reproduction steps?
Comment From: sundb
@codekrafter It looks like this crash is similar to #10341. If you are running docker on M1, can you confirm if you are using qemu? What is the redis docker image you are using?
Comment From: codekrafter
@sundb How would I confirm I am using qemu? I believe so as my laptop is m1 but it is running as x86. It is using the redislabs/rejson container.
Comment From: sundb
@codekrafter You can tell by the warning shown in the docker. ref comment: https://github.com/redis/redis/issues/10341#issuecomment-1053789795
Comment From: codekrafter
@sundb can confirm it is running via qemu
Comment From: sundb
@codekrafter It seems to be necessary to make an issue with https://github.com/RedisJSON/RedisJSON and ask them to build an image that supports arm, otherwise it will definitely have problems running on m1 docker.
Comment From: oranagra
i'm closing all of these reports, should be tracked in the ReJson repo. @sundb are all these reports about Arm / docker issues coming from the JSON / module dockers? or any of them are plain redis?
Comment From: sundb
@oranagra ~~RedisJson and RedisSearch, no plain redis.~~
9561 plain redis
10341 plain redis
9187 RedisGraph
10162 RedisSearch
10442 RedisJson
Here are the available ARM docker image:
plain redis: hub.docker.com/_/redis modules: hub.docker.com/r/redis/redis-stack-server
The above version may not be the latest, someon can find it on the docker hub if needed.
Comment From: oranagra
I see you mentioned that ReJson doesn't have an ARM image, does redis have one? if we can't fix this by creating a docker image, i think we need to solve this by preventing redis from running (these reports are starting to be annoying).
I think we should find a way to detect this issue, and abort redis on startup with a clear message. like we did in #8224.
i suppose that being under qemu, we don't have a good way to detect that combination, but maybe we can just do a calloc() and check if the memory is zeroed?
can someone with spare time and an M1 machine test this?
Comment From: sundb
@oranagra I modified the above comment to add the available images. I remembered a silly thing on the bike ride, jemalloc already gives us a way to detect memory not zero in https://github.com/jemalloc/jemalloc/pull/2005/files, we can actually use memset in zcalloc when it is detected. Wait for me to make a pr.
Comment From: oranagra
@sundb i don't think it's wise to add code to memset on calloc, if that's what you meant. first, IIUC, the problem is not with any calloc, just when re-using an page that was previously dismissed, so doing that would mean that in some cases we zero twice, and in other cases we're missing out on lazy zeroing. secondly, even adding an additional branch in our low level function, just for better support of a platform that's not a realistic target for redis, would be wasteful.
Another alternative is to just enable the feature added in the jemalloc PR you mentioned (which causes jemalloc to never dismiss pages when running on an affected platform), that would fix the problem on qemu, by adding a sub-optimal effect on the runtime.
what i suggest is to copy that detection code from jemalloc into redis, and use it in a similar manner as we did in #8224.
If the user disables the warning, we can then enable that feature (disable trust_madvise)
Comment From: sundb
@oranagra I found some problems when I did with the way you said.
1) When the MADV_DONTNEED zeroes page failure is detected (it must be placed after the config loaded because we need to log something), it is actually too late, and jemalloc may have allocte some memory before this.
2) trust_madvis is read-only, I'm not sure jemalloc is trying to prevent us from setting it dynamically, as in the question in 1).
Please correct me if I'm wrong, as I'm not familiar with jemalloc.
Comment From: oranagra
@sundb i don't want to invest too much effort (and code) in this, if it's too late to tweak jemalloc config, let's drop this idea.
if we do this detection, logging and exit after we already allocated things, there's also a slim chance that some calloc already failed zeroing memory (but that's actually unlikely, since it's unlikely that we used MADV_DONTNEED).
i think the most important requirements here are: 1. avoid letting redis start and later crash at runtime (after being populated with data) 2. avoid people opening more issues about odd crashes.
from that perspective, i think it's fine to do that check after init, and forget about the idea of setting trust_madvise to 0 when the warning (exit()) was silenced.
in theory, people can provide jemalloc with a config from outside (there's a mechanism for that), and set trust_madvise. in theory we can detect that it has been disabled, and silence our warning too, but i guess i don't mind requiring these people to also silence the warning in redis config file separately.
Comment From: oranagra
@sundb i understand some progress have been made, and since we posted quite a few links to this comment from other pages and people are likely to hit it, i'd like to ask you to edit / update it.
it currently refers to redisfab which i understand is a testing playground, and not something seriously maintained.
instead, the official multi-platform docker images we should direct people to are:
* plain redis: https://hub.docker.com/_/redis
* modules: https://hub.docker.com/r/redis/redis-stack-server
Comment From: sundb
@oranagra Already updated.