Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 02 May 2022 23:56:09.068 # Redis 6.2.6 crashed by signal: 11, si_code: 128
1:M 02 May 2022 23:56:09.068 # Accessing address: (nil)
1:M 02 May 2022 23:56:09.068 # Crashed running the instruction at: 0x55e68f61fdfe
------ STACK TRACE ------
EIP:
redis-server *:6380(RM_StringPtrLen+0xe)[0x55e68f61fdfe]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f62fe1c2140]
redis-server *:6380(RM_StringPtrLen+0xe)[0x55e68f61fdfe]
/usr/lib/redis/modules/redisearch.so(RSValue_Equal+0x73)[0x7f62fd3272e3]
/usr/lib/redis/modules/redisearch.so(+0x101edd)[0x7f62fd2c5edd]
/usr/lib/redis/modules/redisearch.so(Indexes_FindMatchingSchemaRules+0x1a7)[0x7f62fd318817]
/usr/lib/redis/modules/redisearch.so(IndexSpec_UpdateMatchingWithSchemaRules+0x3c)[0x7f62fd318abc]
/usr/lib/redis/modules/redisearch.so(+0x154d6a)[0x7f62fd318d6a]
redis-server *:6380(+0xd5c42)[0x55e68f61ec42]
redis-server *:6380(dictScan+0x22c)[0x55e68f59387c]
redis-server *:6380(RM_Scan+0x76)[0x55e68f61e306]
/usr/lib/redis/modules/redisearch.so(+0x15327d)[0x7f62fd31727d]
/usr/lib/redis/modules/redisearch.so(+0xf77ed)[0x7f62fd2bb7ed]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f62fe1b6ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f62fe0e6def]
------ REGISTERS ------
1:M 02 May 2022 23:56:09.068 #
RAX:d83b0c2000000008 RBX:00007f62f3ffd280
RCX:0000000000000064 RDX:0000000000000005
RDI:00007f62f9f55d28 RSI:00007f62f3ffd130
RBP:00007f62d83b0c20 RSP:00007f62f3ffd128
R8 :0000000000000008 R9 :00007f62d92c4800
R10:00007f62f3ffdeb0 R11:0000000000000000
R12:00007f62e0c47208 R13:00007f62d92c4810
R14:00007f62f3ffd270 R15:00007f62e0c47208
RIP:000055e68f61fdfe EFL:0000000000010206
CSGSFS:002b000000000033
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd137) -> 00007f62f3ffd230
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd136) -> 00007f62f3ffd280
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd135) -> 00007f62e1ffc2a0
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd134) -> 0000000000000000
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd133) -> 00007f62d92c47c0
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd132) -> 00007f62d83a3a70
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd131) -> 00007f62e1ffc2a0
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd130) -> 00007f62d839f500
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd12f) -> 0000000000000030
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd12e) -> 00007f62fd30ef37
1:M 02 May 2022 23:56:09.068 # (00007f62f3ffd12d) -> 00007f62f3ffd168
1:M 02 May 2022 23:56:09.069 # (00007f62f3ffd12c) -> 0000000000000000
1:M 02 May 2022 23:56:09.069 # (00007f62f3ffd12b) -> 00007f62d92c47e0
1:M 02 May 2022 23:56:09.069 # (00007f62f3ffd12a) -> 0000000000000000
1:M 02 May 2022 23:56:09.069 # (00007f62f3ffd129) -> 0000000000000001
1:M 02 May 2022 23:56:09.069 # (00007f62f3ffd128) -> 00007f62fd3272e3
------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9c335ca9779faba5
redis_mode:standalone
os:Linux 5.15.0-25-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:aeac49c3ebed7b417ecc556c027cbfd2836bf924
tcp_port:6380
server_time_usec:1651535769067815
uptime_in_seconds:1347
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7368601
executable:/data/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:380522520
used_memory_human:362.89M
used_memory_rss:406482944
used_memory_rss_human:387.65M
used_memory_peak:385697120
used_memory_peak_human:367.83M
used_memory_peak_perc:98.66%
used_memory_overhead:3432888
used_memory_startup:904760
used_memory_dataset:377089632
used_memory_dataset_perc:99.33%
allocator_allocated:381069064
allocator_active:389820416
allocator_resident:399089664
total_system_memory:28068499456
total_system_memory_human:26.14G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.02
allocator_frag_bytes:8751352
allocator_rss_ratio:1.02
allocator_rss_bytes:9269248
rss_overhead_ratio:1.02
rss_overhead_bytes:7393280
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:26143752
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1651535723
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:2449408
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:95
total_commands_processed:169240
instantaneous_ops_per_sec:0
total_net_input_bytes:524014101
total_net_output_bytes:7473825
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:13
evicted_keys:0
keyspace_hits:262659
keyspace_misses:6
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:4736
total_forks:16
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:34465
total_writes_processed:32491
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:b13db7ce74521f564af8fbee978fd9d3e3f5637e
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:2.630869
used_cpu_user:6.280312
used_cpu_sys_children:2.253349
used_cpu_user_children:7.465886
used_cpu_sys_main_thread:0.140462
used_cpu_user_main_thread:1.496979
# Modules
module:name=search,ver=20406,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
# Commandstats
cmdstat_FT.DROPINDEX:calls=21,usec=1418,usec_per_call=67.52,rejected_calls=0,failed_calls=3
cmdstat_hset:calls=34938,usec=1601237,usec_per_call=45.83,rejected_calls=0,failed_calls=0
cmdstat_info:calls=1,usec=12,usec_per_call=12.00,rejected_calls=0,failed_calls=0
cmdstat_scan:calls=1224,usec=270034,usec_per_call=220.62,rejected_calls=0,failed_calls=0
cmdstat_smembers:calls=1,usec=2218,usec_per_call=2218.00,rejected_calls=0,failed_calls=0
cmdstat_FT.TAGVALS:calls=6,usec=110,usec_per_call=18.33,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=346,usec=2245,usec_per_call=6.49,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=22,usec=1776,usec_per_call=80.73,rejected_calls=0,failed_calls=0
cmdstat_sadd:calls=36,usec=11661,usec_per_call=323.92,rejected_calls=0,failed_calls=0
cmdstat_del:calls=65407,usec=122754,usec_per_call=1.88,rejected_calls=0,failed_calls=0
cmdstat_FT.AGGREGATE:calls=492,usec=27490,usec_per_call=55.87,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=65388,usec=619283,usec_per_call=9.47,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=602,usec=708,usec_per_call=1.18,rejected_calls=0,failed_calls=0
cmdstat_zrange:calls=26,usec=128,usec_per_call=4.92,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=602,usec=2584389,usec_per_call=4293.00,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=128,usec=3551,usec_per_call=27.74,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_Unknown:count=3
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=50096,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=143 addr=192.168.1.48:55568 laddr=172.18.0.4:6380 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61664 events=r cmd=FT.CREATE user=default redir=-1
------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.0/src/lib.rs:78:33
1: rejson::__info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.0/src/macros.rs:120:13
2: modulesCollectInfo
at /usr/src/redis/src/module.c:7078:9
3: logModulesInfo
at /usr/src/redis/src/debug.c:1598:22
4: printCrashReport
at /usr/src/redis/src/debug.c:1849:5
sigsegvHandler
at /usr/src/redis/src/debug.c:1831:5
5: <unknown>
6: RM_StringPtrLen
at /usr/src/redis/src/module.c:1349:8
7: <unknown>
8: <unknown>
9: <unknown>
10: <unknown>
11: <unknown>
12: moduleScanCallback
at /usr/src/redis/src/module.c:7601:5
13: dictScan
at /usr/src/redis/src/dict.c:911:13
14: RM_Scan
at /usr/src/redis/src/module.c:7689:22
RM_Scan
at /usr/src/redis/src/module.c:7682:5
15: <unknown>
16: <unknown>
17: <unknown>
18: clone
------ FAST MEMORY TEST ------
1:M 02 May 2022 23:56:09.077 # main thread terminated
1:M 02 May 2022 23:56:09.077 # Bio thread for job type #0 terminated
1:M 02 May 2022 23:56:09.078 # Bio thread for job type #1 terminated
1:M 02 May 2022 23:56:09.078 # Bio thread for job type #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: RM_StringPtrLen (base: 0x55e68f61fdf0)
Module: redis-server *:6380 (base 0x55e68f549000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x55e68f61fdf0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 02 May 2022 23:56:09.078 # dump of function (hexdump of 142 bytes):
4885ff747b488b47084885f674300fb648ff89ca83e20780fa040f879dcbf6ff488d3d61de0d000fb6d2486314974801faffe20f1f4400008b48f748890ec390488b48efebf5662e0f1f840000000000c0e9030fb6c9ebe30f1f8400000000000fb648fdebd5662e0f1f8400000000000fb748fbebc5662e0f1f840000000000488d0551e10d004885f674b248c7
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
-
OS distribution and version Host: Ubuntu 20.04 (x86_64) Docker container: docker.io/redislabs/redisearch:2.4.6
-
Steps to reproduce (if any) randomly occurs
Comment From: rockyburt
Further investigation... this occurs when... 1. add > 16k hash keys 2. create 3 redisearch indexes 3. re-import same 16k hash keys 4. create new redisearch index, bam, redis-server crashes
Comment From: oranagra
@rockyburt please report in https://github.com/RediSearch/RediSearch @gkorland FYI
Comment From: ashtul
@rockyburt please reopen on RediSearch. Can you add the filters you have been using in the indexes schema? The crash is related to the filters. Thanks