Crash report
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 02 Nov 2022 21:26:40.719 # Redis 6.2.7 crashed by signal: 6, si_code: -6
1:M 02 Nov 2022 21:26:40.719 # Crashed running the instruction at: 0x7f52b5012ce1
free(): double free detected in tcache 2
------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f52b5012ce1]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f52b51c2140]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x141)[0x7f52b5012ce1]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x123)[0x7f52b4ffc537]
/lib/x86_64-linux-gnu/libc.so.6(+0x7b768)[0x7f52b5055768]
/lib/x86_64-linux-gnu/libc.so.6(+0x82a5a)[0x7f52b505ca5a]
/lib/x86_64-linux-gnu/libc.so.6(+0x84055)[0x7f52b505e055]
/usr/lib/redis/modules/redisearch.so(HybridIterator_Free+0xb2)[0x7f52b42e2d32]
/usr/lib/redis/modules/redisearch.so(AREQ_Free+0x3c)[0x7f52b42be18c]
/usr/lib/redis/modules/redisearch.so(+0xfccbc)[0x7f52b42b8cbc]
redis-server *:6379(RedisModuleCommandDispatcher+0x53)[0x55d6da33fb73]
redis-server *:6379(call+0xdb)[0x55d6da2b425b]
redis-server *:6379(processCommand+0x5a3)[0x55d6da2b5e73]
redis-server *:6379(processInputBuffer+0xf8)[0x55d6da2c9668]
redis-server *:6379(+0xfc458)[0x55d6da361458]
redis-server *:6379(aeProcessEvents+0x292)[0x55d6da2ace32]
redis-server *:6379(aeMain+0x1d)[0x55d6da2ad09d]
redis-server *:6379(main+0x316)[0x55d6da2a92b6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f52b4ffdd0a]
redis-server *:6379(_start+0x2a)[0x55d6da2a978a]
------ REGISTERS ------
1:M 02 Nov 2022 21:26:40.720 #
RAX:0000000000000000 RBX:00007f52b4fd5f80
RCX:00007f52b5012ce1 RDX:0000000000000000
RDI:0000000000000002 RSI:00007ffcf33043f0
RBP:00007ffcf3304740 RSP:00007ffcf33043f0
R8 :0000000000000000 R9 :00007ffcf33043f0
R10:0000000000000008 R11:0000000000000246
R12:00007ffcf3304660 R13:0000000000001000
R14:0000000000000010 R15:00007f52b56cf000
RIP:00007f52b5012ce1 EFL:0000000000000246
CSGSFS:002b000000000033
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043ff) -> 00007ffcf3304500
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043fe) -> 000055d6dbf490e0
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043fd) -> 0000000000022e0a
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043fc) -> 00007f5277f3b448
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043fb) -> 0000000000000001
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043fa) -> 0000000000000007
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f9) -> 00007f52b45ed770
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f8) -> 00007f52b45ed530
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f7) -> 000055d6dbf48c40
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f6) -> 00007f5277f3b448
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f5) -> 00007f52b51bec00
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f4) -> 0000000000000000
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f3) -> 000055d6da3d0882
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f2) -> 00007f52b4c009c0
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f1) -> 00007f51b3bcc0c0
1:M 02 Nov 2022 21:26:40.720 # (00007ffcf33043f0) -> 0000000000000000
------ INFO OUTPUT ------
# Server
redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c3169c49e62a4f59
redis_mode:standalone
os:Linux 5.4.0-125-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:4fba6425b161088ef0529133f2b38e72c50cbed7
tcp_port:6379
server_time_usec:1667424400218738
uptime_in_seconds:1311
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:6480016
executable:/data/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:200
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:1858239496
used_memory_human:1.73G
used_memory_rss:1796923392
used_memory_rss_human:1.67G
used_memory_peak:1858626648
used_memory_peak_human:1.73G
used_memory_peak_perc:99.98%
used_memory_overhead:49316184
used_memory_startup:906896
used_memory_dataset:1808923312
used_memory_dataset_perc:97.39%
allocator_allocated:1858705744
allocator_active:1859268608
allocator_resident:1928785920
total_system_memory:32879435776
total_system_memory_human:30.62G
used_memory_lua:32768
used_memory_lua_human:32.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:562864
allocator_rss_ratio:1.04
allocator_rss_bytes:69517312
rss_overhead_ratio:0.93
rss_overhead_bytes:-131862528
mem_fragmentation_ratio:0.97
mem_fragmentation_bytes:-61453112
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20680
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:6930432
current_cow_size_age:6
current_fork_perc:59.69
current_save_keys_processed:594945
current_save_keys_total:996719
rdb_changes_since_last_save:137010
rdb_bgsave_in_progress:1
rdb_last_save_time:1667424333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:8
rdb_current_bgsave_time_sec:6
rdb_last_cow_size:718512128
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:18
total_commands_processed:1000005
instantaneous_ops_per_sec:511
total_net_input_bytes:481890660
total_net_output_bytes:4001446
instantaneous_input_kbps:240.91
instantaneous_output_kbps:2.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:10
evicted_keys:0
keyspace_hits:1000025
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:53659
total_forks:20
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:2
dump_payload_sanitizations:0
total_reads_processed:1000021
total_writes_processed:1000004
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:14ac7f0ddbae66aa1b031cd3f0a2525c5098f1ad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:221.264992
used_cpu_user:1105.431909
used_cpu_sys_children:9.219990
used_cpu_user_children:70.532210
used_cpu_sys_main_thread:175.483276
used_cpu_user_main_thread:1103.217163
# Modules
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20415,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
# Commandstats
cmdstat_info:calls=1,usec=41,usec_per_call=41.00,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=1,usec=645,usec_per_call=645.00,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=1,usec=2541,usec_per_call=2541.00,rejected_calls=0,failed_calls=0
cmdstat_FT.DROP:calls=2,usec=23,usec_per_call=11.50,rejected_calls=0,failed_calls=2
cmdstat_hset:calls=1000000,usec=1237374908,usec_per_call=1237.37,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_Unknown:count=2
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=1000000,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=7 addr=10.0.0.3:37008 laddr=192.168.64.2:6379 fd=8 name= age=1279 idle=1279 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20680 events=r cmd=FT.CREATE user=default redir=-1
id=58 addr=10.0.0.3:37044 laddr=192.168.64.2:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=750 qbuf-free=40204 argv-mem=584 obl=8 oll=0 omem=0 tot-mem=62248 events=r cmd=FT.SEARCH user=default redir=-1
------ CURRENT CLIENT INFO ------
id=58 addr=10.0.0.3:37044 laddr=192.168.64.2:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=750 qbuf-free=40204 argv-mem=584 obl=8 oll=0 omem=0 tot-mem=62248 events=r cmd=FT.SEARCH user=default redir=-1
argv[0]: 'FT.SEARCH'
argv[1]: 'idx'
argv[2]: '(@a:$a_1 @b:$b_2)=>[KNN $K @vector $vec_param EF_RUNTIME $EF AS vector_score]'
argv[3]: 'RETURN'
argv[4]: '1'
argv[5]: 'vector_score'
argv[6]: 'SORTBY'
argv[7]: 'vector_score'
argv[8]: 'DESC'
argv[9]: 'DIALECT'
argv[10]: '2'
argv[11]: 'LIMIT'
argv[12]: '0'
argv[13]: '1'
argv[14]: 'params'
argv[15]: '10'
argv[16]: 'vec_param'
argv[17]: 'f��>���>ʂ�>�)�=�N?kϲ=w�>t�t?��>o��>z�}#>��I?G"7?��]=4*�>6_?!��>�~=��r>Xݓ>��}?���>�T|>�rW?oN�>7�>�MX?�Rh>��v?��m?�ɍ>�b?J�>X��>Wϩ=
�<��?k�0>���=�'
argv[18]: 'K'
argv[19]: '1'
argv[20]: 'EF'
argv[21]: '64'
argv[22]: 'b_2'
argv[23]: 'YXqIM'
argv[24]: 'a_1'
argv[25]: 'sExTa'
------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.1/src/lib.rs:73:37
1: rejson::__info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.1/src/macros.rs:120:13
2: modulesCollectInfo
at /usr/src/redis/src/module.c:7100:9
3: logModulesInfo
at /usr/src/redis/src/debug.c:1598:22
4: printCrashReport
at /usr/src/redis/src/debug.c:1849:5
sigsegvHandler
at /usr/src/redis/src/debug.c:1831:5
5: <unknown>
6: gsignal
7: abort
8: <unknown>
9: <unknown>
10: <unknown>
11: HybridIterator_Free
at /build/src/hybrid_reader.c:368:17
12: AREQ_Free
at /build/src/aggregate/aggregate_request.c:1356:5
13: execCommandCommon
at /build/src/aggregate/aggregate_exec.c:433:5
14: RedisModuleCommandDispatcher
at /usr/src/redis/src/module.c:695:5
15: call
at /usr/src/redis/src/server.c:3744:5
16: processCommand
at /usr/src/redis/src/server.c:4291:9
17: processCommandAndResetClient
at /usr/src/redis/src/networking.c:2102:9
processInputBuffer
at /usr/src/redis/src/networking.c:2203:17
18: callHandler
at /usr/src/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /usr/src/redis/src/connection.c:295:14
19: aeProcessEvents
at /usr/src/redis/src/ae.c:427:17
20: aeMain
at /usr/src/redis/src/ae.c:487:9
21: main
at /usr/src/redis/src/server.c:6456:5
22: __libc_start_main
23: _start
# search_version
search_RedisSearch_version:6.2.7
# search_index
search_number_of_indexes:1
# search_fields_statistics
search_fields_text:Text=2
search_fields_vector:Vector=1,HSNW=1
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
------ FAST MEMORY TEST ------
1:M 02 Nov 2022 21:26:40.754 # Bio thread for job type #0 terminated
1:M 02 Nov 2022 21:26:40.754 # Bio thread for job type #1 terminated
1:M 02 Nov 2022 21:26:40.755 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55d6da479000 (2281472 bytes)
*** Preparing to test memory region 55d6dbf26000 (368640 bytes)
*** Preparing to test memory region 7f5183800000 (4773253120 bytes)
*** Preparing to test memory region 7f52a4200000 (8388608 bytes)
*** Preparing to test memory region 7f52a4a00000 (4194304 bytes)
*** Preparing to test memory region 7f52a4eb6000 (204472320 bytes)
*** Preparing to test memory region 7f52b11b7000 (8388608 bytes)
*** Preparing to test memory region 7f52b19b8000 (8388608 bytes)
*** Preparing to test memory region 7f52b21b9000 (8388608 bytes)
*** Preparing to test memory region 7f52b29ba000 (8388608 bytes)
*** Preparing to test memory region 7f52b31bb000 (8388608 bytes)
*** Preparing to test memory region 7f52b39bc000 (8388608 bytes)
*** Preparing to test memory region 7f52b45f9000 (28672 bytes)
*** Preparing to test memory region 7f52b4600000 (8388608 bytes)
*** Preparing to test memory region 7f52b4fd4000 (24576 bytes)
*** Preparing to test memory region 7f52b51ab000 (16384 bytes)
*** Preparing to test memory region 7f52b51cd000 (16384 bytes)
*** Preparing to test memory region 7f52b54c1000 (16384 bytes)
*** Preparing to test memory region 7f52b56a2000 (8192 bytes)
*** Preparing to test memory region 7f52b56cf000 (4096 bytes)
*** Preparing to test memory region 7f52b56d2000 (4096 bytes)
.O.O.41:C 02 Nov 2022 21:26:42.750 * DB saved on disk
41:C 02 Nov 2022 21:26:42.789 * RDB: 695 MB of memory used by copy-on-write
O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: gsignal (base: 0x7f52b5012ba0)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7f52b4fda000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7f52b5012ba0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 02 Nov 2022 21:26:57.247 # dump of function (hexdump of 449 bytes):
4881ec180100004189f841ba0800000031ff64488b042528000000488984240801000031c04989e1488db4248000000048b8ffffff7ffeffffff48898424800000004c89cab80e00000048c7842488000000ffffffff48c7842490000000ffffffff48c7842498000000ffffffff48c78424a0000000ffffffff48c78424a8000000ffffffff48c78424b0000000ffffffff48c78424b8000000ffffffff48c78424c0000000ffffffff48c78424c8000000ffffffff48c78424d0000000ffffffff48c78424d8000000ffffffff48c78424e0000000ffffffff48c78424e8000000ffffffff48c78424f0000000ffffffff48c78424f8000000ffffffff0f05b92700000089c80f054889c7b8ba0000000f0589c64489c2b8ea0000000f05483d00f0ffff77394189c041ba0800000031d24c89cebf02000000b80e0000000f05488b84240801000064482b04252800000075204489c04881c418010000c390488b1569611900f7d841b8ffffffff648902ebb6e8b7380d000f1f800000000085ff780cf7dfe9550300000f1f440000488b053961190064c70016000000b8ffffffffc3662e0f1f84000000000066909066662e0f1f8400000000000f1f400048
Function at 0x7f52b50e65d0 is __stack_chk_fail
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
1:M 02 Nov 2022 21:26:57.247 # Redis 6.2.7 crashed by signal: 11, si_code: 128
1:M 02 Nov 2022 21:26:57.247 # Accessing address: (nil)
1:M 02 Nov 2022 21:26:57.247 # Crashed running the instruction at: 0x7f52b4ffc602
------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(abort+0x1ee)[0x7f52b4ffc602]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f52b51c2140]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x1ee)[0x7f52b4ffc602]
/lib/x86_64-linux-gnu/libc.so.6(+0x7b768)[0x7f52b5055768]
/lib/x86_64-linux-gnu/libc.so.6(+0x82a5a)[0x7f52b505ca5a]
/lib/x86_64-linux-gnu/libc.so.6(+0x84055)[0x7f52b505e055]
/usr/lib/redis/modules/redisearch.so(HybridIterator_Free+0xb2)[0x7f52b42e2d32]
/usr/lib/redis/modules/redisearch.so(AREQ_Free+0x3c)[0x7f52b42be18c]
/usr/lib/redis/modules/redisearch.so(+0xfccbc)[0x7f52b42b8cbc]
redis-server *:6379(RedisModuleCommandDispatcher+0x53)[0x55d6da33fb73]
redis-server *:6379(call+0xdb)[0x55d6da2b425b]
redis-server *:6379(processCommand+0x5a3)[0x55d6da2b5e73]
redis-server *:6379(processInputBuffer+0xf8)[0x55d6da2c9668]
redis-server *:6379(+0xfc458)[0x55d6da361458]
redis-server *:6379(aeProcessEvents+0x292)[0x55d6da2ace32]
redis-server *:6379(aeMain+0x1d)[0x55d6da2ad09d]
redis-server *:6379(main+0x316)[0x55d6da2a92b6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f52b4ffdd0a]
redis-server *:6379(_start+0x2a)[0x55d6da2a978a]
------ REGISTERS ------
1:M 02 Nov 2022 21:26:57.248 #
RAX:0000000000000000 RBX:00007f52b4fd5f80
RCX:00007f52b5012ce1 RDX:0000000000000000
RDI:0000000000000002 RSI:00007ffcf33043f0
RBP:00007ffcf3304740 RSP:00007ffcf3304510
R8 :0000000000000000 R9 :00007ffcf33043f0
R10:0000000000000008 R11:0000000000000246
R12:00007ffcf3304660 R13:0000000000001000
R14:0000000000000010 R15:00007f52b56cf000
RIP:00007f52b4ffc602 EFL:0000000000010246
CSGSFS:002b000000000033
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451f) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451e) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451d) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451c) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451b) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf330451a) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304519) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304518) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304517) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304516) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304515) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304514) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304513) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304512) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304511) -> 0000000000000000
1:M 02 Nov 2022 21:26:57.248 # (00007ffcf3304510) -> 0000000000000020
------ INFO OUTPUT ------
# Server
redis_version:6.2.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:c3169c49e62a4f59
redis_mode:standalone
os:Linux 5.4.0-125-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:1
process_supervised:no
run_id:4fba6425b161088ef0529133f2b38e72c50cbed7
tcp_port:6379
server_time_usec:1667424400218738
uptime_in_seconds:1311
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:6480016
executable:/data/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:200
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:1868107064
used_memory_human:1.74G
used_memory_rss:1796923392
used_memory_rss_human:1.67G
used_memory_peak:1868107064
used_memory_peak_human:1.74G
used_memory_peak_perc:100.51%
used_memory_overhead:49316184
used_memory_startup:906896
used_memory_dataset:1818790880
used_memory_dataset_perc:97.41%
allocator_allocated:1858705744
allocator_active:1859268608
allocator_resident:1928785920
total_system_memory:32879435776
total_system_memory_human:30.62G
used_memory_lua:32768
used_memory_lua_human:32.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:562864
allocator_rss_ratio:1.04
allocator_rss_bytes:69517312
rss_overhead_ratio:0.93
rss_overhead_bytes:-131862528
mem_fragmentation_ratio:0.97
mem_fragmentation_bytes:-61453112
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20680
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:6930432
current_cow_size_age:23
current_fork_perc:59.69
current_save_keys_processed:594945
current_save_keys_total:996719
rdb_changes_since_last_save:137010
rdb_bgsave_in_progress:1
rdb_last_save_time:1667424333
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:8
rdb_current_bgsave_time_sec:23
rdb_last_cow_size:718512128
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:18
total_commands_processed:1000005
instantaneous_ops_per_sec:511
total_net_input_bytes:481890660
total_net_output_bytes:4001446
instantaneous_input_kbps:240.91
instantaneous_output_kbps:2.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:10
evicted_keys:0
keyspace_hits:1000025
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:53659
total_forks:20
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:2
dump_payload_sanitizations:0
total_reads_processed:1000021
total_writes_processed:1000004
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:14ac7f0ddbae66aa1b031cd3f0a2525c5098f1ad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:224.754000
used_cpu_user:1118.492546
used_cpu_sys_children:9.219990
used_cpu_user_children:70.532210
used_cpu_sys_main_thread:178.951018
used_cpu_user_main_thread:1116.274629
# Modules
module:name=ReJSON,ver=999999,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20415,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
# Commandstats
cmdstat_info:calls=1,usec=41,usec_per_call=41.00,rejected_calls=0,failed_calls=0
cmdstat_FT.CREATE:calls=1,usec=645,usec_per_call=645.00,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=1,usec=2541,usec_per_call=2541.00,rejected_calls=0,failed_calls=0
cmdstat_FT.DROP:calls=2,usec=23,usec_per_call=11.50,rejected_calls=0,failed_calls=2
cmdstat_hset:calls=1000000,usec=1237374908,usec_per_call=1237.37,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_Unknown:count=2
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=1000000,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=7 addr=10.0.0.3:37008 laddr=192.168.64.2:6379 fd=8 name= age=1279 idle=1279 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20680 events=r cmd=FT.CREATE user=default redir=-1
id=58 addr=10.0.0.3:37044 laddr=192.168.64.2:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=750 qbuf-free=40204 argv-mem=584 obl=8 oll=0 omem=0 tot-mem=62248 events=r cmd=FT.SEARCH user=default redir=-1
------ CURRENT CLIENT INFO ------
id=58 addr=10.0.0.3:37044 laddr=192.168.64.2:6379 fd=9 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=750 qbuf-free=40204 argv-mem=584 obl=8 oll=0 omem=0 tot-mem=62248 events=r cmd=FT.SEARCH user=default redir=-1
argv[0]: 'FT.SEARCH'
argv[1]: 'idx'
argv[2]: '(@a:$a_1 @b:$b_2)=>[KNN $K @vector $vec_param EF_RUNTIME $EF AS vector_score]'
argv[3]: 'RETURN'
argv[4]: '1'
argv[5]: 'vector_score'
argv[6]: 'SORTBY'
argv[7]: 'vector_score'
argv[8]: 'DESC'
argv[9]: 'DIALECT'
argv[10]: '2'
argv[11]: 'LIMIT'
argv[12]: '0'
argv[13]: '1'
argv[14]: 'params'
argv[15]: '10'
argv[16]: 'vec_param'
argv[17]: 'f��>���>ʂ�>�)�=�N?kϲ=w�>t�t?��>o��>z�}#>��I?G"7?��]=4*�>6_?!��>�~=��r>Xݓ>��}?���>�T|>�rW?oN�>7�>�MX?�Rh>��v?��m?�ɍ>�b?J�>X��>Wϩ=
�<��?k�0>���=�'
argv[18]: 'K'
argv[19]: '1'
argv[20]: 'EF'
argv[21]: '64'
argv[22]: 'b_2'
argv[23]: 'YXqIM'
argv[24]: 'a_1'
argv[25]: 'sExTa'
------ MODULES INFO OUTPUT ------
# ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.1/src/lib.rs:73:37
1: rejson::__info_func
at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/redis-module-1.0.1/src/macros.rs:120:13
2: modulesCollectInfo
at /usr/src/redis/src/module.c:7100:9
3: logModulesInfo
at /usr/src/redis/src/debug.c:1598:22
4: printCrashReport
at /usr/src/redis/src/debug.c:1849:5
sigsegvHandler
at /usr/src/redis/src/debug.c:1831:5
5: <unknown>
6: abort
7: <unknown>
8: <unknown>
9: <unknown>
10: HybridIterator_Free
at /build/src/hybrid_reader.c:368:17
11: AREQ_Free
at /build/src/aggregate/aggregate_request.c:1356:5
12: execCommandCommon
at /build/src/aggregate/aggregate_exec.c:433:5
13: RedisModuleCommandDispatcher
at /usr/src/redis/src/module.c:695:5
14: call
at /usr/src/redis/src/server.c:3744:5
15: processCommand
at /usr/src/redis/src/server.c:4291:9
16: processCommandAndResetClient
at /usr/src/redis/src/networking.c:2102:9
processInputBuffer
at /usr/src/redis/src/networking.c:2203:17
17: callHandler
at /usr/src/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /usr/src/redis/src/connection.c:295:14
18: aeProcessEvents
at /usr/src/redis/src/ae.c:427:17
19: aeMain
at /usr/src/redis/src/ae.c:487:9
20: main
at /usr/src/redis/src/server.c:6456:5
21: __libc_start_main
22: _start
# search_version
search_RedisSearch_version:6.2.7
# search_index
search_number_of_indexes:1
# search_fields_statistics
search_fields_text:Text=2
search_fields_vector:Vector=1,HSNW=1
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
------ FAST MEMORY TEST ------
*** Preparing to test memory region 55d6da479000 (2281472 bytes)
*** Preparing to test memory region 55d6dbf26000 (368640 bytes)
*** Preparing to test memory region 7f5183800000 (4773253120 bytes)
*** Preparing to test memory region 7f52a4200000 (8388608 bytes)
*** Preparing to test memory region 7f52a4a00000 (4194304 bytes)
*** Preparing to test memory region 7f52a4eb6000 (204472320 bytes)
*** Preparing to test memory region 7f52b11b7000 (8388608 bytes)
*** Preparing to test memory region 7f52b19b8000 (8388608 bytes)
*** Preparing to test memory region 7f52b21b9000 (8388608 bytes)
*** Preparing to test memory region 7f52b29ba000 (8388608 bytes)
*** Preparing to test memory region 7f52b31bb000 (8388608 bytes)
*** Preparing to test memory region 7f52b39bc000 (8388608 bytes)
*** Preparing to test memory region 7f52b45f9000 (28672 bytes)
*** Preparing to test memory region 7f52b4600000 (8388608 bytes)
*** Preparing to test memory region 7f52b4fd4000 (24576 bytes)
*** Preparing to test memory region 7f52b51ab000 (16384 bytes)
*** Preparing to test memory region 7f52b51cd000 (16384 bytes)
*** Preparing to test memory region 7f52b54c1000 (16384 bytes)
*** Preparing to test memory region 7f52b56a2000 (8192 bytes)
*** Preparing to test memory region 7f52b56cf000 (4096 bytes)
*** Preparing to test memory region 7f52b56d2000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: abort (base: 0x7f52b4ffc414)
Module: /lib/x86_64-linux-gnu/libc.so.6 (base 0x7f52b4fda000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7f52b4ffc414 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 02 Nov 2022 21:27:09.764 # dump of function (hexdump of 622 bytes):
5364488b1c25100000004881ec2001000064488b042528000000488984241801000031c048391d79e91a00743e648b04251800000085c0750eba010000000fb11557e91a00eb1d31c0ba01000000f00fb11546e91a00740c488d3d3de91a00e8c8f8050048891d39e91a00ff052fe91a00833d34e91a00007532488d542408b91e00000031c04889e64889d731d2c70514e91a0001000000f3abbf0100000048c7042420000000e8c06a01008b05fae81a0083f8010f85c0000000c705e7e81a00000000008b05d5e81a00ffc88905cde81a00754448c705c4e81a0000000000648b14251800000085d27508ff0daae81a00eb258705a2e81a00ffc87e1b4531d2ba01000000be81000000b8ca000000488d3d85e81a000f05bf06000000e86966010064488b1c251000000048391d71e81a00743e648b04251800000085c0750eba010000000fb1154fe81a00eb1d31c0ba01000000f00fb1153ee81a00740c488d3d35e81a00e8c0f7050048891d31e81a00ff0527e81a00eb0583f802753e488db4248000000031c0b926000000c70513e81a0003000000488d9424880000004889f7f3ab4889d7b980000000b0fff3aa31d2bf06000000e87e690100833de7e71a00037514c705dbe71a0004000000bf06000000e8b1650100833dcae71a0004750bc705bee71a0005000000f4833db6e71a00057514c705aae71a0006000000bf7f000000e8106a0a00f4ebfde8ecfdffffe8e7fdffffe8e2fdffffe8ddfdffffe8d8fdffffe8d3fdffffe8cefdffffe8c9fdffffe8c4fdffffe8bffdffffe8bafdffffe8b5fdffffe8b0fdffffe8abfdffffe8a6fdffffe8a1fdffffe89cfdffffe897fdfffff745000080
Function at 0x7f52b5012f80 is sigprocmask
Function at 0x7f52b4ffc4f1 is abort
Function at 0x7f52b4ffc50c is abort
Function at 0x7f52b5012ba0 is gsignal
Function at 0x7f52b5012f50 is sigaction
Function at 0x7f52b50a3030 is _exit
Function at 0x7f52b4ffc414 is abort
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
- Ubuntu 20.04.5 LTS, running
redislabs/redisearch:2.4.15in a Docker container (single node). - The issue occurs while querying the index with a hybrid search. The index has 100k entries with vectors and two attributes each. It fails on the following query:
FT.SEARCH idx "(@a:$a_1 @b:$b_2)=>[KNN $K @vector $vec_param EF_RUNTIME $EF AS vector_score]" RETURN 1 vector_score SORTBY vector_score DESC DIALECT 2 LIMIT 0 1 params 10 vec_param "f\xf1\x93>\xeb\xf6\xaf>\xca\x82\xee>\xba)\xf7=\xf7N\x1f?k\xcf\xb2=\x1ew\xaf>t\x8ct?\xb0\x16\x9a>o\xe4\x92>z\x1b\x19?\xe3}#>\xea\x99I?G"7?\x97\xcb]=4*\xfd>\x166_?!\x8b\xfd>\x03\x1a~=\xeb\xf9r>X\xdd\x93>\x94\x90}?\xae\x91\xd3>\xb9T|>\x84rW?oN\xc6>\x1d7\xa3>\xeaMX?\xffRh>\xfe\xf3v?\x9e\xf2m?\xb1\xc9\x8d>\xc7b\x1c?J\x7f\xb7>X\xeb\xb9>W\xcf\xa9=\x1f\x0c\xb8<\x90\xe1\x19?k\xcb0>\xb9\xa0\xb8=\x8b\x00\x0f>\xd3{h?\xcb,\xdc>\x0b\xe8\x06?\x90\xcc\x9d>B\xa3O>\xc6\xabf?\xb5\xe9$?a\x9c\xf8>\xceIl?\xd2\xa0\xb8=\xfe\xafm?\xb6\x9a\x90>>k\x89>\xb9\x13->T\xa7\x1a?\x81"&?Oc\x10?\xe0\xe7T>`\xd0\x0e>\'\xd5\xa6>H\x1a\n?S\x1c?>p\x87\x02>\xcc\x1fC?l\xca(?\xf9\xb4Z?\xaf)\xfe=\xa5\x1f#?\xb4\tb>\x94Ou=O\tL>\xea\xbd\x8e>~X\x86=o\xf0\x92<\xef\xd4)>\x88\x0e\x99>\xb5t<?\xfc\xa9\xe4=9\x00\x87<\x04EH?t%r?"\xe9-?\xc8#\x82=\x08\x95\'?:\n\x87=$3\xfb:\xf4\r\xd0>\x81\xdf/?\x08c\x8e>\x0e\xf54?f>O>K\x18P?\x95K$?Q\xc0\x0c?\x9a\xe16?\xb6 g?\xc0TS?\xcaV\x02>\xb8\x98e?" K 1 EF 64 b_2 "YXqIM" a_1 "sExTa"
Comment From: oranagra
@kacperlukawski please open in https://github.com/RediSearch/RediSearch @MeirShpilraien FYI