Crash report Always crash when query the redis with high multi-get load, used_memory_human:6.64G
When crashed , I restart from dump.db, but when query comes, crash again
=== REDIS BUG REPORT START: Cut & paste starting from here ===
8:M 07 Sep 2023 16:04:22.982 # ------------------------------------------------
8:M 07 Sep 2023 16:04:22.982 # !!! Software Failure. Press left mouse button to continue
8:M 07 Sep 2023 16:04:22.982 # Guru Meditation: Redis aborting for OUT OF MEMORY. Allocating 18446744004990074880 bytes! #server.c:6037
------ STACK TRACE ------
Backtrace:
/opt/redis-stack/bin/redis-server *:6379(redisOutOfMemoryHandler+0x2f)[0x560cec8be20f]
/opt/redis-stack/bin/redis-server *:6379(zcalloc+0x2e)[0x560cec8cba0e]
/opt/redis-stack/lib/redisearch.so(RSIndexResult_IterateOffsets+0xf7)[0x7faab73991a7]
/opt/redis-stack/lib/redisearch.so(RSIndexResult_IterateOffsets+0x159)[0x7faab7399209]
/opt/redis-stack/lib/redisearch.so(RSIndexResult_IterateOffsets+0x159)[0x7faab7399209]
/opt/redis-stack/lib/redisearch.so(IndexResult_MinOffsetDelta+0x10a)[0x7faab7388a6a]
/opt/redis-stack/lib/redisearch.so(+0xcd51a)[0x7faab737751a]
/opt/redis-stack/lib/redisearch.so(+0x105119)[0x7faab73af119]
/opt/redis-stack/lib/redisearch.so(+0x1051f7)[0x7faab73af1f7]
/opt/redis-stack/lib/redisearch.so(+0x104cba)[0x7faab73aecba]
/opt/redis-stack/lib/redisearch.so(sendChunk+0xa6)[0x7faab734f906]
/opt/redis-stack/lib/redisearch.so(AREQ_Execute+0x1d)[0x7faab735005d]
/opt/redis-stack/lib/redisearch.so(RSSearchCommand+0xec)[0x7faab735086c]
/opt/redis-stack/bin/redis-server *:6379(RedisModuleCommandDispatcher+0x67)[0x560cec950417]
/opt/redis-stack/bin/redis-server *:6379(call+0xf0)[0x560cec8c1940]
/opt/redis-stack/bin/redis-server *:6379(processCommand+0x643)[0x560cec8c3643]
/opt/redis-stack/bin/redis-server *:6379(processCommandAndResetClient+0x20)[0x560cec8d6d20]
/opt/redis-stack/bin/redis-server *:6379(processInputBuffer+0xea)[0x560cec8d967a]
/opt/redis-stack/bin/redis-server *:6379(+0x1018ac)[0x560cec9728ac]
/opt/redis-stack/bin/redis-server *:6379(aeProcessEvents+0x2ca)[0x560cec8b9e6a]
/opt/redis-stack/bin/redis-server *:6379(aeMain+0x1d)[0x560cec8ba0fd]
/opt/redis-stack/bin/redis-server *:6379(main+0x33a)[0x560cec8b650a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7faab7fa2083]
/opt/redis-stack/bin/redis-server *:6379(_start+0x2e)[0x560cec8b6a0e]
------ INFO OUTPUT ------
# Server
redis_version:6.2.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:4dbc2487343b0024
redis_mode:standalone
os:Linux 5.4.119-19-0009.11 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:8
process_supervised:no
run_id:874b72d4cc4811fb370b07e4ed6c2dc3080afd35
tcp_port:6379
server_time_usec:1694102662954728
uptime_in_seconds:179
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:16381062
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:24
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:7127339936
used_memory_human:6.64G
used_memory_rss:7058096128
used_memory_rss_human:6.57G
used_memory_peak:7127339936
used_memory_peak_human:6.64G
used_memory_peak_perc:100.00%
used_memory_overhead:25248064
used_memory_startup:1014560
used_memory_dataset:7102091872
used_memory_dataset_perc:99.66%
allocator_allocated:7127313984
allocator_active:7749009408
allocator_resident:7816982528
total_system_memory:16512040960
total_system_memory_human:15.38G
used_memory_lua:30720
used_memory_lua_human:30.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.09
allocator_frag_bytes:621695424
allocator_rss_ratio:1.01
allocator_rss_bytes:67973120
rss_overhead_ratio:0.90
rss_overhead_bytes:-758886400
mem_fragmentation_ratio:0.99
mem_fragmentation_bytes:-69044232
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:41008
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1694102483
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:2
total_commands_processed:6
instantaneous_ops_per_sec:0
total_net_input_bytes:13114
total_net_output_bytes:144236
instantaneous_input_kbps:0.02
instantaneous_output_kbps:20.85
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:6
evicted_keys:0
keyspace_hits:1549796
keyspace_misses:178
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:5
total_writes_processed:4
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:4a43287fb0b0b760d5e51efd79701a5911ab3e19
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:3.116882
used_cpu_user:85.621699
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:3.044607
used_cpu_user_main_thread:85.613195
# Modules
module:name=graph,ver=21010,api=1,filters=0,usedby=[],using=[ReJSON],options=[]
module:name=search,ver=20609,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=ReJSON,ver=20407,api=1,filters=0,usedby=[search|graph],using=[],options=[handle-io-errors]
module:name=timeseries,ver=10810,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
module:name=bf,ver=20405,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_FT.SEARCH:calls=1,usec=269,usec_per_call=269.00,rejected_calls=0,failed_calls=0
cmdstat_get:calls=2,usec=61,usec_per_call=30.50,rejected_calls=0,failed_calls=0
cmdstat_info:calls=2,usec=19,usec_per_call=9.50,rejected_calls=0,failed_calls=0
cmdstat_mget:calls=1,usec=29,usec_per_call=29.00,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=418612,expires=91561,avg_ttl=1924955156
db1:keys=89,expires=86,avg_ttl=15621370
------ CLIENT LIST OUTPUT ------
id=52 addr=172.17.0.1:49340 laddr=172.17.0.2:6379 fd=12 name= age=3 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=12804 qbuf-free=28150 argv-mem=12662 obl=0 oll=0 omem=0 tot-mem=74294 events=r cmd=FT.SEARCH user=default redir=-1
id=53 addr=172.17.0.1:49352 laddr=172.17.0.2:6379 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
------ CURRENT CLIENT INFO ------
id=52 addr=172.17.0.1:49340 laddr=172.17.0.2:6379 fd=12 name= age=3 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=12804 qbuf-free=28150 argv-mem=12662 obl=0 oll=0 omem=0 tot-mem=74294 events=r cmd=FT.SEARCH user=default redir=-1
argv[0]: 'FT.SEARCH'
argv[1]: 'sku:idx'
argv[2]: '@n_comments:[817.8 inf] @category:(无线耳机) @brand:(大品牌) @price:[900.0 1000] @negative_ratio:[0 0.2797581622188262] (@combined:(音质好|适合女生)|(@pos_topics_vec:[VECTOR_RANGE $radius $query_vec_0] @pos_topics_vec:[VECTOR_RANGE $radius $query_vec_1]))'
argv[3]: 'SCORER'
argv[4]: 'BM25'
argv[5]: 'WITHSCORES'
argv[6]: 'LANGUAGE'
argv[7]: 'chinese'
argv[8]: 'DIALECT'
argv[9]: '2'
argv[10]: 'LIMIT'
argv[11]: '0'
argv[12]: '50'
argv[13]: 'params'
argv[14]: '6'
argv[15]: 'radius'
argv[16]: '0.2'
argv[17]: 'query_vec_0'
argv[18]: '}5W;`ف�|�E��7���e����<c�Q���D�B��nVټߕ�< ��<z ���U�ຓ���'
argv[19]: 'query_vec_1'
argv[20]: '�1��Ӌ���~�:�B��E�������t�S��A����;�����<]C;<�Ч;���ʡ���|�;�0 =5̛;n9��v#I�;|
;���<�E< ��D'��������<�dn��F��i�w�b�%=A/ ;oV��y��^t��� �;&���;ݷ�;ݷ�;(R����ֻ��!<�"�<)i�Y�< �Ѽ��u�qsW��C'
------ MODULES INFO OUTPUT ------
# graph_executing commands
# search_version
search_version:2.6.9
search_redis_version:6.2.12 - oss
# search_index
search_number_of_indexes:8
# search_fields_statistics
search_fields_text:Text=23
search_fields_numeric:Numeric=24
search_fields_tag:Tag=26
search_fields_vector:Vector=7,HSNW=7
# search_dialect_statistics
search_dialect_1:1
search_dialect_2:1
search_dialect_3:0
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:10000
search_max_aggregate_results:10000
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
# ReJSON_trace
ReJSON_trace: 0: redis_module::base_info_func
1: rejson::__info_func
2: modulesCollectInfo
at /__w/redis-stack/redis-stack/redis/src/module.c:7100:9
3: logModulesInfo
at /__w/redis-stack/redis-stack/redis/src/debug.c:1624:22
4: printCrashReport
at /__w/redis-stack/redis-stack/redis/src/debug.c:1896:5
5: _serverPanic
at /__w/redis-stack/redis-stack/redis/src/debug.c:1015:9
6: redisOutOfMemoryHandler
at /__w/redis-stack/redis-stack/redis/src/server.c:6037:5
7: zcalloc
at /__w/redis-stack/redis-stack/redis/src/zmalloc.c:180:15
8: RSIndexResult_IterateOffsets
9: RSIndexResult_IterateOffsets
10: RSIndexResult_IterateOffsets
11: IndexResult_MinOffsetDelta
12: BM25Scorer
13: rpscoreNext
14: rpsortNext_Accum
15: rploaderNext
16: sendChunk
17: AREQ_Execute
18: RSSearchCommand
19: RedisModuleCommandDispatcher
at /__w/redis-stack/redis-stack/redis/src/module.c:695:5
20: call
at /__w/redis-stack/redis-stack/redis/src/server.c:3750:5
21: processCommand
at /__w/redis-stack/redis-stack/redis/src/server.c:4297:9
22: processCommandAndResetClient
at /__w/redis-stack/redis-stack/redis/src/networking.c:2105:9
23: processInputBuffer
at /__w/redis-stack/redis-stack/redis/src/networking.c:2206:17
24: callHandler
at /__w/redis-stack/redis-stack/redis/src/connhelpers.h:79:18
connSocketEventHandler
at /__w/redis-stack/redis-stack/redis/src/connection.c:295:14
25: aeProcessEvents
at /__w/redis-stack/redis-stack/redis/src/ae.c:427:17
26: aeMain
at /__w/redis-stack/redis-stack/redis/src/ae.c:487:9
27: main
at /__w/redis-stack/redis-stack/redis/src/server.c:6474:5
28: __libc_start_main
29: _start
------ FAST MEMORY TEST ------
8:M 07 Sep 2023 16:04:23.060 # Bio thread for job type #0 terminated
8:M 07 Sep 2023 16:04:23.060 # Bio thread for job type #1 terminated
8:M 07 Sep 2023 16:04:23.060 # Bio thread for job type #2 terminated
*** Preparing to test memory region 560ceca8a000 (2281472 bytes)
*** Preparing to test memory region 560cecfb2000 (405504 bytes)
*** Preparing to test memory region 7fa8b0000000 (135168 bytes)
*** Preparing to test memory region 7fa8b8000000 (8388608 bytes)
*** Preparing to test memory region 7fa8b8800000 (7774142464 bytes)
*** Preparing to test memory region 7faa87e46000 (596115456 bytes)
*** Preparing to test memory region 7faaab6c7000 (8388608 bytes)
*** Preparing to test memory region 7faaabec8000 (8388608 bytes)
*** Preparing to test memory region 7faaac6c9000 (8388608 bytes)
*** Preparing to test memory region 7faaaceca000 (8388608 bytes)
*** Preparing to test memory region 7faaad986000 (8388608 bytes)
*** Preparing to test memory region 7faaae187000 (8388608 bytes)
*** Preparing to test memory region 7faaae988000 (8388608 bytes)
*** Preparing to test memory region 7faaaf189000 (8388608 bytes)
*** Preparing to test memory region 7faaaf98a000 (8388608 bytes)
*** Preparing to test memory region 7faab018b000 (8388608 bytes)
*** Preparing to test memory region 7faab098c000 (8388608 bytes)
*** Preparing to test memory region 7faab118d000 (8388608 bytes)
*** Preparing to test memory region 7faab198e000 (8388608 bytes)
*** Preparing to test memory region 7faab218f000 (8388608 bytes)
*** Preparing to test memory region 7faab2990000 (8388608 bytes)
*** Preparing to test memory region 7faab3191000 (8388608 bytes)
*** Preparing to test memory region 7faab3992000 (8388608 bytes)
*** Preparing to test memory region 7faab4193000 (8388608 bytes)
*** Preparing to test memory region 7faab60c3000 (12288 bytes)
*** Preparing to test memory region 7faab60c7000 (8388608 bytes)
*** Preparing to test memory region 7faab68c8000 (8388608 bytes)
*** Preparing to test memory region 7faab72a7000 (12288 bytes)
*** Preparing to test memory region 7faab75fd000 (12288 bytes)
*** Preparing to test memory region 7faab7600000 (8388608 bytes)
*** Preparing to test memory region 7faab7f1a000 (4096 bytes)
*** Preparing to test memory region 7faab7f78000 (24576 bytes)
*** Preparing to test memory region 7faab816c000 (16384 bytes)
*** Preparing to test memory region 7faab818f000 (16384 bytes)
*** Preparing to test memory region 7faab8466000 (16384 bytes)
*** Preparing to test memory region 7faab8652000 (8192 bytes)
*** Preparing to test memory region 7faab8685000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
-
OS distribution and version Linux VM-0-16-tencentos 5.4.119-19-0009.11 8 core，16G ram
-
Steps to reproduce (if any) depends a large dump file, can't share
Comment From: sundb
It seems that integer value overflow occurred and tried to allocate 18446744004990074880 bytes of memory. Please raise this issue with the developer at https://github.com/RediSearch/RediSearch/security.