Describe the bug
The used_memory is larger than the true value and the server's capacity. This prevents maxmemory configurations from properly functioning.
To reproduce
Not sure unfortunately, seems to happen randomly.
Expected behavior
Value to be accurate
Additional information
I have 8 instances of version 7.2 (compiled from source) running on the same Linux server, each with their own port. They all store hashes and have RediSearch run on each of them with the same index specifications. The server has 32G of memory and 1G swap. Most instances have the right memory usage (~2.93G), but some do not. The high used_memory is causing errors when a maxmemory is set.
The used_memory balloons higher relatively quickly (within a few hours of normal operation) after restarting the instances, but also randomly. Given the data we have and the usage of the instances, only a few MB of data is added daily to each server and no old hashes are deleted/updated.
There are no snapshots and automatic aof rewrites are turned off. Rewrites are triggered manually and were done so when the data was fully loaded in the first time. Currently, aof incremental files are a few MB in size, for each instance.
for p in $(seq 6379 6376); do
redis-cli -p $p info memory | grep used_memory_human
done
used_memory_human:2.93G
used_memory_human:2.93G
used_memory_human:74.93G
used_memory_human:2.93G
used_memory_human:18.94G
used_memory_human:2.93G
used_memory_human:2.94G
used_memory_human:10.94G
for p in $(seq 6379 6386); do
redis-cli -p $p info memory | grep "used_memory:"
done
used_memory:3148271088
used_memory:3150117232
used_memory:80457651296
used_memory:3147575024
used_memory:20332828416
used_memory:3145739552
used_memory:3155594680
used_memory:11741557472
redis-cli -p 6381 info memory
# Memory
used_memory:80457650272
used_memory_human:74.93G
used_memory_rss:3250946048
used_memory_rss_human:3.03G
used_memory_peak:80457714544
used_memory_peak_human:74.93G
used_memory_peak_perc:100.00%
used_memory_overhead:65090928
used_memory_startup:1037984
used_memory_dataset:80392559344
used_memory_dataset_perc:99.92%
allocator_allocated:80457852024
allocator_active:80492441600
allocator_resident:80591007744
total_system_memory:33028308992
total_system_memory_human:30.76G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:3670016000
maxmemory_human:3.42G
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:34589576
allocator_rss_ratio:1.00
allocator_rss_bytes:98566144
rss_overhead_ratio:0.04
rss_overhead_bytes:-77340061696
mem_fragmentation_ratio:0.04
mem_fragmentation_bytes:-77206665816
mem_not_counted_for_evict:8
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:50584
mem_cluster_links:0
mem_aof_buffer:8
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
ps -o rss,%mem,command ax | grep redis-server
3168452 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6379
3170876 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6380
3175704 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6382
3174868 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6384
3182932 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6385
3172908 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6386
3174752 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6381
3179896 9.8 /home/ubuntu/redis-7.2.0/src/redis-server *:6383
Comment From: sundb
Could you give the INFO ALL output of instance with abnormal memory usage?
Comment From: fast-facts
I had to restart the instances and move this to a dev server for testing. I recreated the problem on another server.
I also discovered that this has to do with RediSearch's ft.aggregate, specifically the below command (note the large MAX argument). It does seem to me that we have a problem on our side with improper arguments that I'll address, but shouldn't this cause consistent behavior amongst all the instances, and not just some of them increasing in memory? The same command is run on all instances concurrently. Is this a better question for the RediSearch github?
"FT.AGGREGATE" "idx" "@name:sales*" "LOAD" "1" "@__key" "SORTBY" "2" "@dr" "DESC" "MAX" "1000000000" "WITHCURSOR" "COUNT" "10"
redis-cli -p 6385 info all
# Server
redis_version:7.2.0
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:4e788b67d7426d9b
redis_mode:standalone
os:Linux 6.2.0-1009-aws aarch64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:602382
process_supervised:no
run_id:de444a2bec6268a78ad372b2412acd64110d0f56
tcp_port:6385
server_time_usec:1693404347059772
uptime_in_seconds:51280
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:15682747
executable:/home/ubuntu/redis-7.2.0/src/redis-server
config_file:/home/ubuntu/redis-7.2.0/redis.6385.conf
io_threads_active:0
listener0:name=tcp,bind=*,bind=-::*,port=6385
# Clients
connected_clients:6
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:63284419952
used_memory_human:58.94G
used_memory_rss:3262771200
used_memory_rss_human:3.04G
used_memory_peak:63284432536
used_memory_peak_human:58.94G
used_memory_peak_perc:100.00%
used_memory_overhead:65132216
used_memory_startup:1037984
used_memory_dataset:63219287736
used_memory_dataset_perc:99.90%
allocator_allocated:63284867160
allocator_active:63324950528
allocator_resident:63417421824
total_system_memory:33028308992
total_system_memory_human:30.76G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:3670016000
maxmemory_human:3.42G
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:40083368
allocator_rss_ratio:1.00
allocator_rss_bytes:92471296
rss_overhead_ratio:0.05
rss_overhead_bytes:-60154650624
mem_fragmentation_ratio:0.05
mem_fragmentation_bytes:-60021608584
mem_not_counted_for_evict:8
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:57752
mem_cluster_links:0
mem_aof_buffer:8
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:1586730
rdb_bgsave_in_progress:0
rdb_last_save_time:1693353067
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:1180682
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:545760562
aof_base_size:461764482
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:24126
total_commands_processed:472872
instantaneous_ops_per_sec:9
total_net_input_bytes:86228532
total_net_output_bytes:400113401
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.15
instantaneous_output_kbps:10.12
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:328
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:206407
current_eviction_exceeded_time:206407
keyspace_hits:2488157
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:58938
total_forks:49
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1829
dump_payload_sanitizations:0
total_reads_processed:399761
total_writes_processed:374395
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:9372
reply_buffer_expands:27685
eventloop_cycles:934292
eventloop_duration_sum:75246598
eventloop_duration_cmd_sum:26449189
instantaneous_eventloop_cycles_per_sec:19
instantaneous_eventloop_duration_usec:68
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:d2c7c76339e739a4f9c8d49e25e04edbc7107868
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:122807
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:52.920681
used_cpu_user:136.129445
used_cpu_sys_children:39.389500
used_cpu_user_children:559.916304
used_cpu_sys_main_thread:28.435854
used_cpu_user_main_thread:126.717824
# Modules
module:name=search,ver=20804,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_FT.INFO:calls=205,usec=19337,usec_per_call=94.33,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=3,usec_per_call=3.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=75358,usec=7469391,usec_per_call=99.12,rejected_calls=0,failed_calls=0
cmdstat_FT.AGGREGATE:calls=19,usec=5052,usec_per_call=265.89,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=122397,usec=18588519,usec_per_call=151.87,rejected_calls=0,failed_calls=0
cmdstat_FT.SEARCH:calls=3,usec=46882,usec_per_call=15627.33,rejected_calls=0,failed_calls=0
cmdstat_monitor:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=256035,usec=75497,usec_per_call=0.29,rejected_calls=200,failed_calls=0
cmdstat_multi:calls=205,usec=395,usec_per_call=1.93,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=14046,usec=9458,usec_per_call=0.67,rejected_calls=1629,failed_calls=0
cmdstat_exec:calls=205,usec=18823195,usec_per_call=91820.47,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_LOADING:count=1829
# Latencystats
latency_percentiles_usec_FT.INFO:p50=87.039,p99=198.655,p99.9=249.855
latency_percentiles_usec_config|get:p50=3.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_info:p50=132.095,p99=161.791,p99.9=413.695
latency_percentiles_usec_FT.AGGREGATE:p50=249.855,p99=524.287,p99.9=524.287
latency_percentiles_usec_hset:p50=97.279,p99=720.895,p99.9=2392.063
latency_percentiles_usec_FT.SEARCH:p50=15335.423,p99=31195.135,p99.9=31195.135
latency_percentiles_usec_monitor:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_multi:p50=1.003,p99=14.015,p99.9=17.023
latency_percentiles_usec_dbsize:p50=1.003,p99=3.007,p99.9=12.031
latency_percentiles_usec_exec:p50=88080.383,p99=217055.231,p99.9=249561.087
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=1181476,expires=0,avg_ttl=0
Comment From: madolson
Yes, you should open the issue here then: https://github.com/RediSearch/RediSearch