Redis [BUG] Redis Stack cluster 7.2.5 replicas consume much more memory than the masters

Describe the bug Hello! We've installed new Redis stack cluster and switched traffic to that cluster. Docker image redis/redis-stack-server:7.2.0-v11 We noticed that redis replicas use much more memory than masters but the number of keys the same. On master:

used_memory_human:3.93G
db0:keys=8324096

On replica:

used_memory_human:16.96G
db0:keys=8324367

On old cluster everything was ok Docker image redis:7.0.15

Please help to find the cause of this behavior

Expected behavior

Memory usage should be the same on master and replica

Additional information

Master Info

# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2c467334f3e9b53d
redis_mode:cluster
os:Linux 6.8.0-35-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:8
process_supervised:no
run_id:14ec6725b75b3ffdd173ff1129f78be5cf97c407
tcp_port:7000
server_time_usec:1719391855949973
uptime_in_seconds:418199
uptime_in_days:4
hz:20
configured_hz:10
lru_clock:8115823
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=7000

# Clients
connected_clients:2635
cluster_connections:254
maxclients:20000
client_recent_max_input_buffer:131072
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:4216335504
used_memory_human:3.93G
used_memory_rss:4810436608
used_memory_rss_human:4.48G
used_memory_peak:5716106944
used_memory_peak_human:5.32G
used_memory_peak_perc:73.76%
used_memory_overhead:1106894952
used_memory_startup:2451608
used_memory_dataset:3109440552
used_memory_dataset_perc:73.79%
allocator_allocated:4217115208
allocator_active:4758700032
allocator_resident:4794286080
total_system_memory:810772246528
total_system_memory_human:755.09G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:21536119193
maxmemory_human:20.06G
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.13
allocator_frag_bytes:541584824
allocator_rss_ratio:1.01
allocator_rss_bytes:35586048
rss_overhead_ratio:1.00
rss_overhead_bytes:16150528
mem_fragmentation_ratio:1.14
mem_fragmentation_bytes:594001016
mem_not_counted_for_evict:0
mem_replication_backlog:134219224
mem_total_replication_buffers:134620368
mem_clients_slaves:402640
mem_clients_normal:35185864
mem_cluster_links:272288
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:21
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:56557029
rdb_bgsave_in_progress:0
rdb_last_save_time:1719349548
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:117
rdb_current_bgsave_time_sec:-1
rdb_saves:2
rdb_last_cow_size:1990275072
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:104363
total_commands_processed:453210114
instantaneous_ops_per_sec:5380
total_net_input_bytes:166123350426
total_net_output_bytes:257981435716
total_net_repl_input_bytes:0
total_net_repl_output_bytes:154466585171
instantaneous_input_kbps:2388.04
instantaneous_output_kbps:3679.06
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:2240.82
rejected_connections:0
sync_full:1
sync_partial_ok:1
sync_partial_err:1
expired_keys:93846616
expired_stale_perc:21.79
expired_time_cap_reached_count:11
expire_cycle_cpu_milliseconds:445725
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:105231137
keyspace_misses:170452182
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:175818
total_forks:3
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:9317255
active_defrag_misses:42066724
active_defrag_key_hits:3799233
active_defrag_key_misses:9321366
total_active_defrag_time:81115458
current_active_defrag_time:81115458
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:461001106
total_writes_processed:614492976
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2115628
reply_buffer_expands:3181110
eventloop_cycles:447771636
eventloop_duration_sum:10275716340
eventloop_duration_cmd_sum:1292557317
instantaneous_eventloop_cycles_per_sec:5350
instantaneous_eventloop_duration_usec:19
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:1
slave0:ip=10.61.198.52,port=8000,state=online,offset=154464753073,lag=0
master_failover_state:no-failover
master_replid:b0bd8ac59ad149b12ab62d6062d562974938794c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:154466567804
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:134217728
repl_backlog_first_byte_offset:154332335889
repl_backlog_histlen:134231916

# CPU
used_cpu_sys:9011.357187
used_cpu_user:3907.550499
used_cpu_sys_children:27.849434
used_cpu_user_children:54.939353
used_cpu_sys_main_thread:9010.300377
used_cpu_user_main_thread:3907.323148

# Modules
module:name=ReJSON,ver=20610,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=timeseries,ver=11012,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=redisgears_2,ver=20023,api=1,filters=0,usedby=[],using=[],options=[]
module:name=search,ver=20814,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=bf,ver=20612,api=1,filters=0,usedby=[],using=[],options=[]

# Errorstats

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=8324096,expires=8324096,avg_ttl=32755455

Replica info

# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2c467334f3e9b53d
redis_mode:cluster
os:Linux 6.8.0-35-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:8
process_supervised:no
run_id:3dad3647e32deb4cf648d544b1ff1ebfcc0544cb
tcp_port:8000
server_time_usec:1719391856465762
uptime_in_seconds:165078
uptime_in_days:1
hz:10
configured_hz:10
lru_clock:8115824
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=8000

# Clients
connected_clients:6
cluster_connections:254
maxclients:20000
client_recent_max_input_buffer:163840
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:18207236904
used_memory_human:16.96G
used_memory_rss:19306369024
used_memory_rss_human:17.98G
used_memory_peak:18207327304
used_memory_peak_human:16.96G
used_memory_peak_perc:100.00%
used_memory_overhead:1072148528
used_memory_startup:2806592
used_memory_dataset:17135088376
used_memory_dataset_perc:94.13%
allocator_allocated:18207893296
allocator_active:19165020160
allocator_resident:19291045888
total_system_memory:810772246528
total_system_memory_human:755.09G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:21536119193
maxmemory_human:20.06G
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.05
allocator_frag_bytes:957126864
allocator_rss_ratio:1.01
allocator_rss_bytes:126025728
rss_overhead_ratio:1.00
rss_overhead_bytes:15323136
mem_fragmentation_ratio:1.06
mem_fragmentation_bytes:1099090552
mem_not_counted_for_evict:0
mem_replication_backlog:134642240
mem_total_replication_buffers:134640728
mem_clients_slaves:0
mem_clients_normal:42400
mem_cluster_links:272288
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:7
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:92860890
rdb_bgsave_in_progress:0
rdb_last_save_time:1719349502
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:71
rdb_current_bgsave_time_sec:-1
rdb_saves:2
rdb_last_cow_size:1687769088
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3252
total_commands_processed:259959776
instantaneous_ops_per_sec:2960
total_net_input_bytes:154468574324
total_net_output_bytes:63494702
total_net_repl_input_bytes:154467861428
total_net_repl_output_bytes:0
instantaneous_input_kbps:2478.55
instantaneous_output_kbps:0.06
instantaneous_input_repl_kbps:2478.55
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:348598
total_forks:2
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:2078870
active_defrag_misses:19779844
active_defrag_key_hits:292225
active_defrag_key_misses:4450467
total_active_defrag_time:81107369
current_active_defrag_time:81107369
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:166043155
total_writes_processed:186927
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:495
reply_buffer_expands:0
eventloop_cycles:176987796
eventloop_duration_sum:2756812111
eventloop_duration_cmd_sum:1026908666
instantaneous_eventloop_cycles_per_sec:1983
instantaneous_eventloop_duration_usec:13
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:slave
master_host:10.61.206.12
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:154467861442
slave_repl_offset:154467861442
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:b0bd8ac59ad149b12ab62d6062d562974938794c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:154467861442
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:134217728
repl_backlog_first_byte_offset:154333595671
repl_backlog_histlen:134265772

# CPU
used_cpu_sys:1681.486374
used_cpu_user:2045.363954
used_cpu_sys_children:8.004717
used_cpu_user_children:57.442918
used_cpu_sys_main_thread:1680.803070
used_cpu_user_main_thread:2045.149639

# Modules
module:name=ReJSON,ver=20610,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=timeseries,ver=11012,api=1,filters=0,usedby=[],using=[],options=[]
module:name=search,ver=20814,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=bf,ver=20612,api=1,filters=0,usedby=[],using=[],options=[]
module:name=redisgears_2,ver=20023,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]

# Errorstats

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=8324367,expires=8324367,avg_ttl=0

Comment From: artful88533

Replica restarting temporarily fixes the issue, but then memory usage on the replica becomes higher than on the master

Comment From: sundb

@artful88533 can you give the infos of master and replication now? btw, please use info all instead of info.

Comment From: artful88533

@sundb info all

Master

# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2c467334f3e9b53d
redis_mode:cluster
os:Linux 6.8.0-35-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:8
process_supervised:no
run_id:14ec6725b75b3ffdd173ff1129f78be5cf97c407
tcp_port:7000
server_time_usec:1719411182497502
uptime_in_seconds:437526
uptime_in_days:5
hz:20
configured_hz:10
lru_clock:8135150
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=7000

# Clients
connected_clients:2739
cluster_connections:254
maxclients:20000
client_recent_max_input_buffer:114688
client_recent_max_output_buffer:49176
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:5052370408
used_memory_human:4.71G
used_memory_rss:5266243584
used_memory_rss_human:4.90G
used_memory_peak:5716106944
used_memory_peak_human:5.32G
used_memory_peak_perc:88.39%
used_memory_overhead:1207524492
used_memory_startup:2451608
used_memory_dataset:3844845916
used_memory_dataset_perc:76.14%
allocator_allocated:5053043632
allocator_active:5216206848
allocator_resident:5252055040
total_system_memory:810772246528
total_system_memory_human:755.09G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:21536119193
maxmemory_human:20.06G
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.03
allocator_frag_bytes:163163216
allocator_rss_ratio:1.01
allocator_rss_bytes:35848192
rss_overhead_ratio:1.00
rss_overhead_bytes:14188544
mem_fragmentation_ratio:1.04
mem_fragmentation_bytes:213611632
mem_not_counted_for_evict:0
mem_replication_backlog:134219244
mem_total_replication_buffers:134650600
mem_clients_slaves:432872
mem_clients_normal:37507552
mem_cluster_links:272288
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:41234717
rdb_bgsave_in_progress:0
rdb_last_save_time:1719392816
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:71
rdb_current_bgsave_time_sec:-1
rdb_saves:3
rdb_last_cow_size:1409675264
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:137545
total_commands_processed:573773169
instantaneous_ops_per_sec:6588
total_net_input_bytes:212281248854
total_net_output_bytes:332599065716
total_net_repl_input_bytes:0
total_net_repl_output_bytes:199765961528
instantaneous_input_kbps:2383.74
instantaneous_output_kbps:3574.87
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:2239.49
rejected_connections:0
sync_full:2
sync_partial_ok:1
sync_partial_err:2
expired_keys:119159842
expired_stale_perc:25.42
expired_time_cap_reached_count:12
expire_cycle_cpu_milliseconds:563633
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:133493539
keyspace_misses:216862575
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:119115
total_forks:5
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:9336814
active_defrag_misses:44453486
active_defrag_key_hits:3799233
active_defrag_key_misses:9321366
total_active_defrag_time:94672043
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:583996420
total_writes_processed:777153812
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:2669507
reply_buffer_expands:4037937
eventloop_cycles:565993588
eventloop_duration_sum:12680422878
eventloop_duration_cmd_sum:1628556073
instantaneous_eventloop_cycles_per_sec:6406
instantaneous_eventloop_duration_usec:20
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:1
slave0:ip=10.61.198.52,port=8000,state=online,offset=197442264394,lag=1
master_failover_state:no-failover
master_replid:b0bd8ac59ad149b12ab62d6062d562974938794c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:197443690201
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:134217728
repl_backlog_first_byte_offset:197309422657
repl_backlog_histlen:134267545

# CPU
used_cpu_sys:11137.077937
used_cpu_user:4828.276041
used_cpu_sys_children:56.291261
used_cpu_user_children:136.931994
used_cpu_sys_main_thread:11135.819402
used_cpu_user_main_thread:4828.019535

# Modules
module:name=ReJSON,ver=20610,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=timeseries,ver=11012,api=1,filters=0,usedby=[],using=[],options=[]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=redisgears_2,ver=20023,api=1,filters=0,usedby=[],using=[],options=[]
module:name=search,ver=20814,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]
module:name=bf,ver=20612,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_replconf:calls=195637,usec=258620,usec_per_call=1.32,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=3261,usec=3046,usec_per_call=0.93,rejected_calls=0,failed_calls=0
cmdstat_latency|histogram:calls=3261,usec=1157782,usec_per_call=355.04,rejected_calls=0,failed_calls=0
cmdstat_command|docs:calls=1,usec=683,usec_per_call=683.00,rejected_calls=0,failed_calls=0
cmdstat_function|dump:calls=48,usec=84,usec_per_call=1.75,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=108105725,usec=511996068,usec_per_call=4.74,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=128,usec=29,usec_per_call=0.23,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=3299,usec=6909,usec_per_call=2.09,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=6455094,usec=15080794,usec_per_call=2.34,rejected_calls=0,failed_calls=0
cmdstat_del:calls=4464148,usec=18065004,usec_per_call=4.05,rejected_calls=0,failed_calls=0
cmdstat_quit:calls=3137,usec=1489,usec_per_call=0.47,rejected_calls=0,failed_calls=0
cmdstat_get:calls=350356114,usec=607204482,usec_per_call=1.73,rejected_calls=1,failed_calls=0
cmdstat_info:calls=3425,usec=848013,usec_per_call=247.60,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=3261,usec=1957,usec_per_call=0.60,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=3261,usec=1411,usec_per_call=0.43,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=3262,usec=113298,usec_per_call=34.73,rejected_calls=0,failed_calls=0
cmdstat_cluster|myid:calls=25,usec=19,usec_per_call=0.76,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=3046,usec=194229,usec_per_call=63.77,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=347,usec=29109,usec_per_call=83.89,rejected_calls=0,failed_calls=0
cmdstat_cluster|setslot:calls=13056,usec=30047,usec_per_call=2.30,rejected_calls=0,failed_calls=0
cmdstat_cluster|addslots:calls=1,usec=35,usec_per_call=35.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|getkeysinslot:calls=768,usec=769,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|set-config-epoch:calls=1,usec=12,usec_per_call=12.00,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=943,usec=9625,usec_per_call=10.21,rejected_calls=0,failed_calls=0
cmdstat_set:calls=104148646,usec=472837000,usec_per_call=4.54,rejected_calls=0,failed_calls=0
cmdstat_dbsize:calls=2,usec=2,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=3,usec=75,usec_per_call=25.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=3266,usec=715510,usec_per_call=219.08,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=3,usec=9,usec_per_call=3.00,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_ERR:count=1

# Latencystats
latency_percentiles_usec_replconf:p50=1.003,p99=7.007,p99.9=51.199
latency_percentiles_usec_latency|latest:p50=1.003,p99=6.015,p99.9=10.047
latency_percentiles_usec_latency|histogram:p50=387.071,p99=522.239,p99.9=598.015
latency_percentiles_usec_command|docs:p50=684.031,p99=684.031,p99.9=684.031
latency_percentiles_usec_function|dump:p50=2.007,p99=2.007,p99.9=2.007
latency_percentiles_usec_setex:p50=4.015,p99=17.023,p99.9=28.031
latency_percentiles_usec_client|setinfo:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_client|setname:p50=2.007,p99=8.031,p99.9=21.119
latency_percentiles_usec_expire:p50=2.007,p99=12.031,p99.9=22.015
latency_percentiles_usec_del:p50=3.007,p99=16.063,p99.9=28.031
latency_percentiles_usec_quit:p50=0.001,p99=5.023,p99.9=14.015
latency_percentiles_usec_get:p50=1.003,p99=11.007,p99.9=20.095
latency_percentiles_usec_info:p50=246.783,p99=350.207,p99.9=436.223
latency_percentiles_usec_slowlog|len:p50=1.003,p99=3.007,p99.9=14.015
latency_percentiles_usec_slowlog|get:p50=0.001,p99=1.003,p99.9=9.023
latency_percentiles_usec_cluster|info:p50=32.127,p99=60.159,p99.9=80.383
latency_percentiles_usec_cluster|myid:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_cluster|slots:p50=62.207,p99=100.351,p99.9=120.319
latency_percentiles_usec_cluster|nodes:p50=75.263,p99=197.631,p99.9=207.871
latency_percentiles_usec_cluster|setslot:p50=2.007,p99=8.031,p99.9=18.047
latency_percentiles_usec_cluster|addslots:p50=35.071,p99=35.071,p99.9=35.071
latency_percentiles_usec_cluster|getkeysinslot:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_cluster|set-config-epoch:p50=12.031,p99=12.031,p99.9=12.031
latency_percentiles_usec_hello:p50=8.031,p99=27.007,p99.9=38.143
latency_percentiles_usec_set:p50=4.015,p99=18.047,p99.9=36.095
latency_percentiles_usec_dbsize:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_psync:p50=26.111,p99=29.055,p99.9=29.055
latency_percentiles_usec_config|get:p50=228.351,p99=333.823,p99.9=401.407
latency_percentiles_usec_ping:p50=1.003,p99=8.031,p99.9=8.031

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=9552566,expires=9552566,avg_ttl=21142371

Replica

# Server
redis_version:7.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2c467334f3e9b53d
redis_mode:cluster
os:Linux 6.8.0-35-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.4.0
process_id:8
process_supervised:no
run_id:7eea49cc4947ad3fe81cb92da92dc91405cbb4b2
tcp_port:8000
server_time_usec:1719411185334252
uptime_in_seconds:13384
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:8135153
executable:/opt/redis-stack/bin/redis-server
config_file:
io_threads_active:0
listener0:name=tcp,bind=0.0.0.0,port=8000

# Clients
connected_clients:3
cluster_connections:254
maxclients:25000
client_recent_max_input_buffer:327680
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0

# Memory
used_memory:7736771296
used_memory_human:7.21G
used_memory_rss:7869845504
used_memory_rss_human:7.33G
used_memory_peak:7737210440
used_memory_peak_human:7.21G
used_memory_peak_perc:99.99%
used_memory_overhead:1170844472
used_memory_startup:3199776
used_memory_dataset:6565926824
used_memory_dataset_perc:84.90%
allocator_allocated:7737636208
allocator_active:7807827968
allocator_resident:7859470336
total_system_memory:810772246528
total_system_memory_human:755.09G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:21536119193
maxmemory_human:20.06G
maxmemory_policy:volatile-ttl
allocator_frag_ratio:1.01
allocator_frag_bytes:70191760
allocator_rss_ratio:1.01
allocator_rss_bytes:51642368
rss_overhead_ratio:1.00
rss_overhead_bytes:10375168
mem_fragmentation_ratio:1.02
mem_fragmentation_bytes:133165136
mem_not_counted_for_evict:0
mem_replication_backlog:134614144
mem_total_replication_buffers:134612632
mem_clients_slaves:0
mem_clients_normal:118536
mem_cluster_links:272288
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:49131156
rdb_bgsave_in_progress:0
rdb_last_save_time:1719397801
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:8871242
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:365
total_commands_processed:49135853
instantaneous_ops_per_sec:3482
total_net_input_bytes:32714093799
total_net_output_bytes:6648884
total_net_repl_input_bytes:32714032440
total_net_repl_output_bytes:0
instantaneous_input_kbps:2536.52
instantaneous_output_kbps:0.06
instantaneous_input_repl_kbps:2536.52
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:34532516
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:30826526
total_writes_processed:15153
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:141
reply_buffer_expands:0
eventloop_cycles:31900394
eventloop_duration_sum:439098958
eventloop_duration_cmd_sum:177461812
instantaneous_eventloop_cycles_per_sec:2354
instantaneous_eventloop_duration_usec:11
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:slave
master_host:10.61.206.12
master_port:7000
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_read_repl_offset:197450677970
slave_repl_offset:197450677970
slave_priority:100
slave_read_only:1
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:b0bd8ac59ad149b12ab62d6062d562974938794c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:197450677970
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:134217728
repl_backlog_first_byte_offset:197316444908
repl_backlog_histlen:134233063

# CPU
used_cpu_sys:258.798647
used_cpu_user:372.687360
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:257.721549
used_cpu_user_main_thread:372.727912

# Modules
module:name=timeseries,ver=11012,api=1,filters=0,usedby=[],using=[],options=[]
module:name=ReJSON,ver=20610,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=RedisCompat,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
module:name=bf,ver=20612,api=1,filters=0,usedby=[],using=[],options=[]
module:name=redisgears_2,ver=20023,api=1,filters=0,usedby=[],using=[],options=[]
module:name=search,ver=20814,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]

# Commandstats
cmdstat_pexpireat:calls=1037954,usec=2759005,usec_per_call=2.66,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=223,usec=194,usec_per_call=0.87,rejected_calls=0,failed_calls=0
cmdstat_latency|histogram:calls=223,usec=48833,usec_per_call=218.98,rejected_calls=0,failed_calls=0
cmdstat_select:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=1318,usec=240,usec_per_call=0.18,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=708,usec=5908,usec_per_call=8.34,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=12,usec=7,usec_per_call=0.58,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=223,usec=367,usec_per_call=1.65,rejected_calls=0,failed_calls=0
cmdstat_info:calls=229,usec=40011,usec_per_call=174.72,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=224,usec=46177,usec_per_call=206.15,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=708,usec=89,usec_per_call=0.13,rejected_calls=0,failed_calls=0
cmdstat_del:calls=18960123,usec=62272407,usec_per_call=3.28,rejected_calls=0,failed_calls=0
cmdstat_set:calls=29133079,usec=112278645,usec_per_call=3.85,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=7,usec=72,usec_per_call=10.29,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=222,usec=6392,usec_per_call=28.79,rejected_calls=1,failed_calls=0
cmdstat_cluster|myid:calls=7,usec=4,usec_per_call=0.57,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=141,usec=7577,usec_per_call=53.74,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=223,usec=847,usec_per_call=3.80,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=223,usec=167,usec_per_call=0.75,rejected_calls=0,failed_calls=0
cmdstat_quit:calls=5,usec=4,usec_per_call=0.80,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_LOADING:count=1

# Latencystats
latency_percentiles_usec_pexpireat:p50=3.007,p99=5.023,p99.9=8.031
latency_percentiles_usec_latency|latest:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_latency|histogram:p50=221.183,p99=268.287,p99.9=270.335
latency_percentiles_usec_select:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=5.023
latency_percentiles_usec_exec:p50=8.031,p99=16.063,p99.9=22.015
latency_percentiles_usec_client|setinfo:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_client|setname:p50=2.007,p99=2.007,p99.9=4.015
latency_percentiles_usec_info:p50=177.151,p99=214.015,p99.9=217.087
latency_percentiles_usec_config|get:p50=208.895,p99=272.383,p99.9=296.959
latency_percentiles_usec_multi:p50=0.001,p99=1.003,p99.9=1.003
latency_percentiles_usec_del:p50=3.007,p99=8.031,p99.9=18.047
latency_percentiles_usec_set:p50=4.015,p99=10.047,p99.9=37.119
latency_percentiles_usec_hello:p50=10.047,p99=15.039,p99.9=15.039
latency_percentiles_usec_cluster|info:p50=27.007,p99=41.215,p99.9=44.031
latency_percentiles_usec_cluster|myid:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_cluster|slots:p50=53.247,p99=77.311,p99.9=80.383
latency_percentiles_usec_slowlog|get:p50=4.015,p99=5.023,p99.9=5.023
latency_percentiles_usec_slowlog|len:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_quit:p50=1.003,p99=1.003,p99.9=1.003

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=9552551,expires=9552551,avg_ttl=0

Comment From: sundb

@artful88533 the main memory footprint is on the dataset, but I don't know why, what data tyes do you mainly use, include module data type?

Comment From: artful88533

@sundb we use key-value data and update the TTL after consuming keys. We haven't used modules yet.

For now, we've reinstalled the cluster to redis oss 7.0.15

Comment From: aeber

We had the same behavior when switching from a Redis 7.0.10 sentinel setup to Redis Stack 7.2.0.

In https://github.com/redis-stack/redis-stack/issues/527 @druidai-devops noticed that removing the redisgears module solved their memory issue and this also worked for us.

Comment From: sundb

@aeber thanks for figuring out, i see that RedisGear is no longer in the lastest RedisStack(7.4), @artful8853 can you try the newly one?

Comment From: artful88533

Unfortunately, I cannot reinstall the current Redis cluster to Redis Stack at this time. Maybe I'll try later.