Hello!
At my company, we recently decided to tackle our redis clusters obsolescence.
We upgraded them from redis 3 to latest redis 7, it is mainly used to store sessions for our fronts.
We also changed host os from ubuntu 16 to 24 and decided to move from a clurter archi where we had 2 redis nodes (one master and one replica) on a single VM to one node a VM.
We kept the same maxmemory (22GB) and maybe i missed some new default config parameters the redis versions above 3 brought but i think i mimicked the same config params between both redis clusters (3 and 7).
What I observe for the same workload (during a few days for the redis 7 cluster and a few months/years for the redis 3 cluster), the same number of cluster nodes and what i believed the same config parameters, is redis 3 cluster ram consumption is around 9GB when the redis 7 cluster only uses 1GB.
Ane everything works fine !
Did you guys made some fantastic ram consumption optimizations or am i missing something ?
Comment From: ginolegigot
Additional note: we did not move the old cluster data to the new one, it is brand new and the ram consumption behavior for both clusters is really steady: nearly constant ram consumption for both clusters with 9GB (redis 3 cluster) and 1GB (redis 7 cluster)
Comment From: sundb
@ginolegigot can you provide INFO ALL for both instances?
Comment From: ginolegigot
hey @sundb, thanks for your reply here the info all files: infoall_redis3.txt infoall_redis7.txt
Comment From: ginolegigot
I just realized downloading files from a random person might not be very secured i'll paste it here sorry ^^
Redis 3 cluster info all:
# Server
redis_version:3.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7785291a3d2152db
redis_mode:cluster
os:Linux 4.15.0-1098-gcp x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:5.4.0
process_id:1530
run_id:10291b728ba110d31c19d975f2276ff2c162ac26
tcp_port:6379
uptime_in_seconds:23039866
uptime_in_days:266
hz:10
lru_clock:5344380
config_file:/etc/redis/redis-server-6379.conf
# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:9725636272
used_memory_human:9.06G
used_memory_rss:10047160320
used_memory_peak:10204486848
used_memory_peak_human:9.50G
used_memory_lua:56320
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:208378403
rdb_bgsave_in_progress:0
rdb_last_save_time:1729613007
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:57
rdb_current_bgsave_time_sec:-1
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:39
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_current_size:7492298455
aof_base_size:5510158374
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:813842663
total_commands_processed:3651919525
instantaneous_ops_per_sec:1
total_net_input_bytes:1468117446318
total_net_output_bytes:1632825920296
instantaneous_input_kbps:0.06
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:3
sync_partial_ok:0
sync_partial_err:0
expired_keys:61514322
evicted_keys:0
keyspace_hits:578351045
keyspace_misses:211430359
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:119708
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:1
slave0:ip=xx.xxx.xxx.xxx,port=6380,state=online,offset=1397882139868,lag=1
master_repl_offset:1397882139868
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1397881091293
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:181581.83
used_cpu_user:177509.28
used_cpu_sys_children:1626.29
used_cpu_user_children:6618.23
# Commandstats
cmdstat_get:calls=358360704,usec=1005523342,usec_per_call=2.81
cmdstat_set:calls=34520,usec=676103,usec_per_call=19.59
cmdstat_setex:calls=808028266,usec=13351011160,usec_per_call=16.52
cmdstat_del:calls=320534895,usec=8413860071,usec_per_call=26.25
cmdstat_zadd:calls=19384,usec=60684,usec_per_call=3.13
cmdstat_zremrangebyscore:calls=823,usec=2274,usec_per_call=2.76
cmdstat_zcount:calls=200,usec=97,usec_per_call=0.49
cmdstat_zrevrange:calls=9692,usec=21967,usec_per_call=2.27
cmdstat_zscore:calls=19184,usec=70950,usec_per_call=3.70
cmdstat_select:calls=2,usec=2,usec_per_call=1.00
cmdstat_expire:calls=431290706,usec=1348800029,usec_per_call=3.13
cmdstat_pexpireat:calls=44212,usec=66026,usec_per_call=1.49
cmdstat_auth:calls=76813,usec=98733,usec_per_call=1.29
cmdstat_ping:calls=1923245,usec=149970,usec_per_call=0.08
cmdstat_echo:calls=631893415,usec=1755052510,usec_per_call=2.78
cmdstat_type:calls=9692,usec=27638,usec_per_call=2.85
cmdstat_psync:calls=3,usec=342416,usec_per_call=114138.66
cmdstat_replconf:calls=3776926,usec=4557244,usec_per_call=1.21
cmdstat_info:calls=1979567,usec=189694164,usec_per_call=95.83
cmdstat_role:calls=38,usec=167,usec_per_call=4.39
cmdstat_config:calls=1902711,usec=100771551,usec_per_call=52.96
cmdstat_cluster:calls=1086296672,usec=78467357555,usec_per_call=72.23
cmdstat_client:calls=6,usec=10,usec_per_call=1.67
cmdstat_eval:calls=9692,usec=635186,usec_per_call=65.54
cmdstat_slowlog:calls=3805406,usec=22247309,usec_per_call=5.85
cmdstat_latency:calls=1902751,usec=5219452,usec_per_call=2.74
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=22638767,expires=14,avg_ttl=1368088767691
Redis 7 cluster info all:
# Server
redis_version:7.4.1
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7fd3520e9c14a41b
redis_mode:cluster
os:Linux 6.8.0-1017-gcp x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:13.2.0
process_id:4477
process_supervised:systemd
run_id:0cf76394cfba7bca41ab004bcd28c88948330f8a
tcp_port:6379
server_time_usec:1733397612202487
uptime_in_seconds:247158
uptime_in_days:2
hz:10
configured_hz:10
lru_clock:5344364
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
listener2:name=tls,bind=xx.xxx.xxx.xxx,port=6379
# Clients
connected_clients:960
cluster_connections:10
maxclients:4064
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:29851808
used_memory_human:28.47M
used_memory_rss:107413504
used_memory_rss_human:102.44M
used_memory_peak:193059936
used_memory_peak_human:184.12M
used_memory_peak_perc:15.46%
used_memory_overhead:20361064
used_memory_startup:2004064
used_memory_dataset:9490744
used_memory_dataset_perc:34.08%
allocator_allocated:30816320
allocator_active:34045952
allocator_resident:38481920
allocator_muzzy:0
total_system_memory:33653579776
total_system_memory_human:31.34G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:192
used_memory_scripts:192
used_memory_scripts_human:192B
maxmemory:23622320128
maxmemory_human:22.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.10
allocator_frag_bytes:3153600
allocator_rss_ratio:1.13
allocator_rss_bytes:4435968
rss_overhead_ratio:2.79
rss_overhead_bytes:68931584
mem_fragmentation_ratio:3.58
mem_fragmentation_bytes:77392104
mem_not_counted_for_evict:34616
mem_replication_backlog:1048592
mem_total_replication_buffers:1086712
mem_clients_slaves:38136
mem_clients_normal:6217408
mem_cluster_links:10720
mem_aof_buffer:640
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4962
rdb_bgsave_in_progress:0
rdb_last_save_time:1733397562
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_saves:1500
rdb_last_cow_size:12648448
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:240
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:11767808
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:27576393
aof_base_size:8531058
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:24194741
total_commands_processed:87212263
instantaneous_ops_per_sec:670
total_net_input_bytes:20485241481
total_net_output_bytes:33583876966
total_net_repl_input_bytes:0
total_net_repl_output_bytes:13591895064
instantaneous_input_kbps:76.65
instantaneous_output_kbps:85.13
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:24.24
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_subkeys:0
expired_keys:3789791
expired_stale_perc:7.76
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:49937
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:10548388
keyspace_misses:13534650
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:3601
total_forks:1741
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:24132505
dump_payload_sanitizations:0
total_reads_processed:111931838
total_writes_processed:102752996
io_threaded_reads_processed:0
io_threaded_writes_processed:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:14075067
reply_buffer_expands:2
eventloop_cycles:167893294
eventloop_duration_sum:37409548976
eventloop_duration_cmd_sum:291132266
instantaneous_eventloop_cycles_per_sec:1205
instantaneous_eventloop_duration_usec:225
acl_access_denied_auth:0
acl_access_denied_cmd:24132501
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:1
slave0:ip=xxx.xxx.xxx.xxx,port=6379,state=online,offset=13591881495,lag=1
master_failover_state:no-failover
master_replid:50e271b68d62020979f9b99bdcf31d4583d3d27b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:13591894820
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:13590830617
repl_backlog_histlen:1064204
# CPU
used_cpu_sys:5197.556438
used_cpu_user:33602.534636
used_cpu_sys_children:55.595814
used_cpu_user_children:282.548767
used_cpu_sys_main_thread:5162.923291
used_cpu_user_main_thread:33591.260360
# Modules
# Commandstats
cmdstat_set:calls=0,usec=0,usec_per_call=0.00,rejected_calls=2,failed_calls=0
cmdstat_unlink:calls=3032,usec=38122,usec_per_call=12.57,rejected_calls=0,failed_calls=0
cmdstat_command|docs:calls=5,usec=5255,usec_per_call=1051.00,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=48327237,usec=129947514,usec_per_call=2.69,rejected_calls=0,failed_calls=0
cmdstat_echo:calls=0,usec=0,usec_per_call=0.00,rejected_calls=24132501,failed_calls=0
cmdstat_dbsize:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=18265,usec=3986569,usec_per_call=218.26,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=18258,usec=3439284,usec_per_call=188.37,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=18258,usec=36637,usec_per_call=2.01,rejected_calls=0,failed_calls=0
cmdstat_client|setinfo:calls=7978,usec=4520,usec_per_call=0.57,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=5344821,usec=21713851,usec_per_call=4.06,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=16,usec=868,usec_per_call=54.25,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=28656,usec=932620,usec_per_call=32.55,rejected_calls=0,failed_calls=0
cmdstat_cluster|set-config-epoch:calls=1,usec=56,usec_per_call=56.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=18259,usec=494680,usec_per_call=27.09,rejected_calls=0,failed_calls=0
cmdstat_cluster|addslots:calls=1,usec=175,usec_per_call=175.00,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=3989,usec=1772,usec_per_call=0.44,rejected_calls=1,failed_calls=0
cmdstat_replconf:calls=245053,usec=258894,usec_per_call=1.06,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=9022362,usec=56354404,usec_per_call=6.25,rejected_calls=0,failed_calls=0
cmdstat_get:calls=24083038,usec=68638377,usec_per_call=2.85,rejected_calls=1,failed_calls=0
cmdstat_latency|latest:calls=18258,usec=27032,usec_per_call=1.48,rejected_calls=0,failed_calls=0
cmdstat_latency|histogram:calls=18258,usec=5174885,usec_per_call=283.43,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=18258,usec=15592,usec_per_call=0.85,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=18258,usec=61026,usec_per_call=3.34,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=1,usec=132,usec_per_call=132.00,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_MOVED:count=3
errorstat_NOAUTH:count=1
errorstat_NOPERM:count=24132501
# Latencystats
latency_percentiles_usec_unlink:p50=6.015,p99=29.055,p99.9=1507.327
latency_percentiles_usec_command|docs:p50=1003.519,p99=1343.487,p99.9=1343.487
latency_percentiles_usec_auth:p50=2.007,p99=8.031,p99.9=28.031
latency_percentiles_usec_dbsize:p50=1.003,p99=1.003,p99.9=1.003
latency_percentiles_usec_info:p50=214.015,p99=315.391,p99.9=385.023
latency_percentiles_usec_config|get:p50=183.295,p99=270.335,p99.9=399.359
latency_percentiles_usec_client|setname:p50=2.007,p99=4.015,p99.9=26.111
latency_percentiles_usec_client|setinfo:p50=1.003,p99=2.007,p99.9=12.031
latency_percentiles_usec_expire:p50=4.015,p99=12.031,p99.9=33.023
latency_percentiles_usec_cluster|nodes:p50=50.175,p99=86.015,p99.9=86.015
latency_percentiles_usec_cluster|slots:p50=29.055,p99=71.167,p99.9=106.495
latency_percentiles_usec_cluster|set-config-epoch:p50=56.063,p99=56.063,p99.9=56.063
latency_percentiles_usec_cluster|info:p50=24.063,p99=58.111,p99.9=102.399
latency_percentiles_usec_cluster|addslots:p50=175.103,p99=175.103,p99.9=175.103
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=4.015
latency_percentiles_usec_replconf:p50=1.003,p99=2.007,p99.9=17.023
latency_percentiles_usec_setex:p50=6.015,p99=19.071,p99.9=44.031
latency_percentiles_usec_get:p50=2.007,p99=14.015,p99.9=38.143
latency_percentiles_usec_latency|latest:p50=1.003,p99=4.015,p99.9=20.095
latency_percentiles_usec_latency|histogram:p50=278.527,p99=442.367,p99.9=522.239
latency_percentiles_usec_slowlog|len:p50=1.003,p99=2.007,p99.9=12.031
latency_percentiles_usec_slowlog|get:p50=4.015,p99=10.047,p99.9=34.047
latency_percentiles_usec_psync:p50=132.095,p99=132.095,p99.9=132.095
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=29024,expires=29024,avg_ttl=590373,subexpiry=0
Comment From: ginolegigot
hope it's clearer now @sundb
Comment From: sundb
@ginolegigot the key number of 7.0 node is 29024, and it's memory usage is 28M not 1GB, did you give the wrong info?
Comment From: ginolegigot
Ok i was misinterpretating our ram consumption dashboard, indeed the redis cluster only consumes ~40MB. Thanks for pointing the key numbers it helped me greatly to understand. Have a good day!