Describe the bug
In replica mode, with read-only set to no, Redis does not honour the maxmemory parameter
To reproduce
Setup Redis as a Replica, read-only no. Add data to Redis until it gets killed by the OOM. I added a dictionary provided by wbritish on Ubuntu. In the following setup, I have maxmemory set to 100MB on a system with 2GB of RAM. Eviction Policy set to allkeys-lfu.
sudo apt install wbritish
while :
do
key=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c10)
redis-cli -x set $key < /usr/share/dict/british-english
done
You can check the used_memory_rss vs the maxmemory parameters with redis-cli info memory | grep -e used_memory_rss_human -e maxmemory_human
$ redis-cli info memory | grep -e used_memory_rss_human -e maxmemory_human
used_memory_rss_human:1.74G
maxmemory_human:100.00M
user@redis-host:~$ redis-cli info memory | grep -e used_memory_rss_human -e maxmemory_human
<Redis killed by OOM>
Error: Connection reset by peer
user@redis-host:~$
user@redis-host:~$ redis-cli info memory | grep -e used_memory_rss_human -e maxmemory_human
used_memory_rss_human:73.84M
maxmemory_human:100.00M
Please note - My use case for Redis is as a cache for Ruby on Rails - I don't normally use it to store dictionaries.
Expected behavior
This Redis instance is using allkeys-lfu, as such I think Redis should expire keys and as a last resort, reject writes.
Additional information
This instance is set as a replica of a Redis master and has replica-read-only set to no.
$ redis-cli config get replica-read-only
1) "replica-read-only"
2) "no"
Swap is enabled at the same size as RAM
$ free -m
total used free shared buff/cache available
Mem: 1975 1377 425 0 173 457
Swap: 1976 120 1856
VM Overcommit is enabled:
$ sysctl vm.overcommit_memory
vm.overcommit_memory = 1
Hugepages are disabled:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
Eviction Policy is set to LFU
~$ redis-cli config get maxmemory-policy
1) "maxmemory-policy"
2) "allkeys-lfu"
Max Memory is set to 100MB
$ redis-cli config get maxmemory
1) "maxmemory"
2) "104857600"
System memory is 2GB:
$ cat /proc/meminfo | head -n 1
MemTotal: 2023376 kB
Redis version is 7.0.9
$ redis-server --version
Redis server v=7.0.9 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=448f00a942bc1f77
OS is Ubuntu 22.04.02 LTS on an Intel based VPS.
Comment From: enjoy-binbin
i guess the reason is allocator_frag_bytes, can you try to post the whole INFO result?
Comment From: dkam
Hi @enjoy-binbin - sure!
redis01:~$ redis-cli info
# Server
redis_version:7.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:448f00a942bc1f77
redis_mode:standalone
os:Linux 5.15.0-67-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.3.0
process_id:823561
process_supervised:systemd
run_id:58fcb82c7cb7cca82454663ebe920f66e2a02b5c
tcp_port:6379
server_time_usec:1678594008682019
uptime_in_seconds:86
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:872408
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:24
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:2469512608
used_memory_human:2.30G
used_memory_rss:1848303616
used_memory_rss_human:1.72G
used_memory_peak:2470520848
used_memory_peak_human:2.30G
used_memory_peak_perc:99.96%
used_memory_overhead:1011948
used_memory_startup:862440
used_memory_dataset:2468500660
used_memory_dataset_perc:99.99%
allocator_allocated:2469779344
allocator_active:2479742976
allocator_resident:2486583296
total_system_memory:2071937024
total_system_memory_human:1.93G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:943718400
maxmemory_human:900.00M
maxmemory_policy:allkeys-lfu
allocator_frag_ratio:1.00
allocator_frag_bytes:9963632
allocator_rss_ratio:1.00
allocator_rss_bytes:6840320
rss_overhead_ratio:0.74
rss_overhead_bytes:-638279680
mem_fragmentation_ratio:0.75
mem_fragmentation_bytes:-621169064
mem_not_counted_for_evict:0
mem_replication_backlog:20508
mem_total_replication_buffers:20504
mem_clients_slaves:0
mem_clients_normal:1816
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:305
rdb_bgsave_in_progress:0
rdb_last_save_time:1678593922
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:2050
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:514
total_commands_processed:1585
instantaneous_ops_per_sec:0
total_net_input_bytes:497431094
total_net_output_bytes:24149
total_net_repl_input_bytes:17864
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.02
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:204
dump_payload_sanitizations:0
total_reads_processed:7142
total_writes_processed:591
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:1
reply_buffer_expands:0
# Replication
role:slave
master_host:100.75.219.35
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_read_repl_offset:18140
slave_repl_offset:18140
slave_priority:100
slave_read_only:0
replica_announced:1
connected_slaves:0
master_failover_state:no-failover
master_replid:faa424333eeef08293687bf46e048ad4f13add98
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:18140
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:277
repl_backlog_histlen:17864
# CPU
used_cpu_sys:1.127987
used_cpu_user:4.931278
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:1.127144
used_cpu_user_main_thread:4.930548
# Modules
# Errorstats
errorstat_LOADING:count=204
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=2355,expires=0,avg_ttl=0
Comment From: oranagra
if you set replica-read-only to no, you may also want to set replica-ignore-maxmemory to no
note that writable replicas can easily get out of sync with the master, and commands it forward can easily fail (e.g. on WRONGTYPE, or no such key).
Comment From: dkam
Oh - that's an unexpected setting. What scenario would you want a replica to be writable but to ignore maxmemory?
edit I meant to say "unexpected setting!"
Comment From: oranagra
you can ask the same question about the master... if you wanna conduct your own deletions mechanism, and get OOM error if it fails.
Comment From: dkam
Perhaps there should be a warning in the Redis log when it's setup as a read-write replica that maxmemory will be ignored?
Comment From: enjoy-binbin
we do have a redis.conf that saying that:
# Starting from Redis 5, by default a replica will ignore its maxmemory setting
# (unless it is promoted to master after a failover or manually). It means
# that the eviction of keys will be just handled by the master, sending the
# DEL commands to the replica as keys evict in the master side.
#
# This behavior ensures that masters and replicas stay consistent, and is usually
# what you want, however if your replica is writable, or you want the replica
# to have a different memory setting, and you are sure all the writes performed
# to the replica are idempotent, then you may change this default (but be sure
# to understand what you are doing).
#
# Note that since the replica by default does not evict, it may end using more
# memory than the one set via maxmemory (there are certain buffers that may
# be larger on the replica, or data structures may sometimes take more memory
# and so forth). So make sure you monitor your replicas and make sure they
# have enough memory to never hit a real out-of-memory condition before the
# master hits the configured maxmemory setting.
#
# replica-ignore-maxmemory yes