I get an out of memory error when running redis without persistence as a writable slave on my laptop with 16GB RAM:
1197s139 redis-server --bind 172.17.0.1 --unixsocket /tmp/redis.sock --save "" --appendonly no --slaveof 172.17.0.1 6380 --slave-read-only no
12168:C 05 Mar 2020 17:21:58.226 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
12168:C 05 Mar 2020 17:21:58.226 # Redis version=5.0.7, bits=64, commit=e8d3276a, modified=0, pid=12168, just started
12168:C 05 Mar 2020 17:21:58.226 # Configuration loaded
12168:S 05 Mar 2020 17:21:58.227 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
12168:S 05 Mar 2020 17:21:58.227 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
12168:S 05 Mar 2020 17:21:58.227 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 5.0.7 (e8d3276a/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 12168
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
12168:S 05 Mar 2020 17:21:58.228 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
12168:S 05 Mar 2020 17:21:58.228 # Server initialized
12168:S 05 Mar 2020 17:21:58.228 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
12168:S 05 Mar 2020 17:22:00.032 * DB loaded from disk: 1.803 seconds
12168:S 05 Mar 2020 17:22:00.032 * Ready to accept connections
12168:S 05 Mar 2020 17:22:00.032 * The server is now ready to accept connections at /tmp/redis.sock
12168:S 05 Mar 2020 17:22:00.032 * Connecting to MASTER 172.17.0.1:6380
12168:S 05 Mar 2020 17:22:00.032 * MASTER <-> REPLICA sync started
12168:S 05 Mar 2020 17:22:00.032 * Non blocking connect for SYNC fired the event.
12168:S 05 Mar 2020 17:22:00.124 * Master replied to PING, replication can continue...
12168:S 05 Mar 2020 17:22:00.215 * Partial resynchronization not possible (no cached master)
12168:S 05 Mar 2020 17:22:06.862 * Full resync from master: ebce80a93728672f6b0def6f58a5117e8d78db2a:1553896998986
12168:S 05 Mar 2020 17:22:06.887 * MASTER <-> REPLICA sync: receiving streamed RDB from master
12168:S 05 Mar 2020 17:23:50.920 * MASTER <-> REPLICA sync: Flushing old data
12168:S 05 Mar 2020 17:23:50.960 * MASTER <-> REPLICA sync: Loading DB in memory
12168:S 05 Mar 2020 17:23:52.682 * MASTER <-> REPLICA sync: Finished with success
12168:S 05 Mar 2020 17:34:09.646 # Out Of Memory allocating 7675916 bytes!
=== REDIS BUG REPORT START: Cut & paste starting from here ===
12168:S 05 Mar 2020 17:34:09.646 # ------------------------------------------------
12168:S 05 Mar 2020 17:34:09.646 # !!! Software Failure. Press left mouse button to continue
12168:S 05 Mar 2020 17:34:09.646 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:3925
12168:S 05 Mar 2020 17:34:09.646 # ------------------------------------------------
zsh: segmentation fault redis-server --bind 172.17.0.1 --unixsocket /tmp/redis.sock --save "" no
I'm monitoring free memory:
% while true; do date; free; redis-cli -h 172.17.0.1 -p 6379 -n 15 info Memory | grep used_memory | grep human; sleep 10; done
Thu Mar 5 17:21:54 CET 2020
total used free shared buff/cache available
Mem: 15989524 11647248 2829500 1317572 1512776 2652536
Swap: 0 0 0
Could not connect to Redis at 172.17.0.1:6379: Connection refused
Thu Mar 5 17:22:04 CET 2020
total used free shared buff/cache available
Mem: 15989524 12359072 1853488 1317492 1776964 1851280
Swap: 0 0 0
used_memory_human:687.74M
used_memory_rss_human:697.35M
used_memory_peak_human:687.74M
used_memory_lua_human:37.00K
used_memory_scripts_human:0B
… [about the time MASTER <-> REPLICA sync: Finished with success]
Thu Mar 5 17:23:04 CET 2020
total used free shared buff/cache available
Mem: 15989524 12422380 1534552 1371544 2032592 1732000
Swap: 0 0 0
used_memory_human:687.74M
used_memory_rss_human:697.35M
used_memory_peak_human:687.74M
used_memory_lua_human:37.00K
used_memory_scripts_human:0B
Thu Mar 5 17:23:14 CET 2020
total used free shared buff/cache available
Mem: 15989524 12460972 1457284 1371584 2071268 1692948
Swap: 0 0 0
used_memory_human:687.74M
used_memory_rss_human:697.35M
used_memory_peak_human:687.74M
used_memory_lua_human:37.00K
used_memory_scripts_human:0B
…
Thu Mar 5 17:33:54 CET 2020
total used free shared buff/cache available
Mem: 15989524 11463496 2887568 922656 1638460 3138284
Swap: 0 0 0
used_memory_human:689.16M
used_memory_rss_human:699.35M
used_memory_peak_human:689.16M
used_memory_lua_human:114.00K
used_memory_scripts_human:48.71K
Thu Mar 5 17:34:04 CET 2020
total used free shared buff/cache available
Mem: 15989524 11473152 2874432 925868 1641940 3125356
Swap: 0 0 0
used_memory_human:689.21M
used_memory_rss_human:699.39M
used_memory_peak_human:689.23M
used_memory_lua_human:132.00K
used_memory_scripts_human:55.38K
Thu Mar 5 17:34:14 CET 2020
total used free shared buff/cache available
Mem: 15989524 10287708 4638016 919500 1063800 4435480
Swap: 0 0 0
Could not connect to Redis at 172.17.0.1:6379: Connection refused
It looks like there were still 2.8GB free when Redis crashed.
Comment From: niko
Found out more: I'm inserting data into a stream. That seems to take a huge amount of memory. I still have to investigate why.
Comment From: niko
Ok. My bad. I accidentally shoveled much more data into Redis that I intended to … so fast that my 10 second watch interval didn't capture the crucial part. Sorry for the fuzz.