I have a large appendonly.aof file 1.2G. When I try startup redis-server in a docker container I it is Killed after about 12 seconds with message "Killed" and no descriptive error...
I started redis with docker (with and without --oom-kill-disable) ...
docker run --oom-kill-disable -v my_volume_name:/data redis:6.0.5-alpine sleep 100000
Then from the shell ran redis-server...
/data # redis-server --appendonly yes
43:C 31 Jul 2020 04:08:29.466 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
43:C 31 Jul 2020 04:08:29.466 # Redis version=6.0.5, bits=64, commit=00000000, modified=0, pid=43, just started
43:C 31 Jul 2020 04:08:29.466 # Configuration loaded
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.0.5 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 43
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
43:M 31 Jul 2020 04:08:29.467 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
43:M 31 Jul 2020 04:08:29.467 # Server initialized
43:M 31 Jul 2020 04:08:29.467 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
43:M 31 Jul 2020 04:08:29.468 * Reading RDB preamble from AOF file...
43:M 31 Jul 2020 04:08:29.468 * Loading RDB produced by version 6.0.5
43:M 31 Jul 2020 04:08:29.468 * RDB age 8647 seconds
43:M 31 Jul 2020 04:08:29.468 * RDB memory usage when created 3258.41 Mb
43:M 31 Jul 2020 04:08:29.468 * RDB has an AOF tail
Killed
appendonly.aof file is apparently valid...
redis-check-aof appendonly.aof
The AOF appears to start with an RDB preamble.
Checking the RDB preamble to start:
[offset 0] Checking RDB file appendonly.aof
[offset 26] AUX FIELD redis-ver = '6.0.5'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1596159862'
[offset 73] AUX FIELD used-mem = '3416689400'
[offset 89] AUX FIELD aof-preamble = '1'
[offset 91] Selecting DB ID 0
[offset 1280111072] Checksum OK
[offset 1280111072] \o/ RDB looks OK! \o/
[info] 18115 keys read
[info] 1 expires
[info] 0 already expired
RDB preamble is OK, proceeding with AOF tail...
AOF analyzed: size=1280111072, ok_up_to=1280111072, diff=0
AOF is valid
Comment From: calbot
The problem only seems to affect redis-server in a docker container. When I run it outside a container it reads the file...
redis-server --appendonly yes
14783:C 30 Jul 2020 21:27:57.128 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
14783:C 30 Jul 2020 21:27:57.128 # Redis version=6.0.6, bits=64, commit=00000000, modified=0, pid=14783, just started
14783:C 30 Jul 2020 21:27:57.128 # Configuration loaded
14783:M 30 Jul 2020 21:27:57.129 * Increased maximum number of open files to 10032 (it was originally set to 256).
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.0.6 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 14783
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
14783:M 30 Jul 2020 21:27:57.130 # Server initialized
14783:M 30 Jul 2020 21:27:57.130 * Reading RDB preamble from AOF file...
14783:M 30 Jul 2020 21:27:57.130 * Loading RDB produced by version 6.0.5
14783:M 30 Jul 2020 21:27:57.130 * RDB age 9815 seconds
14783:M 30 Jul 2020 21:27:57.130 * RDB memory usage when created 3258.41 Mb
14783:M 30 Jul 2020 21:27:57.130 * RDB has an AOF tail
14783:M 30 Jul 2020 21:28:03.868 * Reading the remaining AOF tail...
14783:M 30 Jul 2020 21:28:03.868 * DB loaded from append only file: 6.738 seconds
14783:M 30 Jul 2020 21:28:03.868 * Ready to accept connections
Comment From: calbot
It looks like the docker engine memory was too low to handle loading the large file into memory since when I boosted the RAM to 6 gb it worked. It would be nice if there was a better error message for this issue. I'm not sure if that is possible from the redis side or it's a docker issue.
Comment From: hwware
@calbot hello, AFAIK the killed message is coming from docker side, and you are right, when the container reaching the max memory limit the daemon will kill it and shows the 'killed' message. So I don't think this is anything need to do in Redis side. Thank you!
Comment From: oranagra
Indeed.. redis gets SIGKILL and has no way to handle it and print a message.