My Redis has crashed, what should I do now? Thanks
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
16409:C 24 Aug 12:10:05.103 # --- FAST MEMORY TEST
16409:C 24 Aug 12:10:07.524 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
16409:C 24 Aug 12:10:07.524 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
6004:M 24 Aug 12:10:07.601 # Background saving terminated by signal 11
6004:M 24 Aug 12:10:11.007 * 1 changes in 900 seconds. Saving...
6004:M 24 Aug 12:10:11.008 * Background saving started by pid 16438
16438:C 24 Aug 12:10:11.013 #
=== REDIS BUG REPORT START: Cut & paste starting from here ===
16438:C 24 Aug 12:10:11.013 # === ASSERTION FAILED OBJECT CONTEXT ===
16438:C 24 Aug 12:10:11.013 # Object type: 7
16438:C 24 Aug 12:10:11.013 # Object encoding: 3
16438:C 24 Aug 12:10:11.013 # Object refcount: 0
16438:C 24 Aug 12:10:11.013 # === ASSERTION FAILED ===
16438:C 24 Aug 12:10:11.013 # ==> rdb.c:328 'sdsEncodedObject(obj)' is not true
16438:C 24 Aug 12:10:11.013 # (forcing SIGSEGV to print the bug report.)
16438:C 24 Aug 12:10:11.013 # Redis 3.0.6 crashed by signal: 11
16438:C 24 Aug 12:10:11.013 # SIGSEGV caused by address: 0xffffffffffffffff
16438:C 24 Aug 12:10:11.013 # Failed assertion: sdsEncodedObject(obj) (rdb.c:328)
16438:C 24 Aug 12:10:11.013 # --- STACK TRACE
redis-rdb-bgsave *:6379(logStackTrace+0xa6)[0x55798dff0c06]
redis-rdb-bgsave *:6379(_redisAssert+0x70)[0x55798dfef780]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6cb2252390]
redis-rdb-bgsave *:6379(_redisAssert+0x70)[0x55798dfef780]
redis-rdb-bgsave *:6379(rdbSaveStringObject+0x45)[0x55798dfd43f5]
redis-rdb-bgsave *:6379(rdbSaveObject+0xa7)[0x55798dfd4ce7]
redis-rdb-bgsave *:6379(rdbSaveKeyValuePair+0xc3)[0x55798dfd50b3]
redis-rdb-bgsave *:6379(rdbSaveRio+0x249)[0x55798dfd5389]
redis-rdb-bgsave *:6379(rdbSave+0x8b)[0x55798dfd573b]
redis-rdb-bgsave *:6379(rdbSaveBackground+0x67)[0x55798dfd5907]
redis-rdb-bgsave *:6379(serverCron+0x653)[0x55798dfbd963]
redis-rdb-bgsave *:6379(aeProcessEvents+0x27a)[0x55798dfb736a]
redis-rdb-bgsave *:6379(aeMain+0x2b)[0x55798dfb758b]
redis-rdb-bgsave *:6379(main+0x364)[0x55798dfb6204]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f6cb1e97840]
redis-rdb-bgsave *:6379(_start+0x29)[0x55798dfb6379]
16438:C 24 Aug 12:10:11.013 # --- INFO OUTPUT
16438:C 24 Aug 12:10:11.013 # # Server
redis_version:3.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7785291a3d2152db
redis_mode:standalone
os:Linux 4.15.0-52-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:5.4.0
process_id:16438
run_id:3e13bcf8209ce76b4080aaf3079ec5c998f33089
tcp_port:6379
uptime_in_seconds:2844654
uptime_in_days:32
hz:10
lru_clock:4435491
config_file:/etc/redis/redis.conf
# Clients
connected_clients:4
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:21249416
used_memory_human:20.27M
used_memory_rss:60723200
used_memory_peak:309100552
used_memory_peak_human:294.78M
used_memory_lua:36864
mem_fragmentation_ratio:2.86
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:3145
rdb_bgsave_in_progress:0
rdb_last_save_time:1598068707
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:2
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:5816989
total_commands_processed:342956949
instantaneous_ops_per_sec:0
total_net_input_bytes:53705612378
total_net_output_bytes:4452016981
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:365576
evicted_keys:0
keyspace_hits:31185229
keyspace_misses:183043558
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1028
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.00
used_cpu_user:0.01
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
cmdstat_del:calls=8591590,usec=91254871,usec_per_call=10.62
cmdstat_hset:calls=114310540,usec=380437050,usec_per_call=3.33
cmdstat_hget:calls=205335983,usec=514062639,usec_per_call=2.50
cmdstat_expire:calls=8892804,usec=28599920,usec_per_call=3.22
cmdstat_auth:calls=5826031,usec=11174140,usec_per_call=1.92
cmdstat_monitor:calls=1,usec=3,usec_per_call=3.00
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=6696,expires=6604,avg_ttl=36962513
hash_init_value: 1595760975
16438:C 24 Aug 12:10:11.013 # --- CLIENT LIST OUTPUT
16438:C 24 Aug 12:10:11.013 # id=356374 addr=101.36.181.186:45059 fd=7 name= age=2740634 idle=2740634 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=4873750 addr=63.171.18.34:8312 fd=128 name= age=616323 idle=616322 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=5253301 addr=60.190.248.12:64017 fd=152 name= age=400777 idle=400777 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=666215 addr=183.134.104.173:34987 fd=8 name= age=2612374 idle=2612374 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
16438:C 24 Aug 12:10:11.014 # --- REGISTERS
16438:C 24 Aug 12:10:11.014 #
RAX:0000000000000000 RBX:0000000000000148
RCX:00007f6cb2694370 RDX:0000000000000002
RDI:00007f6cb1406000 RSI:0000000000000181
RBP:000055798e033bc0 RSP:00007ffce8538810
R8 :00007f6cb2bb5740 R9 :00007f6cb2bb5740
R10:000000000000002a R11:0000000000000246
R12:000055798e033bd2 R13:00007f6c9d5d89d0
R14:00007f6c9e0f81f0 R15:00007ffce8538930
RIP:000055798dfef780 EFL:0000000000010202
CSGSFS:002b000000000033
16438:C 24 Aug 12:10:11.014 # (00007ffce853881f) -> 0000000000000000
16438:C 24 Aug 12:10:11.014 # (00007ffce853881e) -> 0000000000000008
16438:C 24 Aug 12:10:11.014 # (00007ffce853881d) -> 00007ffce85389b0
16438:C 24 Aug 12:10:11.014 # (00007ffce853881c) -> 0000000000000008
16438:C 24 Aug 12:10:11.014 # (00007ffce853881b) -> 7a22aac0b76e6f00
16438:C 24 Aug 12:10:11.014 # (00007ffce853881a) -> 0000000000000008
16438:C 24 Aug 12:10:11.014 # (00007ffce8538819) -> 7a22aac0b76e6f00
16438:C 24 Aug 12:10:11.014 # (00007ffce8538818) -> 0000000000000000
16438:C 24 Aug 12:10:11.014 # (00007ffce8538817) -> 00007f6c9e0f81f0
16438:C 24 Aug 12:10:11.014 # (00007ffce8538816) -> 0000000000000017
16438:C 24 Aug 12:10:11.014 # (00007ffce8538815) -> 000055798dfd4ce7
16438:C 24 Aug 12:10:11.014 # (00007ffce8538814) -> 0000000000000037
16438:C 24 Aug 12:10:11.014 # (00007ffce8538813) -> 000055798dfd43f5
16438:C 24 Aug 12:10:11.014 # (00007ffce8538812) -> 00007f6c9e0b81a0
16438:C 24 Aug 12:10:11.014 # (00007ffce8538811) -> 00007ffce85389b0
16438:C 24 Aug 12:10:11.014 # (00007ffce8538810) -> 0000000000000478
16438:C 24 Aug 12:10:11.014 # --- FAST MEMORY TEST
16438:C 24 Aug 12:10:13.424 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
16438:C 24 Aug 12:10:13.424 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
6004:M 24 Aug 12:10:13.512 # Background saving terminated by signal 11
Aditional information
I did run redis-server --test-memory and got: *** MEMORY ERROR DETECTED: 0x7f9cdd8d3ea0 != 0x7f9d5d8d3ea0 (0 vs 2048)......
Comment From: JimWestergren
UPDATE: Ok, I have found out that it was the background saving process that was stuck. BGSAVE did not work. SAVE did work and things went back to normal for the server.
I check the error log I see this:
6004:M 24 Aug 12:28:33.296 # --- CURRENT CLIENT INFO
6004:M 24 Aug 12:28:33.296 # client: id=5817047 addr=127.0.0.1:48392 fd=6 name= age=368 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=save
6004:M 24 Aug 12:28:33.296 # argv[0]: 'SAVE'
25043:M 24 Aug 12:28:33.839 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
25043:M 24 Aug 12:28:33.839 # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.
25043:M 24 Aug 12:28:33.839 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.0.6 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 25043
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
25043:M 24 Aug 12:28:33.839 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
25043:M 24 Aug 12:28:33.839 # Server started, Redis version 3.0.6
25043:M 24 Aug 12:28:33.839 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
25043:M 24 Aug 12:28:33.839 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
25043:M 24 Aug 12:28:33.984 * DB loaded from disk: 0.145 seconds
25043:M 24 Aug 12:28:33.984 * The server is now ready to accept connections on port 6379
25043:M 24 Aug 12:28:44.102 * DB saved on disk
25043:M 24 Aug 12:29:12.704 * DB saved on disk
25043:M 24 Aug 12:34:13.099 * 10 changes in 300 seconds. Saving...
25043:M 24 Aug 12:34:13.100 * Background saving started by pid 31383
31383:C 24 Aug 12:34:13.224 * DB saved on disk
31383:C 24 Aug 12:34:13.225 * RDB: 1 MB of memory used by copy-on-write
25043:M 24 Aug 12:34:13.300 * Background saving terminated with success
So I see that Redis is warning me about background save and memory problems. I have now done the changes according to the instructions in the warning messages. It should be fine now.
Comment From: oranagra
16438:C 24 Aug 12:10:11.013 # Object type: 7
16438:C 24 Aug 12:10:11.013 # Object encoding: 3
that's very odd. the highest object type in redis 3.0 is
#define REDIS_HASH 4