Our production redis server crashed after about 2 months continuous uptime. Running ubuntu 14.04 Server on Amazon r3.large instance.
Here is the bug report from the logs :
=== REDIS BUG REPORT START: Cut & paste starting from here === 3916:M 13 Apr 18:20:04.854 # ------------------------------------------------ 3916:M 13 Apr 18:20:04.854 # !!! Software Failure. Press left mouse button to continue 3916:M 13 Apr 18:20:04.854 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3560 3916:M 13 Apr 18:20:04.854 # (forcing SIGSEGV in order to print the stack trace) 3916:M 13 Apr 18:20:04.854 # ------------------------------------------------ 3916:M 13 Apr 18:20:04.855 # Redis 3.0.3 crashed by signal: 11 3916:M 13 Apr 18:20:04.855 # Failed assertion:( :0) 3916:M 13 Apr 18:20:04.855 # --- STACK TRACE /usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x33)[0x44cc03] /usr/bin/redis-server 127.0.0.1:6379(_redisPanic+0x7a)[0x44bcaa] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7ffd022e4340] /usr/bin/redis-server 127.0.0.1:6379(_redisPanic+0x7a)[0x44bcaa] /usr/bin/redis-server 127.0.0.1:6379(redisOutOfMemoryHandler+0x2c)[0x41a47c] /usr/bin/redis-server 127.0.0.1:6379(zrealloc+0xd1)[0x421c51] /usr/bin/redis-server 127.0.0.1:6379[0x420410] /usr/bin/redis-server 127.0.0.1:6379(readQueryFromClient+0x40)[0x429280] /usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x250)[0x4174b0] /usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x4176eb] /usr/bin/redis-server 127.0.0.1:6379(main+0x324)[0x416474] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffd01f30ec5] /usr/bin/redis-server 127.0.0.1:6379[0x4165dd] 3916:M 13 Apr 18:20:04.867 # --- INFO OUTPUT 3916:M 13 Apr 18:20:04.870 # # Server redis_version:3.0.3 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:4284de9902883ae9 redis_mode:standalone os:Linux 3.13.0-48-generic x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.8.4 process_id:3916 run_id:df0008ed9e01742a1b67c82df6ed7cd9009fc914 tcp_port:6379 uptime_in_seconds:5010888 uptime_in_days:57 hz:10 lru_clock:953812 config_file:/etc/redis/redis.conf # Clients connected_clients:2 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 # Memory used_memory:15347358792 used_memory_human:14.29G used_memory_rss:15674212352 used_memory_peak:15347394056 used_memory_peak_human:14.29G used_memory_lua:36864 mem_fragmentation_ratio:1.02 mem_allocator:jemalloc-3.6.0 # Persistence loading:0 rdb_changes_since_last_save:123243343 rdb_bgsave_in_progress:0 rdb_last_save_time:1455560716 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok # Stats total_connections_received:165640680 total_commands_processed:1377978697 instantaneous_ops_per_sec:219 total_net_input_bytes:1173462639528 total_net_output_bytes:1588404651473 instantaneous_input_kbps:199.10 instantaneous_output_kbps:176.60 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:77861514 evicted_keys:0 keyspace_hits:87765756 keyspace_misses:77874722 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 # Replication role:master connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0 # CPU used_cpu_sys:56115.25 used_cpu_user:23884.62 used_cpu_sys_children:0.00 used_cpu_user_children:0.00 # Commandstats cmdstat_get:calls=165640478,usec=765297531,usec_per_call=4.62 cmdstat_setex:calls=77868202,usec=505434399,usec_per_call=6.49 cmdstat_sadd:calls=1134470017,usec=3159315929,usec_per_call=2.78 # Cluster cluster_enabled:0 # Keyspace db0:keys=10583,expires=9200,avg_ttl=466723 hash_init_value: 1456098809 3916:M 13 Apr 18:20:04.870 # --- CLIENT LIST OUTPUT 3916:M 13 Apr 18:20:04.870 # id=165640680 addr=127.0.0.1:40123 fd=5 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=165640681 addr=127.0.0.1:40125 fd=6 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get 3916:M 13 Apr 18:20:04.870 # --- CURRENT CLIENT INFO 3916:M 13 Apr 18:20:04.870 # client: id=165640680 addr=127.0.0.1:40123 fd=5 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get 3916:M 13 Apr 18:20:04.870 # --- REGISTERS 3916:M 13 Apr 18:20:04.870 # RAX:0000000000000000 RBX:00000000004885c8 RCX:0000000000000000 RDX:0000000000000001 RDI:00007ffd01406000 RSI:0000000000000159 RBP:0000000000498262 RSP:00007fff7d18db40 R8 :00007ff973d5fb60 R9 :00007ffd02c42780 R10:00007ffd022cc6a0 R11:0000000000000206 R12:0000000000000de8 R13:0000000000000005 R14:00007ffd014101f0 R15:0000000000000008 RIP:000000000044bcaa EFL:0000000000010206 CSGSFS:0000000000000033 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4f) -> 00007ffd014e40a0 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4e) -> 00007ffd0140ffd8 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4d) -> 0000000000429280 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4c) -> 0000000000000000 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4b) -> 00007ff97f7c6000 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db4a) -> 0000000000000006 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db49) -> 0000000000420410 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db48) -> 0000000000004000 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db47) -> 0000000000000000 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db46) -> 0000000000008000 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db45) -> 0000000000421c51 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db44) -> 0000000000000008 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db43) -> 000000000041a47c 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db42) -> 0000000000008009 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db41) -> 0000000000000010 3916:M 13 Apr 18:20:04.870 # (00007fff7d18db40) -> 0000000000000000 3916:M 13 Apr 18:20:04.870 # --- FAST MEMORY TEST 3916:M 13 Apr 18:20:04.871 # Bio thread for job type #0 terminated 3916:M 13 Apr 18:20:04.871 # Bio thread for job type #1 terminated 3916:M 13 Apr 18:22:28.728 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. 3916:M 13 Apr 18:22:28.732 # === REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: badboy
3916:M 13 Apr 18:20:04.854 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3560
used_memory_human:14.29G
You simply run out of memory.
Set maxmemory to avoid this, put less data into Redis or buy more memory. :)