=== REDIS BUG REPORT START: Cut & paste starting from here ===
[2602] 23 Jun 13:09:47.162 # ------------------------------------------------
[2602] 23 Jun 13:09:47.162 # !!! Software Failure. Press left mouse button to continue
[2602] 23 Jun 13:09:47.162 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3034
[2602] 23 Jun 13:09:47.162 # (forcing SIGSEGV in order to print the stack trace)
[2602] 23 Jun 13:09:47.162 # ------------------------------------------------
[2602] 23 Jun 13:09:47.162 # Redis 2.8.9 crashed by signal: 11
[2602] 23 Jun 13:09:47.162 # Failed assertion:
Clients
connected_clients:7 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0
Memory
used_memory:20951265272 used_memory_human:19.51G used_memory_rss:21347344384 used_memory_peak:20951283128 used_memory_peak_human:19.51G used_memory_lua:33792 mem_fragmentation_ratio:1.02 mem_allocator:jemalloc-3.2.0
Persistence
loading:0 rdb_changes_since_last_save:652068 rdb_bgsave_in_progress:0 rdb_last_save_time:1403514587 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:180 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok
Stats
total_connections_received:52 total_commands_processed:734212769 instantaneous_ops_per_sec:5274 rejected_connections:0 sync_full:2 sync_partial_ok:0 sync_partial_err:0 expired_keys:26895354 evicted_keys:0 keyspace_hits:159279606 keyspace_misses:5759 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:388414
Replication
role:master connected_slaves:0 master_repl_offset:834095387 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:833046812 repl_backlog_histlen:1048576
CPU
used_cpu_sys:17022.28 used_cpu_user:14886.55 used_cpu_sys_children:55456.23 used_cpu_user_children:545360.00
Commandstats
cmdstat_get:calls=216948,usec=1933291,usec_per_call=8.91 cmdstat_setex:calls=10202,usec=501259,usec_per_call=49.13 cmdstat_incr:calls=221108,usec=3338667,usec_per_call=15.10 cmdstat_sadd:calls=355944681,usec=2015039039,usec_per_call=5.66 cmdstat_scard:calls=1143833,usec=5519998,usec_per_call=4.83 cmdstat_smembers:calls=82843,usec=4624665,usec_per_call=55.82 cmdstat_zadd:calls=1075643,usec=15109398,usec_per_call=14.05 cmdstat_zincrby:calls=128918956,usec=2130632869,usec_per_call=16.53 cmdstat_zrange:calls=5811,usec=84018673,usec_per_call=14458.56 cmdstat_zrevrange:calls=460,usec=47251,usec_per_call=102.72 cmdstat_hset:calls=442216,usec=38878046,usec_per_call=87.92 cmdstat_hget:calls=33184,usec=822482,usec_per_call=24.79 cmdstat_hgetall:calls=2,usec=6428043,usec_per_call=3214021.50 cmdstat_incrby:calls=88314590,usec=739955598,usec_per_call=8.38 cmdstat_expire:calls=157802283,usec=1342698124,usec_per_call=8.51 cmdstat_ping:calls=2,usec=3,usec_per_call=1.50 cmdstat_psync:calls=2,usec=339,usec_per_call=169.50 cmdstat_replconf:calls=2,usec=637,usec_per_call=318.50 cmdstat_info:calls=3,usec=268,usec_per_call=89.33
Keyspace
db0:keys=36404205,expires=33556031,avg_ttl=175378447 hash_init_value: 1402844671
[2602] 23 Jun 13:09:47.166 # --- CLIENT LIST OUTPUT [2602] 23 Jun 13:09:47.166 # addr=192.168.0.66:52571 fd=9 name= age=1019054 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=zincrby addr=192.168.0.66:52579 fd=10 name= age=1018986 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire addr=192.168.0.66:52612 fd=11 name= age=1018756 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=expire addr=192.168.0.66:57712 fd=5 name= age=5766 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get addr=192.168.0.66:57713 fd=6 name= age=5765 idle=2834 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=smembers addr=192.168.0.66:57714 fd=7 name= age=5756 idle=2836 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=zrange addr=192.168.0.66:57756 fd=8 name= age=4048 idle=2837 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
[2602] 23 Jun 13:09:47.166 # --- CURRENT CLIENT INFO [2602] 23 Jun 13:09:47.166 # client: addr=192.168.0.66:52579 fd=10 name= age=1018986 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire [2602] 23 Jun 13:09:47.166 # argv[0]: 'EXPIRE' [2602] 23 Jun 13:09:47.166 # argv[1]: 'ips_91.219.158.52_23062014_2963416_cntr' [2602] 23 Jun 13:09:47.166 # argv[2]: '604800' [2602] 23 Jun 13:09:47.166 # key 'ips_91.219.158.52_23062014_2963416_cntr' found in DB containing the following object: [2602] 23 Jun 13:09:47.167 # Object type: 0 [2602] 23 Jun 13:09:47.167 # Object encoding: 1 [2602] 23 Jun 13:09:47.167 # Object refcount: 14549607 [2602] 23 Jun 13:09:47.167 # --- REGISTERS [2602] 23 Jun 13:09:47.167 # RAX:0000000000000000 RBX:0000000000000bda RCX:0000000000705610 RDX:0000000000000001 RDI:00007fce7b6c1ea0 RSI:00007fce7b48cee8 RBP:0000000040000000 RSP:00007fff415744c0 R8 :00007fce7b6c1ea0 R9 :00007fce7b6c1ef8 R10:00007fce7bf6b720 R11:00007fce7b6c1ef0 R12:00000000004b381e R13:00000000004a9e18 R14:0000000000000007 R15:0000000000000000 RIP:000000000044e108 EFL:0000000000010202 CSGSFS:0000000000000033 [2602] 23 Jun 13:09:47.167 # (00007fff415744cf) -> 00007fce7a93b280 [2602] 23 Jun 13:09:47.167 # (00007fff415744ce) -> 00000146c7fd0816 [2602] 23 Jun 13:09:47.167 # (00007fff415744cd) -> 000000000041d5b1 [2602] 23 Jun 13:09:47.167 # (00007fff415744cc) -> 00000146ec098c16 [2602] 23 Jun 13:09:47.167 # (00007fff415744cb) -> 00007fce7a812200 [2602] 23 Jun 13:09:47.167 # (00007fff415744ca) -> 0000000000000001 [2602] 23 Jun 13:09:47.167 # (00007fff415744c9) -> 000000000041d457 [2602] 23 Jun 13:09:47.167 # (00007fff415744c8) -> 00007fce7a812200 [2602] 23 Jun 13:09:47.167 # (00007fff415744c7) -> 0000000008000000 [2602] 23 Jun 13:09:47.167 # (00007fff415744c6) -> 00007fcb4648a7c0 [2602] 23 Jun 13:09:47.167 # (00007fff415744c5) -> 00000000004250b9 [2602] 23 Jun 13:09:47.167 # (00007fff415744c4) -> 0000000000000000 [2602] 23 Jun 13:09:47.167 # (00007fff415744c3) -> 0000000000421d22 [2602] 23 Jun 13:09:47.167 # (00007fff415744c2) -> 00007fc981bf7228 [2602] 23 Jun 13:09:47.167 # (00007fff415744c1) -> 0000000007ffffff [2602] 23 Jun 13:09:47.167 # (00007fff415744c0) -> 0000000000000000 [2602] 23 Jun 13:09:47.167 # --- FAST MEMORY TEST [2602] 23 Jun 13:09:47.167 # Bio thread for job type #0 terminated [2602] 23 Jun 13:09:47.167 # Bio thread for job type #1 terminated [2602] 23 Jun 13:14:03.358 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. [2602] 23 Jun 13:14:03.358 # === REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: skysbird
how much is your server memory ? 2014Äê6ÔÂ24ÈÕ ÏÂÎç1:52ÓÚ "911kg" notifications@github.comдµÀ£º
=== REDIS BUG REPORT START: Cut & paste starting from here ===
[2602] 23 Jun 13:09:47.162
[2602] 23 Jun 13:09:47.162 # !!! Software Failure. Press left mouse button to continue [2602] 23 Jun 13:09:47.162 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:3034 [2602] 23 Jun 13:09:47.162 # (forcing SIGSEGV in order to print the stack trace)
[2602] 23 Jun 13:09:47.162
[2602] 23 Jun 13:09:47.162 # Redis 2.8.9 crashed by signal: 11 [2602] 23 Jun 13:09:47.162 # Failed assertion: (:0) [2602] 23 Jun 13:09:47.162 # --- STACK TRACE /usr/sbin/redis-server :6379(logStackTrace+0x4b)[0x44e66b] /usr/sbin/redis-server :6379(_redisPanic+0x88)[0x44e108] /lib64/libpthread.so.0(+0xf7c0)[0x7fce7b6d57c0] /usr/sbin/redis-server :6379(_redisPanic+0x88)[0x44e108] /usr/sbin/redis-server :6379(redisOutOfMemoryHandler+0x32)[0x421d22] /usr/sbin/redis-server :6379(zcalloc+0x79)[0x4250b9] /usr/sbin/redis-server :6379(dictExpand+0x87)[0x41d457] /usr/sbin/redis-server :6379(dictAddRaw+0x111)[0x41d5b1] /usr/sbin/redis-server :6379(setExpire+0x39)[0x42ec79] /usr/sbin/redis-server :6379(expireGenericCommand+0xa1)[0x42ff51] /usr/sbin/redis-server :6379(call+0x78)[0x41fad8] /usr/sbin/redis-server :6379(processCommand+0x4a5)[0x420205] /usr/sbin/redis-server :6379(processInputBuffer+0x4f)[0x42b8af] /usr/sbin/redis-server :6379(readQueryFromClient+0xc9)[0x42b9e9] /usr/sbin/redis-server :6379(aeProcessEvents+0x142)[0x41aad2] /usr/sbin/redis-server :6379(aeMain+0x2b)[0x41ad7b] /usr/sbin/redis-server :6379(main+0x2d2)[0x4234a2] /lib64/libc.so.6(__libc_start_main+0xe6)[0x7fce7b36dc36] /usr/sbin/redis-server *:6379[0x41a169] [2602] 23 Jun 13:09:47.166 # --- INFO OUTPUT [2602] 23 Jun 13:09:47.166 # # Server redis_version:2.8.9 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:8a388cdd0244d30c redis_mode:standalone os:Linux 3.0.76-0.11-default x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.3.4 process_id:2602 run_id:1373467ec872126bb9ce9fc50d88a34d99e740af tcp_port:6379 uptime_in_seconds:1024031 uptime_in_days:11 hz:10 lru_clock:11005659 config_file:/etc/redis/6379.conf Clients
connected_clients:7 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 Memory
used_memory:20951265272 used_memory_human:19.51G used_memory_rss:21347344384 used_memory_peak:20951283128 used_memory_peak_human:19.51G used_memory_lua:33792 mem_fragmentation_ratio:1.02 mem_allocator:jemalloc-3.2.0 Persistence
loading:0 rdb_changes_since_last_save:652068 rdb_bgsave_in_progress:0 rdb_last_save_time:1403514587 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:180 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok Stats
total_connections_received:52 total_commands_processed:734212769 instantaneous_ops_per_sec:5274 rejected_connections:0 sync_full:2 sync_partial_ok:0 sync_partial_err:0 expired_keys:26895354 evicted_keys:0 keyspace_hits:159279606 keyspace_misses:5759 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:388414 Replication
role:master connected_slaves:0 master_repl_offset:834095387 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:833046812 repl_backlog_histlen:1048576 CPU
used_cpu_sys:17022.28 used_cpu_user:14886.55 used_cpu_sys_children:55456.23 used_cpu_user_children:545360.00 Commandstats
cmdstat_get:calls=216948,usec=1933291,usec_per_call=8.91 cmdstat_setex:calls=10202,usec=501259,usec_per_call=49.13 cmdstat_incr:calls=221108,usec=3338667,usec_per_call=15.10 cmdstat_sadd:calls=355944681,usec=2015039039,usec_per_call=5.66 cmdstat_scard:calls=1143833,usec=5519998,usec_per_call=4.83 cmdstat_smembers:calls=82843,usec=4624665,usec_per_call=55.82 cmdstat_zadd:calls=1075643,usec=15109398,usec_per_call=14.05 cmdstat_zincrby:calls=128918956,usec=2130632869,usec_per_call=16.53 cmdstat_zrange:calls=5811,usec=84018673,usec_per_call=14458.56 cmdstat_zrevrange:calls=460,usec=47251,usec_per_call=102.72 cmdstat_hset:calls=442216,usec=38878046,usec_per_call=87.92 cmdstat_hget:calls=33184,usec=822482,usec_per_call=24.79 cmdstat_hgetall:calls=2,usec=6428043,usec_per_call=3214021.50 cmdstat_incrby:calls=88314590,usec=739955598,usec_per_call=8.38 cmdstat_expire:calls=157802283,usec=1342698124,usec_per_call=8.51 cmdstat_ping:calls=2,usec=3,usec_per_call=1.50 cmdstat_psync:calls=2,usec=339,usec_per_call=169.50 cmdstat_replconf:calls=2,usec=637,usec_per_call=318.50 cmdstat_info:calls=3,usec=268,usec_per_call=89.33 Keyspace
db0:keys=36404205,expires=33556031,avg_ttl=175378447 hash_init_value: 1402844671
[2602] 23 Jun 13:09:47.166 # --- CLIENT LIST OUTPUT [2602] 23 Jun 13:09:47.166 # addr=192.168.0.66:52571 fd=9 name= age=1019054 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=zincrby addr=192.168.0.66:52579 fd=10 name= age=1018986 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire addr=192.168.0.66:52612 fd=11 name= age=1018756 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=expire addr=192.168.0.66:57712 fd=5 name= age=5766 idle=8 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get addr=192.168.0.66:57713 fd=6 name= age=5765 idle=2834 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=smembers addr=192.168.0.66:57714 fd=7 name= age=5756 idle=2836 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=zrange addr=192.168.0.66:57756 fd=8 name= age=4048 idle=2837 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
[2602] 23 Jun 13:09:47.166 # --- CURRENT CLIENT INFO [2602] 23 Jun 13:09:47.166 # client: addr=192.168.0.66:52579 fd=10 name= age=1018986 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire [2602] 23 Jun 13:09:47.166 # argv[0]: 'EXPIRE' [2602] 23 Jun 13:09:47.166 # argv[1]: 'ips_91.219.158.52_23062014_2963416_cntr' [2602] 23 Jun 13:09:47.166 # argv[2]: '604800' [2602] 23 Jun 13:09:47.166 # key 'ips_91.219.158.52_23062014_2963416_cntr' found in DB containing the following object: [2602] 23 Jun 13:09:47.167 # Object type: 0 [2602] 23 Jun 13:09:47.167 # Object encoding: 1 [2602] 23 Jun 13:09:47.167 # Object refcount: 14549607 [2602] 23 Jun 13:09:47.167 # --- REGISTERS [2602] 23 Jun 13:09:47.167 # RAX:0000000000000000 RBX:0000000000000bda RCX:0000000000705610 RDX:0000000000000001 RDI:00007fce7b6c1ea0 RSI:00007fce7b48cee8 RBP:0000000040000000 RSP:00007fff415744c0 R8 :00007fce7b6c1ea0 R9 :00007fce7b6c1ef8 R10:00007fce7bf6b720 R11:00007fce7b6c1ef0 R12:00000000004b381e R13:00000000004a9e18 R14:0000000000000007 R15:0000000000000000 RIP:000000000044e108 EFL:0000000000010202 CSGSFS:0000000000000033 [2602] 23 Jun 13:09:47.167 # (00007fff415744cf) -> 00007fce7a93b280 [2602] 23 Jun 13:09:47.167 # (00007fff415744ce) -> 00000146c7fd0816 [2602] 23 Jun 13:09:47.167 # (00007fff415744cd) -> 000000000041d5b1 [2602] 23 Jun 13:09:47.167 # (00007fff415744cc) -> 00000146ec098c16 [2602] 23 Jun 13:09:47.167 # (00007fff415744cb) -> 00007fce7a812200 [2602] 23 Jun 13:09:47.167 # (00007fff415744ca) -> 0000000000000001 [2602] 23 Jun 13:09:47.167 # (00007fff415744c9) -> 000000000041d457 [2602] 23 Jun 13:09:47.167 # (00007fff415744c8) -> 00007fce7a812200 [2602] 23 Jun 13:09:47.167 # (00007fff415744c7) -> 0000000008000000 [2602] 23 Jun 13:09:47.167 # (00007fff415744c6) -> 00007fcb4648a7c0 [2602] 23 Jun 13:09:47.167 # (00007fff415744c5) -> 00000000004250b9 [2602] 23 Jun 13:09:47.167 # (00007fff415744c4) -> 0000000000000000 [2602] 23 Jun 13:09:47.167 # (00007fff415744c3) -> 0000000000421d22 [2602] 23 Jun 13:09:47.167 # (00007fff415744c2) -> 00007fc981bf7228 [2602] 23 Jun 13:09:47.167 # (00007fff415744c1) -> 0000000007ffffff [2602] 23 Jun 13:09:47.167 # (00007fff415744c0) -> 0000000000000000 [2602] 23 Jun 13:09:47.167 # --- FAST MEMORY TEST [2602] 23 Jun 13:09:47.167 # Bio thread for job type #0 terminated [2602] 23 Jun 13:09:47.167 # Bio thread for job type #1 https://github.com/antirez/redis/issues/1 terminated [2602] 23 Jun 13:14:03.358 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. [2602] 23 Jun 13:14:03.358 # === REDIS BUG REPORT END. Make sure to include from START to END. ===
¡ª Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/1833.
Comment From: 911kg
24 GB
Comment From: badboy
Redis consumed about 20 GB of that 24 GB. That means only 4 GB are available for the system and other applications. You should set maxmemory and choose a maxmemory-policy, so Redis will do the right thing when it consumes all memory (or better: all memory that you allowed).
Comment From: mattsta
This was an explicit out of memory error (/usr/sbin/redis-server *:6379(redisOutOfMemoryHandler+0x32)[0x421d22]) that happens when Redis has no memory limit and can't allocate more space.
So, there's no Redis bug here, it's just default behavior. If Redis is set to use unlimited memory (the default) and can't actually allocate memory, and no keys are available for expiration or eviction, Redis exits. If you set maxmemory, Redis will avoid crashing and just reply with OOM errors to your commands until your memory usage is reduced.
Closing this issue under @badboy's advice of setting maxmemory along with an eviction policy (alternatively: upgrade to a system with more memory if you need all your data).
Comment From: DeeKey
Is it a peace of humor? "[2602] 23 Jun 13:09:47.162 # !!! Software Failure. Press left mouse button to continue" Mouse button in server log? :)