=== REDIS BUG REPORT START: Cut & paste starting from here ===
23529:M 15 Dec 12:11:01.062 # ------------------------------------------------
23529:M 15 Dec 12:11:01.062 # !!! Software Failure. Press left mouse button to continue
23529:M 15 Dec 12:11:01.062 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.c:3580
23529:M 15 Dec 12:11:01.062 # (forcing SIGSEGV in order to print the stack trace)
23529:M 15 Dec 12:11:01.062 # ------------------------------------------------
23529:M 15 Dec 12:11:01.062 # Redis 4.0.6 crashed by signal: 11
23529:M 15 Dec 12:11:01.062 # Crashed running the instuction at: 0x4666a7
23529:M 15 Dec 12:11:01.062 # Accessing address: 0xffffffffffffffff
23529:M 15 Dec 12:11:01.062 # Failed assertion:
------ STACK TRACE ------ EIP: src/redis-server 127.0.0.1:21379(_serverPanic+0x117)[0x4666a7]
Backtrace: src/redis-server 127.0.0.1:21379(logStackTrace+0x29)[0x4680c9] src/redis-server 127.0.0.1:21379(sigsegvHandler+0xac)[0x46876c] /lib64/libpthread.so.0(+0xf5a0)[0x7fa22c0d95a0] src/redis-server 127.0.0.1:21379(_serverPanic+0x117)[0x4666a7] src/redis-server 127.0.0.1:21379(redisOutOfMemoryHandler+0x2e)[0x429bfe] src/redis-server 127.0.0.1:21379(zmalloc+0x49)[0x432829] src/redis-server 127.0.0.1:21379(createReplicationBacklog+0x1a)[0x44236a] src/redis-server 127.0.0.1:21379(syncCommand+0x248)[0x445d38] src/redis-server 127.0.0.1:21379(call+0x9e)[0x42bd6e] src/redis-server 127.0.0.1:21379(processCommand+0x367)[0x42c417] src/redis-server 127.0.0.1:21379(processInputBuffer+0x105)[0x43b4a5] src/redis-server 127.0.0.1:21379(aeProcessEvents+0x23d)[0x42655d] src/redis-server 127.0.0.1:21379(aeMain+0x2b)[0x42683b] src/redis-server 127.0.0.1:21379(main+0x49f)[0x4236af] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa22bd27c05] src/redis-server 127.0.0.1:21379[0x4239a5]
------ INFO OUTPUT ------
Server
redis_version:4.0.6 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:9cc322fbcf966c19 redis_mode:standalone os:Linux 4.9.62-21.56.amzn1.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:4.8.5 process_id:23529 run_id:24ec82c341dbc958379f308a8968da45ba66aed8 tcp_port:21379 uptime_in_seconds:1 uptime_in_days:0 hz:10 lru_clock:3390420 executable:/var/www/backend/redis-stable/src/redis-server config_file:/var/www/backend/redis-stable/./tests/tmp/redis.conf.21577.36
Clients
connected_clients:4 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0
Memory
used_memory:834312 used_memory_human:814.76K used_memory_rss:4222976 used_memory_rss_human:4.03M used_memory_peak:834312 used_memory_peak_human:814.76K used_memory_peak_perc:109.08% used_memory_overhead:719042 used_memory_startup:487232 used_memory_dataset:115270 used_memory_dataset_perc:33.21% total_system_memory:1041707008 total_system_memory_human:993.45M used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:5.06 mem_allocator:jemalloc-4.0.3 active_defrag_running:0 lazyfree_pending_objects:0
Persistence
loading:0 rdb_changes_since_last_save:769 rdb_bgsave_in_progress:0 rdb_last_save_time:1513339859 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0
Stats
total_connections_received:6 total_commands_processed:2533 instantaneous_ops_per_sec:1446 total_net_input_bytes:193706 total_net_output_bytes:26668 instantaneous_input_kbps:107.27 instantaneous_output_kbps:14.67 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:1 expired_keys:0 evicted_keys:0 keyspace_hits:690 keyspace_misses:672 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0
Replication
role:master connected_slaves:1 slave0:ip=127.0.0.1,port=21380,state=wait_bgsave,offset=0,lag=0 master_replid:d821bc1350c8f6d122d616a14a1653b1d2ead3fb master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:100000000 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
CPU
used_cpu_sys:0.00 used_cpu_user:0.04 used_cpu_sys_children:0.00 used_cpu_user_children:0.00
Commandstats
cmdstat_hset:calls=118,usec=494,usec_per_call=4.19 cmdstat_select:calls=4,usec=4,usec_per_call=1.00 cmdstat_rpop:calls=20,usec=97,usec_per_call=4.85 cmdstat_type:calls=1362,usec=1291,usec_per_call=0.95 cmdstat_sinterstore:calls=13,usec=54,usec_per_call=4.15 cmdstat_sadd:calls=130,usec=421,usec_per_call=3.24 cmdstat_zunionstore:calls=21,usec=317,usec_per_call=15.10 cmdstat_zadd:calls=120,usec=878,usec_per_call=7.32 cmdstat_zrem:calls=27,usec=127,usec_per_call=4.70 cmdstat_del:calls=99,usec=86,usec_per_call=0.87 cmdstat_lrem:calls=31,usec=153,usec_per_call=4.94 cmdstat_sdiffstore:calls=5,usec=20,usec_per_call=4.00 cmdstat_zinterstore:calls=13,usec=81,usec_per_call=6.23 cmdstat_lpush:calls=119,usec=467,usec_per_call=3.92 cmdstat_randomkey:calls=215,usec=340,usec_per_call=1.58 cmdstat_set:calls=102,usec=323,usec_per_call=3.17 cmdstat_rpush:calls=19,usec=55,usec_per_call=2.89 cmdstat_replconf:calls=2,usec=2,usec_per_call=1.00 cmdstat_ping:calls=2,usec=1,usec_per_call=0.50 cmdstat_lpop:calls=19,usec=87,usec_per_call=4.58 cmdstat_sunionstore:calls=8,usec=51,usec_per_call=6.38 cmdstat_config:calls=4,usec=10,usec_per_call=2.50 cmdstat_hdel:calls=41,usec=182,usec_per_call=4.44 cmdstat_srem:calls=39,usec=162,usec_per_call=4.15
Cluster
cluster_enabled:0
Keyspace
db9:keys=110,expires=0,avg_ttl=0 db11:keys=102,expires=0,avg_ttl=0 db12:keys=122,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------ id=3 addr=127.0.0.1:33968 fd=7 name= age=1 idle=0 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config id=4 addr=127.0.0.1:33978 fd=8 name= age=0 idle=0 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hdel id=5 addr=127.0.0.1:33980 fd=9 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=type id=6 addr=127.0.0.1:33982 fd=10 name= age=0 idle=0 flags=N db=12 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd id=7 addr=127.0.0.1:45427 fd=11 name= age=0 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=psync
------ CURRENT CLIENT INFO ------ id=7 addr=127.0.0.1:45427 fd=11 name= age=0 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=psync argv[0]: 'PSYNC' argv[1]: '02adb4524db0b1bf12aa62f763453153bdb6229e' argv[2]: '1'
------ REGISTERS ------ 23529:M 15 Dec 12:11:01.067 # RAX:0000000000000000 RBX:00000000004f1b15 RCX:00007fa22bdef80d RDX:0000000000000000 RDI:0000000000000000 RSI:00007fa22c0c5a10 RBP:0000000000000dfc RSP:00007fffbe4011d0 R8 :00007fa22ca02780 R9 :000000000000004e R10:0000000000000030 R11:0000000000000000 R12:0000000000000000 R13:000000000000000b R14:0000000000000000 R15:000000000000000f RIP:00000000004666a7 EFL:0000000000010206 CSGSFS:002b000000000033 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011df) -> 00007fa22ba00188 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011de) -> 0000000006000000 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011dd) -> 00007fffbe4012a0 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011dc) -> 00007fffbe401330 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011db) -> 0000000000200000 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011da) -> 0000000006000000 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d9) -> 00000000004d2a9e 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d8) -> 00007fa22ba00100 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d7) -> 59524f4d454d2046 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d6) -> 4f2054554f20726f 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d5) -> 6620676e6974726f 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d4) -> 6261207369646552 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d3) -> 00007fffbe4012f0 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d2) -> 00007fffbe4013c0 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d1) -> 0000003000000018 23529:M 15 Dec 12:11:01.067 # (00007fffbe4011d0) -> 0000000006000000
------ FAST MEMORY TEST ------ 23529:M 15 Dec 12:11:01.067 # Bio thread for job type #0 terminated 23529:M 15 Dec 12:11:01.067 # Bio thread for job type #1 terminated 23529:M 15 Dec 12:11:01.067 # Bio thread for job type #2 terminated *** Preparing to test memory region 744000 (94208 bytes) *** Preparing to test memory region 85d000 (135168 bytes) *** Preparing to test memory region 7fa2239fe000 (8388608 bytes) *** Preparing to test memory region 7fa2241ff000 (8388608 bytes) *** Preparing to test memory region 7fa224a00000 (10485760 bytes) *** Preparing to test memory region 7fa22ba00000 (2097152 bytes) *** Preparing to test memory region 7fa22c0c5000 (20480 bytes) *** Preparing to test memory region 7fa22c2e2000 (16384 bytes) *** Preparing to test memory region 7fa22ca02000 (16384 bytes) *** Preparing to test memory region 7fa22ca0a000 (12288 bytes) *** Preparing to test memory region 7fa22ca0f000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: _serverPanic (base: 0x466590) Module: src/redis-server 127.0.0.1:21379 (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x466590 -D -b binary -m i386:x86-64 /tmp/dump.bin
23529:M 15 Dec 12:11:02.439 # dump of function (hexdump of 407 bytes): 5589f5534889fb4881ecd801000084c048898c24380100004c898424400100004c898c244801000074400f298424500100000f298c24600100000f299424700100000f299c24800100000f29a424900100000f29ac24a00100000f29b424b00100000f29bc24c0010000488d8424f0010000488d4c2408488d7c2420be000100004889442410488d842420010000c744240818000000c744240c300000004889442418e848b1fbff8b051ef12d0085c07505e8f9f8ffffbed8f24f00bf0300000031c0e89834fcffbe10f34f00bf0300000031c0e88734fcff488d5424204189e84889d9bee8025000bf0300000031c0e86b34fcffbe50f34f00bf0300000031c0e85a34fcffbed8f24f00bf0300000031c0e84934fcffc60425ffffffff784881c4d80100005b5dc30f1f8000000000415741564155415455534881ec4801000048c7070000000048c7470800000000c74710000000008b059beb2d0048897c241885c00f8ee108000048c744242800000000c7442424000000004c8db42494000000eb1e0f1f0083442424014883442428388b442424 Function at 0x429af0 is serverLog
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: ghost
This report indicates that redis does not have enough memory to continue when calling redisOutOfMemoryHandler (Guru Meditation: Redis aborting for OUT OF MEMORY). Which AWS instance are you using?
Comment From: hwware
I agree with @nashe , this looks like OOM issue indeed. Most likely your EC2 instance doesn't have enough memory.. @OscarGibson Since this is too old, can you close this issue? thanks
Comment From: oranagra
also note that it fails in createReplicationBacklog, maybe the config value of repl-backlog-size is set incorrectly to some insane value.
p.s. if you still have that full log file, look up one line above the portion you posted, and you'll see how much memory was attempted to be allocated.
meanwhile, i'm closing this issue since it doesn't indicate a bug in redis.
feel free to re-open if you feel otherwise.
Comment From: oranagra
ohh, seeing the title i now realize it was part of the test suite (so unlikely that repl-backlog-size was set incorrectly.
p.s. improved the crash report: #7670
Comment From: oranagra
total_system_memory:1041707008
total_system_memory_human:993.45M
That's very low.. although certainly more than enough for the test suite, I suppose the OS and other programs are consuming much of it, so that not much remains for Redis.