=== REDIS BUG REPORT START: Cut & paste starting from here ===
19415:M 22 Jun 11:12:57.911 # Redis 3.2.6 crashed by signal: 11
19415:M 22 Jun 11:12:57.911 # Crashed running the instuction at: 0x7ff6edea3bc7
19415:M 22 Jun 11:12:57.911 # Accessing address: 0x8
19415:M 22 Jun 11:12:57.911 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/usr/bin/redis-server 127.0.0.1:6379(listAddNodeTail+0x37)[0x7ff6edea3bc7]
Backtrace:
/usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x4a)[0x7ff6edee83ba]
/usr/bin/redis-server 127.0.0.1:6379(sigsegvHandler+0xc3)[0x7ff6edee8b83]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7ff6ed3130c0]
/usr/bin/redis-server 127.0.0.1:6379(listAddNodeTail+0x37)[0x7ff6edea3bc7]
/usr/bin/redis-server 127.0.0.1:6379(blockForKeys+0x8e)[0x7ff6edecf09e]
/usr/bin/redis-server 127.0.0.1:6379(blockingPopGenericCommand+0x1ea)[0x7ff6edecf8ba]
/usr/bin/redis-server 127.0.0.1:6379(call+0x91)[0x7ff6edeacb51]
/usr/bin/redis-server 127.0.0.1:6379(processCommand+0x3bf)[0x7ff6edeb007f]
/usr/bin/redis-server 127.0.0.1:6379(processInputBuffer+0x115)[0x7ff6edebd615]
/usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x12f)[0x7ff6edea662f]
/usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x7ff6edea69cb]
/usr/bin/redis-server 127.0.0.1:6379(main+0x440)[0x7ff6edea37a0]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7ff6ecf842b1]
/usr/bin/redis-server 127.0.0.1:6379(_start+0x2a)[0x7ff6edea395a]
------ INFO OUTPUT ------
# Server
redis_version:3.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:826601c992442478
redis_mode:standalone
os:Linux 3.14-2-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:6.2.1
process_id:19415
run_id:015ba6e7e1cee386cca08acffdd72becef194219
tcp_port:6379
uptime_in_seconds:2029564
uptime_in_days:23
hz:10
lru_clock:4946953
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
# Clients
connected_clients:43
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:23
# Memory
used_memory:29788056
used_memory_human:28.41M
used_memory_rss:34848768
used_memory_rss_human:33.23M
used_memory_peak:73056144
used_memory_peak_human:69.67M
total_system_memory:33665568768
total_system_memory_human:31.35G
used_memory_lua:65536
used_memory_lua_human:64.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.17
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:12645
rdb_bgsave_in_progress:0
rdb_last_save_time:1498119160
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:4885
total_commands_processed:1813698811
instantaneous_ops_per_sec:1187
total_net_input_bytes:54201470108
total_net_output_bytes:30335075233
instantaneous_input_kbps:34.42
instantaneous_output_kbps:7.86
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:21774
evicted_keys:0
keyspace_hits:15580647
keyspace_misses:2322615734
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:562
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:2205.76
used_cpu_user:9965.22
used_cpu_sys_children:299.04
used_cpu_user_children:5090.48
# Commandstats
cmdstat_get:calls=1009057,usec=2986589,usec_per_call=2.96
cmdstat_set:calls=124426,usec=376197,usec_per_call=3.02
cmdstat_setex:calls=73895,usec=368224,usec_per_call=4.98
cmdstat_del:calls=9699,usec=62498,usec_per_call=6.44
cmdstat_exists:calls=3309,usec=5012,usec_per_call=1.51
cmdstat_incr:calls=39,usec=161,usec_per_call=4.13
cmdstat_decr:calls=21,usec=81,usec_per_call=3.86
cmdstat_mget:calls=580568175,usec=870820618,usec_per_call=1.50
cmdstat_rpush:calls=580572074,usec=495708987,usec_per_call=0.85
cmdstat_lpop:calls=14,usec=17,usec_per_call=1.21
cmdstat_brpop:calls=16738103,usec=637989227,usec_per_call=38.12
cmdstat_llen:calls=1295,usec=800,usec_per_call=0.62
cmdstat_lrange:calls=12032135,usec=61811626,usec_per_call=5.14
cmdstat_lrem:calls=580570714,usec=700715207,usec_per_call=1.21
cmdstat_sadd:calls=4820,usec=10045,usec_per_call=2.08
cmdstat_srem:calls=14,usec=81,usec_per_call=5.79
cmdstat_scard:calls=43,usec=40,usec_per_call=0.93
cmdstat_smembers:calls=780002,usec=8152223,usec_per_call=10.45
cmdstat_zadd:calls=4482,usec=44841,usec_per_call=10.00
cmdstat_zrem:calls=1441,usec=12385,usec_per_call=8.59
cmdstat_zremrangebyscore:calls=3036,usec=17539,usec_per_call=5.78
cmdstat_zrangebyscore:calls=315273,usec=1938419,usec_per_call=6.15
cmdstat_zcount:calls=2,usec=9,usec_per_call=4.50
cmdstat_zcard:calls=105,usec=92,usec_per_call=0.88
cmdstat_hset:calls=100058,usec=468105,usec_per_call=4.68
cmdstat_hget:calls=302,usec=12167,usec_per_call=40.29
cmdstat_hmset:calls=3390,usec=28305,usec_per_call=8.35
cmdstat_hdel:calls=17,usec=163,usec_per_call=9.59
cmdstat_hgetall:calls=1783280,usec=10636749,usec_per_call=5.96
cmdstat_hscan:calls=20,usec=264,usec_per_call=13.20
cmdstat_incrby:calls=85,usec=517,usec_per_call=6.08
cmdstat_select:calls=1031,usec=1089,usec_per_call=1.06
cmdstat_expire:calls=114,usec=310,usec_per_call=2.72
cmdstat_scan:calls=218,usec=108466,usec_per_call=497.55
cmdstat_multi:calls=1012842,usec=622618,usec_per_call=0.61
cmdstat_exec:calls=5730506,usec=21878168,usec_per_call=3.82
cmdstat_info:calls=43,usec=8572,usec_per_call=199.35
cmdstat_eval:calls=595,usec=27362,usec_per_call=45.99
cmdstat_evalsha:calls=32253697,usec=6577586625,usec_per_call=203.93
cmdstat_command:calls=439,usec=137316,usec_per_call=312.79
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=112571,expires=25730,avg_ttl=621770493
hash_init_value: 1495857325
------ CLIENT LIST OUTPUT ------
id=4679 addr=127.0.0.1:53711 fd=15 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4680 addr=127.0.0.1:53712 fd=17 name= age=150089 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=4681 addr=127.0.0.1:53713 fd=18 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4876 addr=127.0.0.1:53189 fd=37 name= age=1072 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=4682 addr=127.0.0.1:53714 fd=19 name= age=150089 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4877 addr=127.0.0.1:53209 fd=42 name= age=1071 idle=169 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=4683 addr=127.0.0.1:53715 fd=20 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4684 addr=127.0.0.1:53716 fd=21 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4685 addr=127.0.0.1:53717 fd=22 name= age=150089 idle=2396 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=4878 addr=127.0.0.1:53245 fd=48 name= age=1030 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex
id=4686 addr=127.0.0.1:53718 fd=23 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4687 addr=127.0.0.1:53719 fd=24 name= age=150089 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=4885 addr=127.0.0.1:54591 fd=43 name= age=467 idle=171 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=4688 addr=127.0.0.1:53720 fd=25 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4689 addr=127.0.0.1:53721 fd=26 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4884 addr=127.0.0.1:54590 fd=39 name= age=469 idle=170 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex
id=4852 addr=127.0.0.1:44659 fd=41 name= age=60325 idle=615 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex
id=4861 addr=127.0.0.1:46153 fd=47 name= age=59665 idle=49437 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=4729 addr=127.0.0.1:56138 fd=44 name= age=149454 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4690 addr=127.0.0.1:53722 fd=27 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4691 addr=127.0.0.1:53723 fd=28 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4692 addr=127.0.0.1:53724 fd=29 name= age=150089 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4693 addr=127.0.0.1:53725 fd=30 name= age=150089 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4694 addr=127.0.0.1:53726 fd=31 name= age=150089 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4695 addr=127.0.0.1:53727 fd=32 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4696 addr=127.0.0.1:53728 fd=33 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4666 addr=127.0.0.1:53684 fd=5 name= age=150098 idle=150097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=4670 addr=127.0.0.1:53700 fd=16 name= age=150093 idle=3172 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=4667 addr=127.0.0.1:53690 fd=6 name= age=150097 idle=150097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=4671 addr=127.0.0.1:53701 fd=7 name= age=150092 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4672 addr=127.0.0.1:53704 fd=8 name= age=150089 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4848 addr=127.0.0.1:44351 fd=34 name= age=60470 idle=173 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=4673 addr=127.0.0.1:53705 fd=9 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4738 addr=127.0.0.1:60069 fd=36 name= age=148373 idle=774 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=eval
id=4674 addr=127.0.0.1:53706 fd=10 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4847 addr=127.0.0.1:44262 fd=40 name= age=60475 idle=173 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=4883 addr=127.0.0.1:54552 fd=38 name= age=472 idle=169 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=4882 addr=127.0.0.1:54526 fd=35 name= age=473 idle=169 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=4675 addr=127.0.0.1:53707 fd=11 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4676 addr=127.0.0.1:53708 fd=12 name= age=150089 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4677 addr=127.0.0.1:53709 fd=13 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
id=4886 addr=127.0.0.1:54727 fd=45 name= age=363 idle=170 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex
id=4678 addr=127.0.0.1:53710 fd=14 name= age=150089 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
------ CURRENT CLIENT INFO ------
id=4682 addr=127.0.0.1:53714 fd=19 name= age=150089 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=brpop
argv[0]: 'brpop'
argv[1]: 'resque:gitlab:queue:group_destroy'
argv[2]: 'resque:gitlab:queue:project_export'
argv[3]: 'resque:gitlab:queue:build'
argv[4]: 'resque:gitlab:queue:post_receive'
argv[5]: 'resque:gitlab:queue:project_cache'
argv[6]: 'resque:gitlab:queue:pipeline'
argv[7]: 'resque:gitlab:queue:gitlab_shell'
argv[8]: 'resque:gitlab:queue:merge'
argv[9]: 'resque:gitlab:queue:project_destroy'
argv[10]: 'resque:gitlab:queue:process_commit'
argv[11]: 'resque:gitlab:queue:repository_fork'
argv[12]: 'resque:gitlab:queue:cronjob'
argv[13]: 'resque:gitlab:queue:update_merge_requests'
argv[14]: 'resque:gitlab:queue:new_note'
argv[15]: 'resque:gitlab:queue:project_service'
argv[16]: 'resque:gitlab:queue:update_user_activity'
argv[17]: 'resque:gitlab:queue:use_key'
argv[18]: 'resque:gitlab:queue:emails_on_push'
argv[19]: 'resque:gitlab:queue:propagate_service_template'
argv[20]: 'resque:gitlab:queue:default'
argv[21]: 'resque:gitlab:queue:delete_merged_branches'
argv[22]: 'resque:gitlab:queue:mailers'
argv[23]: 'resque:gitlab:queue:repository_import'
argv[24]: 'resque:gitlab:queue:upload_checksum'
argv[25]: 'resque:gitlab:queue:authorized_projects'
argv[26]: 'resque:gitlab:queue:system_hook'
argv[27]: 'resque:gitlab:queue:pages'
argv[28]: 'resque:gitlab:queue:delete_user'
argv[29]: 'resque:gitlab:queue:repository_check'
argv[30]: 'resque:gitlab:queue:email_receiver'
argv[31]: 'resque:gitlab:queue:system_hook_push'
argv[32]: 'resque:gitlab:queue:project_web_hook'
argv[33]: 'resque:gitlab:queue:git_garbage_collect'
argv[34]: 'resque:gitlab:queue:expire_build_instance_artifacts'
argv[35]: 'resque:gitlab:queue:reactive_caching'
argv[36]: 'resque:gitlab:queue:irker'
argv[37]: '2'
------ REGISTERS ------
19415:M 22 Jun 11:12:57.932 #
RAX:00007ff6e6f2c420 RBX:00007ff6e7286220
RCX:0000000000000000 RDX:0000000000000017
RDI:00007ff6e6f2c420 RSI:00007ff6e6eea000
RBP:00007ff6e6eea000 RSP:00007fff21b80920
R8 :0000000000000027 R9 :00000000fffffff8
R10:00000000594b7c09 R11:00007ff6ed0cdd40
R12:00007ff6ec741a28 R13:00007ff6e7286220
R14:0000000000000001 R15:000552880d6e03a4
RIP:00007ff6edea3bc7 EFL:0000000000010206
CSGSFS:0000000000000033
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092f) -> 0000000000000000
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092e) -> 000000000000000f
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092d) -> 00007ff6e6eea000
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092c) -> 0000000000000001
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092b) -> ab1c4ef9e763d500
19415:M 22 Jun 11:12:57.932 # (00007fff21b8092a) -> 0000015ccedc8e78
19415:M 22 Jun 11:12:57.932 # (00007fff21b80929) -> 00007ff6edecf8ba
19415:M 22 Jun 11:12:57.932 # (00007fff21b80928) -> 0000000000000001
19415:M 22 Jun 11:12:57.932 # (00007fff21b80927) -> 0000000000000025
19415:M 22 Jun 11:12:57.932 # (00007fff21b80926) -> 0000000000000128
19415:M 22 Jun 11:12:57.932 # (00007fff21b80925) -> 0000000000000000
19415:M 22 Jun 11:12:57.932 # (00007fff21b80924) -> 00007ff6e6eea000
19415:M 22 Jun 11:12:57.932 # (00007fff21b80923) -> 00007ff6edecf09e
19415:M 22 Jun 11:12:57.932 # (00007fff21b80922) -> 00007ff6e6eea000
19415:M 22 Jun 11:12:57.932 # (00007fff21b80921) -> 00007ff6ec741a10
19415:M 22 Jun 11:12:57.932 # (00007fff21b80920) -> 00007ff6e87214c0
------ FAST MEMORY TEST ------
19415:M 22 Jun 11:12:57.932 # Bio thread for job type #0 terminated
19415:M 22 Jun 11:12:57.933 # Bio thread for job type #1 terminated
*** Preparing to test memory region 7ff6e6800000 (75497472 bytes)
*** Preparing to test memory region 7ff6eb3ff000 (8388608 bytes)
*** Preparing to test memory region 7ff6ebc00000 (8388608 bytes)
*** Preparing to test memory region 7ff6ec400000 (8388608 bytes)
*** Preparing to test memory region 7ff6ed2fe000 (16384 bytes)
*** Preparing to test memory region 7ff6ed51b000 (16384 bytes)
*** Preparing to test memory region 7ff6ed755000 (4096 bytes)
*** Preparing to test memory region 7ff6ede71000 (8192 bytes)
*** Preparing to test memory region 7ff6ede7e000 (12288 bytes)
*** Preparing to test memory region 7ff6ede83000 (4096 bytes)
*** Preparing to test memory region 7ff6ee16d000 (86016 bytes)
.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: listAddNodeTail (base: 0x7ff6edea3b90)
Module: /usr/bin/redis-server 127.0.0.1:6379 (base 0x7ff6ede84000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7ff6edea3b90 -D -b binary -m i386:x86-64 /tmp/dump.bin
------ DUMPING CODE AROUND EIP ------
Symbol: listAddNodeTail (base: 0x7ff6edea3b90)
Module: /usr/bin/redis-server 127.0.0.1:6379 (base 0x7ff6ede84000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x7ff6edea3b90 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
19415:M 22 Jun 11:12:58.136 # dump of function (hexdump of 183 bytes):
55534889fbbf180000004889f54883ec08e8ea0601004885c07455488b5328488968104885d27430488b4b0848c740080000000048890848894108488943084883c2014889d8488953284883c4085b5dc30f1f80000000004889430848890348c740080000000048c70000000000ebcf31c0ebd666906
62e0f1f840000000000415541544989d555534889fbbf180000004889f54189cc4883ec08e8600601004885c0747b4585e44c8968107542483b2b488b55004889
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: antirez
That looks interesting and there is enough data to try an investigation. Thanks.
Comment From: oranagra
looks like we have a non-empty list, but it's tail is NULL.
7ff6edea3b90: 55 push %rbp
7ff6edea3b91: 53 push %rbx
7ff6edea3b92: 48 89 fb mov %rdi,%rbx
7ff6edea3b95: bf 18 00 00 00 mov $0x18,%edi
7ff6edea3b9a: 48 89 f5 mov %rsi,%rbp
7ff6edea3b9d: 48 83 ec 08 sub $0x8,%rsp
7ff6edea3ba1: e8 ea 06 01 00 callq 0x7ff6edeb4290
7ff6edea3ba6: 48 85 c0 test %rax,%rax - test that malloc didn't return null
7ff6edea3ba9: 74 55 je 0x7ff6edea3c00
7ff6edea3bab: 48 8b 53 28 mov 0x28(%rbx),%rdx
7ff6edea3baf: 48 89 68 10 mov %rbp,0x10(%rax)
7ff6edea3bb3: 48 85 d2 test %rdx,%rdx - list->len == 0?
7ff6edea3bb6: 74 30 je 0x7ff6edea3be8
7ff6edea3bb8: 48 8b 4b 08 mov 0x8(%rbx),%rcx - rcx = list->tail
7ff6edea3bbc: 48 c7 40 08 00 00 00 movq $0x0,0x8(%rax) - node->next = NULL
7ff6edea3bc3: 00
7ff6edea3bc4: 48 89 08 mov %rcx,(%rax) - node->prev = list->tail
7ff6edea3bc7: 48 89 41 08 mov %rax,0x8(%rcx) - list->tail->next = node; (crash since list->tail==NULL)
7ff6edea3bcb: 48 89 43 08 mov %rax,0x8(%rbx)
7ff6edea3bcf: 48 83 c2 01 add $0x1,%rdx
7ff6edea3bd3: 48 89 d8 mov %rbx,%rax
7ff6edea3bd6: 48 89 53 28 mov %rdx,0x28(%rbx)
7ff6edea3bda: 48 83 c4 08 add $0x8,%rsp
7ff6edea3bde: 5b pop %rbx
7ff6edea3bdf: 5d pop %rbp
7ff6edea3be0: c3 retq
Comment From: antirez
@oranagra thank you for the investigation, and very happy to see that dumping the code around the crash shows to be helpful in the practice. A few random points:
- It does not look very likely that there is a bug in
adlist.c, never touched for years (if not recently in 4.0). - I wonder if this could be the result of accessing a list that was actually already freed, so that it looks corrupted but is actually no longer valid memory.
- Use case is apparently Resque. Hugely popular use case, strange that if there is an issue this did not happen before.
I'm checking the code around hypothesis number 2.
Comment From: antirez
I can't find anything, the dictionary entry is deleted once the list is empty in a single place in unblockClientWaitingData(). I'll recheck in the next days. Only odd thing (but I'm sure it is normal under Linux, just never noticed) is that in this dump the addresses of unrelated things are all in nearby offsets, like executable memory pages and heap are in contiguous addresses.
I checked past issues looking for crashes in the same code, there aren't AFAIK. There were a few crashes years ago about blockForKeys() but all related to adding the key in the dictionary, not manipulating the list.
Comment From: whyxxx
=== REDIS BUG REPORT START: Cut & paste starting from here ===
[49246] 06 Aug 10:08:24.005 # Redis 2.8.17 crashed by signal: 11
[49246] 06 Aug 10:08:24.005 # Failed assertion:
Comment From: zhipingu
Is this problem solved please? We have the same problem in 5.0.12
Comment From: zhipingu
I can't find anything, the dictionary entry is deleted once the list is empty in a single place in
unblockClientWaitingData(). I'll recheck in the next days. Only odd thing (but I'm sure it is normal under Linux, just never noticed) is that in this dump the addresses of unrelated things are all in nearby offsets, like executable memory pages and heap are in contiguous addresses.I checked past issues looking for crashes in the same code, there aren't AFAIK. There were a few crashes years ago about
blockForKeys()but all related to adding the key in the dictionary, not manipulating the list.
Is this problem solved please? We have the same problem in 5.0.12
Comment From: zhipingu
I can't find anything, the dictionary entry is deleted once the list is empty in a single place in
unblockClientWaitingData(). I'll recheck in the next days. Only odd thing (but I'm sure it is normal under Linux, just never noticed) is that in this dump the addresses of unrelated things are all in nearby offsets, like executable memory pages and heap are in contiguous addresses.I checked past issues looking for crashes in the same code, there aren't AFAIK. There were a few crashes years ago about
blockForKeys()but all related to adding the key in the dictionary, not manipulating the list.
My crash appears in line 336 of the function listRotateTailToHead(adlist.c:336).It's at list->tail->next = NULL. And I also saw that the value of list->tail is NULL, but the length of the list is 870 not 0. Looking forward to your reply!
Comment From: zhipingu
I suspect that other threads have freed the list->tail node
Comment From: oranagra
@zhipingu i don't think we were able to find the reason for this issue, but so much time passed since then, it's hard to tell. did it happen to you only once, or does ti reproduce? if it does, i'd like to learn how, or ask to check if it can reproduce with a more recent version (7.0).
Comment From: zhipingu
t does, i'd like to learn how, or ask to check if it can reproduce with a more recent v
@oranagra Thank you for your reply. This problem has crashed several times, and it has been very frequent recently, but it cannot be reproduced at present. We are working hard to reproduce it. And every crash has the same stack
Comment From: zhipingu
t does, i'd like to learn how, or ask to check if it can reproduce with a more recent v
@oranagra Thank you for your reply. This problem has crashed several times, and it has been very frequent recently, but it cannot be reproduced at present. We are working hard to reproduce it. And every crash has the same stack
Comment From: zhipingu
@zhipingu i don't think we were able to find the reason for this issue, but so much time passed since then, it's hard to tell. did it happen to you only once, or does ti reproduce? if it does, i'd like to learn how, or ask to check if it can reproduce with a more recent version (7.0). And these crashes are distributed on different machines