=== REDIS BUG REPORT START: Cut & paste starting from here ===
23961:S 24 Jun 04:30:56.348 # Redis 4.0.12 crashed by signal: 11
23961:S 24 Jun 04:30:56.348 # Crashed running the instruction at: 0x497ef8
23961:S 24 Jun 04:30:56.348 # Accessing address: 0x7f8ecaa00000
23961:S 24 Jun 04:30:56.348 # Failed assertion:
------ STACK TRACE ------ EIP: /usr/local/redis-server4/bin/redis-server *:6630(siphash+0x58)[0x497ef8]
Backtrace: /usr/local/redis-server4/bin/redis-server :6630(logStackTrace+0x3c)[0x46800c] /usr/local/redis-server4/bin/redis-server :6630(sigsegvHandler+0xa3)[0x469243] /lib64/libpthread.so.0[0x3c93a0f710] /usr/local/redis-server4/bin/redis-server :6630(siphash+0x58)[0x497ef8] /usr/local/redis-server4/bin/redis-server :6630(dictRehash+0x8d)[0x427cad] /usr/local/redis-server4/bin/redis-server :6630(dictAddRaw+0x1aa)[0x4287aa] /usr/local/redis-server4/bin/redis-server :6630(dictAdd+0x20)[0x428900] /usr/local/redis-server4/bin/redis-server :6630(dbAdd+0x6a)[0x44281a] /usr/local/redis-server4/bin/redis-server :6630(setKey+0x76)[0x442b26] /usr/local/redis-server4/bin/redis-server :6630(setGenericCommand+0xa4)[0x44cfa4] /usr/local/redis-server4/bin/redis-server :6630(setCommand+0xfa)[0x44d32a] /usr/local/redis-server4/bin/redis-server :6630(call+0x87)[0x429d57] /usr/local/redis-server4/bin/redis-server :6630(processCommand+0x395)[0x42e885] /usr/local/redis-server4/bin/redis-server :6630(processInputBuffer+0x10d)[0x43a92d] /usr/local/redis-server4/bin/redis-server :6630(readQueryFromClient+0x101)[0x43ab21] /usr/local/redis-server4/bin/redis-server :6630(aeProcessEvents+0x1d3)[0x425453] /usr/local/redis-server4/bin/redis-server :6630(aeMain+0x2b)[0x42577b] /usr/local/redis-server4/bin/redis-server :6630(main+0x500)[0x42dfc0] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3c9361ed5d] /usr/local/redis-server4/bin/redis-server :6630[0x422ae9]
------ INFO OUTPUT ------
Server^M
redis_version:4.0.12^M redis_git_sha1:afc4b36c^M redis_git_dirty:1^M redis_build_id:7b7e61a779d11458^M redis_mode:standalone^M os:Linux 4.9.0 x86_64^M arch_bits:64^M multiplexing_api:epoll^M atomicvar_api:sync-builtin^M gcc_version:4.4.7^M process_id:23961^M run_id:625043d6946b959b01122c3a99c8f17b7587f220^M tcp_port:6630^M uptime_in_seconds:41051645^M uptime_in_days:475^M hz:90^M lru_clock:15885942^M executable:/usr/local/redis-server4/bin/redis-server^M config_file:/usr/local/redis-server4/conf/6630.conf^M ^M
Clients^M
connected_clients:12^M client_longest_output_list:0^M client_biggest_input_buf:0^M blocked_clients:0^M ^M
Memory^M
used_memory:19520278216^M used_memory_human:18.18G^M used_memory_rss:23782514688^M used_memory_rss_human:22.15G^M used_memory_peak:24646208544^M used_memory_peak_human:22.95G^M used_memory_peak_perc:79.20%^M used_memory_overhead:7237428580^M used_memory_startup:2536968^M used_memory_dataset:12282849636^M used_memory_dataset_perc:62.93%^M total_system_memory:270444048384^M total_system_memory_human:251.87G^M used_cpu_user:324183.78^M used_cpu_sys_children:3805082.75^M used_cpu_user_children:14409088.00^M ^M
Commandstats^M
cmdstat_pexpire:calls=20557,usec=174616,usec_per_call=8.49^M cmdstat_bgsave:calls=1,usec=447139,usec_per_call=447139.00^M cmdstat_ping:calls=553472327,usec=686823957,usec_per_call=1.24^M cmdstat_setex:calls=5639667,usec=204356896,usec_per_call=36.24^M cmdstat_command:calls=3,usec=3439,usec_per_call=1146.33^M cmdstat_hset:calls=136829,usec=4152824,usec_per_call=30.35^M cmdstat_expireat:calls=346652324,usec=2280844160,usec_per_call=6.58^M cmdstat_replconf:calls=27508697,usec=46412714,usec_per_call=1.69^M cmdstat_publish:calls=108519810,usec=1439888026,usec_per_call=13.27^M cmdstat_multi:calls=2,usec=2,usec_per_call=1.00^M cmdstat_expire:calls=136829,usec=1087227,usec_per_call=7.95^M cmdstat_set:calls=422330833,usec=11719933753,usec_per_call=27.75^M cmdstat_client:calls=301880,usec=1308107,usec_per_call=4.33^M cmdstat_exec:calls=8,usec=59626,usec_per_call=7453.25^M cmdstat_hgetall:calls=1,usec=4,usec_per_call=4.00^M cmdstat_exists:calls=61664527,usec=524134175,usec_per_call=8.50^M cmdstat_del:calls=3575309,usec=57068735,usec_per_call=15.96^M cmdstat_get:calls=7895171283,usec=57602426975,usec_per_call=7.30^M cmdstat_select:calls=182,usec=276,usec_per_call=1.52^M cmdstat_incr:calls=2,usec=38,usec_per_call=19.00^M cmdstat_unlink:calls=90663485,usec=5339905816,usec_per_call=58.90^M cmdstat_slowlog:calls=443886,usec=49377006,usec_per_call=111.24^M cmdstat_psync:calls=3,usec=599985,usec_per_call=199995.00^M cmdstat_ttl:calls=5,usec=103,usec_per_call=20.60^M cmdstat_zremrangebyscore:calls=214283538,usec=2625948730,usec_per_call=12.25^M cmdstat_mset:calls=20557,usec=633536,usec_per_call=30.82^M cmdstat_info:calls=54392696,usec=6169897161,usec_per_call=113.43^M cmdstat_zadd:calls=228914122,usec=5055772865,usec_per_call=22.09^M cmdstat_config:calls=31963334,usec=1007950668,usec_per_call=31.53^M cmdstat_subscribe:calls=176288,usec=1596745,usec_per_call=9.06^M ^M
Cluster^M
cluster_enabled:0^M ^M
Keyspace^M
db0:keys=81026377,expires=76484657,avg_ttl=7354308897^M
------ CLIENT LIST OUTPUT ------ id=9091179 addr=10.27.17.46:57624 fd=177 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe id=9091202 addr=10.27.4.181:6630 fd=12 name= age=2626 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set id=9091180 addr=10.27.16.46:63792 fd=178 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe id=9091181 addr=10.27.16.45:65086 fd=179 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe id=9091184 addr=10.27.9.231:42281 fd=182 name= age=3123 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config id=9091002 addr=10.27.16.45:13062 fd=8 name=sentinel-716b7d31-cmd age=3393 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish id=9091003 addr=10.27.17.46:5374 fd=9 name=sentinel-3f3ead71-cmd age=3393 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=9091006 addr=10.27.17.44:54030 fd=17 name=sentinel-6c575b9d-cmd age=3393 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=9091173 addr=10.27.17.45:59781 fd=171 name= age=3123 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=9091174 addr=10.27.16.46:63036 fd=172 name= age=3123 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping id=9091177 addr=10.27.17.44:18053 fd=175 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe id=9091178 addr=10.27.17.45:60419 fd=176 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
------ CURRENT CLIENT INFO ------ id=9091202 addr=10.27.4.181:6630 fd=12 name= age=2626 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set argv[0]: 'SET' argv[1]: 'ops_sla_monitor_G1Mb6ifW' argv[2]: 'ops_sla_monitor' 23961:S 24 Jun 04:30:56.350 # key 'ops_sla_monitor_G1Mb6ifW' found in DB containing the following object: 23961:S 24 Jun 04:30:56.350 # Object type: 0 23961:S 24 Jun 04:30:56.350 # Object encoding: 8 23961:S 24 Jun 04:30:56.350 # Object refcount: 1 23961:S 24 Jun 04:30:56.350 # Object raw string len: 15 23961:S 24 Jun 04:30:56.350 # Object raw string content: "ops_sla_monitor"
------ REGISTERS ------ 23961:S 24 Jun 04:30:56.350 # RAX:1c5094fbe11a0eb5 RBX:00007f8ecd170400 RCX:0000000000000000 RDX:c913a70678eabc75 RDI:00007f8eca9ffff9 RSI:fffffffff2d15688 RBP:00007f8a832b3880 RSP:00007ffc2d0ae748 R14:0000000000000000 R15:0000000000000000 RIP:0000000000497ef8 EFL:0000000000010293 CSGSFS:002b000000000033 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae757) -> 00007f8959a63f48 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae756) -> 00007f8d413b2281 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae755) -> 00000000aad3ba51 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae754) -> 00007f8ecd170400 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae753) -> 00007f895ba00000 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae752) -> 00007f894663e340 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae751) -> e00434f666fd1e8f 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae750) -> 0000000037e8f478 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74f) -> 00007f8959a63f48 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74e) -> 00000000004287aa 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74d) -> 0000000000000000 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74c) -> 0000000000000000 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74b) -> 00007f8d413b2281 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74a) -> 00007f8ecd170400 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae749) -> 00007f8ecd170400 23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae748) -> 0000000000427cad
------ FAST MEMORY TEST ------ 23961:S 24 Jun 04:30:56.350 # Bio thread for job type #0 terminated 23961:S 24 Jun 04:30:56.350 # Bio thread for job type #1 terminated 23961:S 24 Jun 04:30:56.350 # Bio thread for job type #2 terminated *** Preparing to test memory region 74e000 (98304 bytes) *** Preparing to test memory region 2197000 (135168 bytes) *** Preparing to test memory region 3c93021000 (4096 bytes) *** Preparing to test memory region 3c9398f000 (20480 bytes) *** Preparing to test memory region 3c93c19000 (16384 bytes) *** Preparing to test memory region 7f88a4000000 (26417823744 bytes) *** Preparing to test memory region 7f8ecabfe000 (8388608 bytes) *** Preparing to test memory region 7f8ecb3ff000 (8388608 bytes) *** Preparing to test memory region 7f8ecbc00000 (10485760 bytes) *** Preparing to test memory region 7f8ecc800000 (8388608 bytes) *** Preparing to test memory region 7f8ecd000000 (2097152 bytes) *** Preparing to test memory region 7f8ed3200000 (2097152 bytes) *** Preparing to test memory region 7f8ed3561000 (16384 bytes) *** Preparing to test memory region 7f8ed3579000 (4096 bytes) *** Preparing to test memory region 7f8ed357a000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: siphash (base: 0x497ea0) Module: /usr/local/redis-server4/bin/redis-server *:6630 (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x497ea0 -D -b binary -m i386:x86-64 /tmp/dump.bin
23961:S 24 Jun 04:32:23.733 # dump of function (hexdump of 216 bytes): 4c8b42084989f2488b0a4983e2f848ba736574796264657448b86d6f646e61726f644e8d14174989f349b96172656e6567796c4c31c24c31c049b875657370656d6f7349c1e3384931c94931c84c39d7744b660f1f440000488b0f4e8d040048c1c8334883c7084c31c049c1c8204831ca4e8d0c0a48c1ca304c31ca4901c148c1c82f4e8d040248c1ca2b4c31c849c1c9204c31c24931c84939fa75bb83e607ff24f5787a5100660f1f840000000000410fb64a0648c1e1304909cb410fb64a0548c1e1284909cb410fb64a0448c1e1204909cb410fb64a
Comment From: madolson
Hey @fuzengjie, can you provide any more context on what your application was doing at the time or any ability to reproduce it? we've seen a couple of previous issues with similar crashes which we think our memory corruption related. (e.g. https://github.com/redis/redis/issues/5835 / https://github.com/redis/redis/issues/7358)
Do you know if the memory you are running with has ECC?