=== REDIS BUG REPORT START: Cut & paste starting from here === [905] 11 Dec 22:17:07.091 # === ASSERTION FAILED CLIENT CONTEXT === [905] 11 Dec 22:17:07.091 # client->flags = 0 [905] 11 Dec 22:17:07.091 # client->fd = 11 [905] 11 Dec 22:17:07.091 # client->argc = 0 [905] 11 Dec 22:17:07.091 # === ASSERTION FAILED === [905] 11 Dec 22:17:07.091 # ==> scripting.c:957 'script != NULL' is not true [905] 11 Dec 22:17:07.091 # (forcing SIGSEGV to print the bug report.) [905] 11 Dec 22:17:07.092 # Redis 2.8.1 crashed by signal: 11 [905] 11 Dec 22:17:07.092 # Failed assertion: script != NULL (scripting.c:957) [905] 11 Dec 22:17:07.092 # --- STACK TRACE /usr/local/bin/redis-server :6379(logStackTrace+0x3e)[0x44654e] /usr/local/bin/redis-server :6379(_redisAssert+0x6f)[0x4455ef] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f9d82aa5cb0] /usr/local/bin/redis-server :6379(_redisAssert+0x6f)[0x4455ef] /usr/local/bin/redis-server :6379(evalGenericCommand+0x463)[0x44b383] /usr/local/bin/redis-server :6379(call+0x70)[0x41b9f0] /usr/local/bin/redis-server :6379(processCommand+0x33d)[0x41dc2d] /usr/local/bin/redis-server :6379(processInputBuffer+0x4f)[0x42627f] /usr/local/bin/redis-server :6379(readQueryFromClient+0xb8)[0x4263a8] /usr/local/bin/redis-server :6379(aeProcessEvents+0x145)[0x416f45] /usr/local/bin/redis-server :6379(aeMain+0x2b)[0x41721b] /usr/local/bin/redis-server :6379(main+0x325)[0x415fc5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f9d826f776d] /usr/local/bin/redis-server :6379[0x4160e9] [905] 11 Dec 22:17:07.092 # --- INFO OUTPUT [905] 11 Dec 22:17:07.093 # # Server redis_version:2.8.1 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:58928392fc5f613f redis_mode:standalone os:Linux 3.8.0-33-generic x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.6.3 process_id:905 run_id:b0564bd13fecc738853b31dd83f12f0be6ba9f86 tcp_port:6379 uptime_in_seconds:1230177 uptime_in_days:14 hz:10 lru_clock:267624 config_file:/etc/redis/6379.conf
Clients
connected_clients:15 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0
Memory
used_memory:1522118384 used_memory_human:1.42G used_memory_rss:1936465920 used_memory_peak:1714715832 used_memory_peak_human:1.60G used_memory_lua:324853760 mem_fragmentation_ratio:1.27 mem_allocator:jemalloc-3.2.0
Persistence
loading:0 rdb_changes_since_last_save:50435219 rdb_bgsave_in_progress:0 rdb_last_save_time:1385569130 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:25 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_current_size:759752425 aof_base_size:523400674 aof_pending_rewrite:0 aof_buffer_length:414 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:1667
Stats
total_connections_received:77366 total_commands_processed:306409890 instantaneous_ops_per_sec:307 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 evicted_keys:0 keyspace_hits:161995378 keyspace_misses:33271524 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:27184
Replication
role:master connected_slaves:1 slave0:ip=10.250.50.68,port=6379,state=online,offset=6403654343,lag=62 master_repl_offset:6403654343 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:6402605768 repl_backlog_histlen:1048576
CPU
used_cpu_sys:7122.81 used_cpu_user:194022.72 used_cpu_sys_children:14.94 used_cpu_user_children:124.90
Commandstats
cmdstat_get:calls=2488511,usec=11366680,usec_per_call=4.57 cmdstat_set:calls=1915,usec=9033,usec_per_call=4.72 cmdstat_del:calls=417399,usec=2174127,usec_per_call=5.21 cmdstat_exists:calls=4323560,usec=9792155,usec_per_call=2.26 cmdstat_incr:calls=898882,usec=4468426,usec_per_call=4.97 cmdstat_rpush:calls=4518261,usec=16866222,usec_per_call=3.73 cmdstat_lpush:calls=5685659,usec=10675324,usec_per_call=1.88 cmdstat_linsert:calls=4109426,usec=12546665,usec_per_call=3.05 cmdstat_rpop:calls=6079402,usec=25117564,usec_per_call=4.13 cmdstat_llen:calls=10036370,usec=30979824,usec_per_call=3.09 cmdstat_lrange:calls=36667146,usec=70528277168,usec_per_call=1923.47 cmdstat_lrem:calls=5474,usec=26090,usec_per_call=4.77 cmdstat_sadd:calls=21585513,usec=86187809,usec_per_call=3.99 cmdstat_srem:calls=4084590,usec=16749349,usec_per_call=4.10 cmdstat_sismember:calls=2245369,usec=8345396,usec_per_call=3.72 cmdstat_scard:calls=2156125,usec=3787218,usec_per_call=1.76 cmdstat_sinterstore:calls=313306,usec=3967668,usec_per_call=12.66 cmdstat_sunionstore:calls=12842,usec=4472713,usec_per_call=348.29 cmdstat_smembers:calls=11521120,usec=32468814858,usec_per_call=2818.20 cmdstat_hget:calls=7728019,usec=38172177,usec_per_call=4.94 cmdstat_hmset:calls=4050986,usec=44152370,usec_per_call=10.90 cmdstat_hmget:calls=3716812,usec=19924871,usec_per_call=5.36 cmdstat_hdel:calls=29,usec=158,usec_per_call=5.45 cmdstat_hgetall:calls=113977323,usec=745205158,usec_per_call=6.54 cmdstat_select:calls=28312562,usec=125544166,usec_per_call=4.43 cmdstat_expire:calls=8,usec=91,usec_per_call=11.38 cmdstat_keys:calls=38,usec=16179679,usec_per_call=425781.03 cmdstat_ping:calls=1,usec=3,usec_per_call=3.00 cmdstat_multi:calls=496788,usec=1610669,usec_per_call=3.24 cmdstat_exec:calls=496788,usec=26805513,usec_per_call=53.96 cmdstat_psync:calls=1,usec=573,usec_per_call=573.00 cmdstat_replconf:calls=1178489,usec=5482883,usec_per_call=4.65 cmdstat_flushdb:calls=24,usec=23705034,usec_per_call=987709.75 cmdstat_info:calls=77351,usec=23231287,usec_per_call=300.34 cmdstat_config:calls=77284,usec=999429,usec_per_call=12.93 cmdstat_watch:calls=3099,usec=8660,usec_per_call=2.79 cmdstat_unwatch:calls=9,usec=79,usec_per_call=8.78 cmdstat_restore:calls=629662,usec=4131289,usec_per_call=6.56 cmdstat_dump:calls=620673,usec=3441884,usec_per_call=5.55 cmdstat_eval:calls=527,usec=5433368,usec_per_call=10310.00 cmdstat_evalsha:calls=27815480,usec=161944427978,usec_per_call=5822.10 cmdstat_script:calls=77067,usec=2902140,usec_per_call=37.66
Keyspace
db10:keys=3131863,expires=1,avg_ttl=0 hash_init_value: 1385428175
[905] 11 Dec 22:17:07.093 # --- CLIENT LIST OUTPUT [905] 11 Dec 22:17:07.093 # addr=10.250.50.68:44687 fd=7 name= age=1227435 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=replconf addr=10.250.50.3:52080 fd=6 name= age=19851 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=smembers addr=10.250.50.3:58948 fd=10 name= age=10684 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha addr=10.250.50.3:49626 fd=12 name= age=1472 idle=1472 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53079 fd=8 name= age=119 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha addr=10.250.50.1:63807 fd=11 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=10 oll=0 omem=0 events=rw cmd=config addr=10.250.50.2:65095 fd=13 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.2:65097 fd=14 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53459 fd=15 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=info addr=10.250.50.3:53464 fd=16 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53469 fd=17 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53472 fd=18 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53497 fd=19 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha addr=10.250.50.3:53498 fd=20 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.3:53501 fd=21 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config addr=10.250.50.1:63852 fd=22 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=config
[905] 11 Dec 22:17:07.093 # --- REGISTERS [905] 11 Dec 22:17:07.093 # RAX:0000000000000000 RBX:00000000000003bd RCX:00007f9d82a8f778 RDX:00007f9d82a8f770 RDI:00007f9d82a8f720 RSI:0000000000000000 RBP:00000000004a6ca7 RSP:00007fffbe5e2f00 R8 :00007f9d82a8f770 R9 :0000000000000001 R10:0000000000000000 R11:0000000000000206 R12:00000000004a6ce6 R13:00007fffbe5e2f30 R14:0000000000000000 R15:0000000002d1cabc RIP:00000000004455ef EFL:0000000000010206 CSGSFS:0000000000000033 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0f) -> 0000000000000007 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0e) -> 00007f9d2a0d2000 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0d) -> 7d319f7171df5100 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0c) -> 0000000000000000 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0b) -> 00007f9d2a003735 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f0a) -> 3335626234623736 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f09) -> 6539386334613364 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f08) -> 3162643464643531 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f07) -> 3334653165636638 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f06) -> 3438383930335f66 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f05) -> 0000000000000001 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f04) -> 00007f9d81c10e50 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f03) -> 000000000044b383 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f02) -> 0000000000000001 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f01) -> 0000000000000000 [905] 11 Dec 22:17:07.093 # (00007fffbe5e2f00) -> 00007f9d2a0d2000 [905] 11 Dec 22:17:07.093 # --- FAST MEMORY TEST [905] 11 Dec 22:17:07.093 # Bio thread for job type #0 terminated [905] 11 Dec 22:17:07.094 # Bio thread for job type #1 terminated [905] 11 Dec 22:17:31.859 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible. [905] 11 Dec 22:17:31.859 # === REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to veryfy it.
Comment From: antirez
Hello, do you have some information about the EVAL call you make? Thanks.
Comment From: antirez
I investigated the bug a bit. If you check the assertion you can see:
client->argc = 0
So the client structure is broken when the EVALSHA should be replicated, like if the client was freed or alike. However it is not clear in the code path how this could happen at all. Apparently since the client structure was corrupted in some way, dictFetchValue fails to find the source of the script since argv[1] is already corrupted.
When we enter the evalGenericCommand function we are sure that client->argc is non zero and argv[1] has the script, so for this issue to be caused by a real bug, there must be a function call using the client structure that as a side effect freed / corrupted the client structure.
Still not sure how it is possible, so would be great if the hardware where this happened could be tested for memory / CPU errors with care.
Comment From: lucaswxp
I'm facing the same problem:
[23355] 05 Nov 13:38:47 # === ASSERTION FAILED ===
[23355] 05 Nov 13:38:47 # ==> networking.c:708 'c->argc == 0' is not true
[23355] 05 Nov 13:38:47 # (forcing SIGSEGV in order to print the stack trace)
[23355] 05 Nov 13:38:47 # ======= Ooops! Redis 2.2.12 got signal: -11- =======
[23355] 05 Nov 13:38:47 # redis_version:2.2.12
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:23355
uptime_in_seconds:33
uptime_in_days:0
lru_clock:1034672
used_cpu_sys:0.06
used_cpu_user:0.20
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:42
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:97
blocked_clients:2
used_memory:2552064
used_memory_human:2.43M
used_memory_rss:5009408
mem_fragmentation_ratio:1.96
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:199
bgsave_in_progress:0
last_save_time:1478353094
bgrewriteaof_in_progress:0
total_connections_received:42
total_commands_processed:500
expired_keys:0
evicted_keys:0
keyspace_hits:371
keyspace_misses:105
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:1
vm_enabled:0
role:master
db0:keys=14,expires=1
Comment From: enjoy-binbin
redis2 is too old, i am closing it...