Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context. 13528:M 21 Dec 2020 05:55:27.516 . --- Processing packet of type 1, 2568 bytes 13528:M 21 Dec 2020 05:55:27.516 . pong packet received: 0x7f67c4660800 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP 1ae0679481fd4597e4aab8e6e2404151f0af0161 172.16.223.196:7001@17001 master 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP a9945aeffc76d4c248b35cc389b59a0f03db5dc3 172.16.223.196:7002@17002 slave 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP ca35d3a0213fefd07de52a37ff8d06bae56203ca 172.16.202.21:7002@17002 slave 13528:M 21 Dec 2020 05:55:27.516 . --- Processing packet of type 1, 2568 bytes 13528:M 21 Dec 2020 05:55:27.516 . pong packet received: 0x7f67c465fe00 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP 2a4a953d192373f768b4b1e27d7250d2ac7882ee 172.16.194.129:7002@17002 slave 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP 6e3ee383fb04e163d8d52cd52deb439e8487caf8 172.16.223.196:7003@17003 slave 13528:M 21 Dec 2020 05:55:27.516 . GOSSIP a3de4187354a7c42efa488b5dbc8839dbd6c9a3c 172.16.202.21:7001@17001 master 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38370 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38376 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38382 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38388 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38394 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38400 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38406 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38412 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38418 13528:M 21 Dec 2020 05:55:27.621 - Accepted 172.31.48.10:38424 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38430 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38436 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38442 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38448 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38454 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38460 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38466 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38472 13528:M 21 Dec 2020 05:55:27.622 - Accepted 172.31.48.10:38478 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38514 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38520 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38526 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38532 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38538 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38544 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38550 13528:M 21 Dec 2020 05:55:27.672 - Accepted 172.31.48.10:38556 13528:M 21 Dec 2020 05:55:27.673 - Accepted 172.31.48.10:38562 13528:M 21 Dec 2020 05:55:27.917 . Pinging node ca35d3a0213fefd07de52a37ff8d06bae56203ca 13528:M 21 Dec 2020 05:55:27.923 . --- Processing packet of type 1, 2568 bytes 13528:M 21 Dec 2020 05:55:27.923 . pong packet received: 0x7f67c465f400 13528:M 21 Dec 2020 05:55:27.923 . GOSSIP a3de4187354a7c42efa488b5dbc8839dbd6c9a3c 172.16.202.21:7001@17001 master 13528:M 21 Dec 2020 05:55:27.923 . GOSSIP a9945aeffc76d4c248b35cc389b59a0f03db5dc3 172.16.223.196:7002@17002 slave 13528:M 21 Dec 2020 05:55:27.923 . GOSSIP 6e3ee383fb04e163d8d52cd52deb439e8487caf8 172.16.223.196:7003@17003 slave 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.209 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.210 - Client closed connection 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38612 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38618 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38624 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38630 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38636 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38642 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38648 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38654 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38672 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38678 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38660 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38684 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38666 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38690 13528:M 21 Dec 2020 05:55:28.211 - Accepted 172.31.48.10:38696 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38702 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38708 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38714 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38720 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38726 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38732 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38738 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38744 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38750 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38756 13528:M 21 Dec 2020 05:55:28.261 - Accepted 172.31.48.10:38762 13528:M 21 Dec 2020 05:55:28.588 . --- Processing packet of type 0, 2568 bytes 13528:M 21 Dec 2020 05:55:28.588 . Ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.588 . ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.588 . GOSSIP ca35d3a0213fefd07de52a37ff8d06bae56203ca 172.16.202.21:7002@17002 slave 13528:M 21 Dec 2020 05:55:28.589 . GOSSIP 1ae0679481fd4597e4aab8e6e2404151f0af0161 172.16.223.196:7001@17001 master 13528:M 21 Dec 2020 05:55:28.589 . GOSSIP 29a98707188455528595a10719d4169d16d234df 172.16.194.129:7001@17001 master 13528:M 21 Dec 2020 05:55:28.630 . --- Processing packet of type 0, 2568 bytes 13528:M 21 Dec 2020 05:55:28.630 . Ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.630 . ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.630 . GOSSIP 1ae0679481fd4597e4aab8e6e2404151f0af0161 172.16.223.196:7001@17001 master 13528:M 21 Dec 2020 05:55:28.630 . GOSSIP a3de4187354a7c42efa488b5dbc8839dbd6c9a3c 172.16.202.21:7001@17001 master 13528:M 21 Dec 2020 05:55:28.630 . GOSSIP 29a98707188455528595a10719d4169d16d234df 172.16.194.129:7001@17001 master 13528:M 21 Dec 2020 05:55:28.658 . --- Processing packet of type 0, 2568 bytes 13528:M 21 Dec 2020 05:55:28.658 . Ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.658 . ping packet received: (nil) 13528:M 21 Dec 2020 05:55:28.658 . GOSSIP 2a4a953d192373f768b4b1e27d7250d2ac7882ee 172.16.194.129:7002@17002 slave 13528:M 21 Dec 2020 05:55:28.658 . GOSSIP ca35d3a0213fefd07de52a37ff8d06bae56203ca 172.16.202.21:7002@17002 slave 13528:M 21 Dec 2020 05:55:28.658 . GOSSIP a9945aeffc76d4c248b35cc389b59a0f03db5dc3 172.16.223.196:7002@17002 slave 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.746 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.747 - Client closed connection 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38782 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38788 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38800 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38794 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38806 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38812 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38818 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38824 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38830 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38836 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38842 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38848 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38854 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38860 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38866 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38872 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38878 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38884 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38890 13528:M 21 Dec 2020 05:55:28.748 - Accepted 172.31.48.10:38896 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38902 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38908 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38914 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38920 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38926 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38932 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38938 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38944 13528:M 21 Dec 2020 05:55:28.798 - Accepted 172.31.48.10:38950 13528:M 21 Dec 2020 05:55:28.799 - Accepted 172.31.48.10:38956 13528:M 21 Dec 2020 05:55:28.799 - Accepted 172.31.48.10:38962 13528:M 21 Dec 2020 05:55:28.799 - Protocol error (expected $ but got something else) from client: id=361 addr=172.31.48.10:38770 fd=30 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.799 - Protocol error (expected $ but got something else) from client: id=362 addr=172.31.48.10:38776 fd=31 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.799 - Protocol error (expected $ but got something else) from client: id=369 addr=172.31.48.10:38818 fd=38 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.799 - Protocol error (expected $ but got something else) from client: id=365 addr=172.31.48.10:38800 fd=34 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.799 - Protocol error (expected $ but got something else) from client: id=366 addr=172.31.48.10:38794 fd=35 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=367 addr=172.31.48.10:38806 fd=36 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=368 addr=172.31.48.10:38812 fd=37 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=363 addr=172.31.48.10:38782 fd=32 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=372 addr=172.31.48.10:38836 fd=41 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=371 addr=172.31.48.10:38830 fd=40 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=364 addr=172.31.48.10:38788 fd=33 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=370 addr=172.31.48.10:38824 fd=39 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=391 addr=172.31.48.10:38950 fd=60 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=390 addr=172.31.48.10:38944 fd=59 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=387 addr=172.31.48.10:38926 fd=56 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=389 addr=172.31.48.10:38938 fd=58 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=380 addr=172.31.48.10:38884 fd=49 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=393 addr=172.31.48.10:38962 fd=62 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=382 addr=172.31.48.10:38896 fd=51 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=386 addr=172.31.48.10:38920 fd=55 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.800 - Protocol error (expected $ but got something else) from client: id=385 addr=172.31.48.10:38914 fd=54 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.801 - Protocol error (expected $ but got something else) from client: id=375 addr=172.31.48.10:38854 fd=44 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.801 - Protocol error (expected $ but got something else) from client: id=373 addr=172.31.48.10:38842 fd=42 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.801 - Protocol error (expected $ but got something else) from client: id=376 addr=172.31.48.10:38860 fd=45 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.801 - Protocol error (expected $ but got something else) from client: id=377 addr=172.31.48.10:38866 fd=46 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=69 qbuf-free=32699 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default. Query buffer during protocol error: '63n.VXK..' 13528:M 21 Dec 2020 05:55:28.801 - Client closed connection 13528:M 21 Dec 2020 05:55:28.801 - Client closed connection 13528:M 21 Dec 2020 05:55:28.801 - Client closed connection 13528:M 21 Dec 2020 05:55:28.801 - Client closed connection

=== REDIS BUG REPORT START: Cut & paste starting from here === 13528:M 21 Dec 2020 05:55:28.801 # === ASSERTION FAILED === 13528:M 21 Dec 2020 05:55:28.801 # ==> networking.c:1124 'ln != NULL' is not true 13528:M 21 Dec 2020 05:55:28.801 # (forcing SIGSEGV to print the bug report.) 13528:M 21 Dec 2020 05:55:28.801 # Redis 6.0.9 crashed by signal: 11, si_code: 1 13528:M 21 Dec 2020 05:55:28.801 # Crashed running the instruction at: 0x478125 13528:M 21 Dec 2020 05:55:28.802 # Accessing address: 0xffffffffffffffff 13528:M 21 Dec 2020 05:55:28.802 # Failed assertion: ln != NULL (networking.c:1124) ------ STACK TRACE ------ EIP: bin/redis-server 0.0.0.0:7001 cluster[0x478125]

Backtrace: bin/redis-server 0.0.0.0:7001 cluster[0x47a9de] bin/redis-server 0.0.0.0:7001 cluster[0x47b0b8] /lib64/libpthread.so.0(+0xf630)[0x7f67c4f45630] bin/redis-server 0.0.0.0:7001 cluster[0x478125] bin/redis-server 0.0.0.0:7001 cluster[0x444c44] bin/redis-server 0.0.0.0:7001 cluster[0x444f21] bin/redis-server 0.0.0.0:7001 cluster[0x4455c5] bin/redis-server 0.0.0.0:7001 cluster[0x434c67] bin/redis-server 0.0.0.0:7001 cluster[0x4318a5] bin/redis-server 0.0.0.0:7001 cluster[0x431abd] bin/redis-server 0.0.0.0:7001 cluster[0x42e4d7] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f67c4b8a555] bin/redis-server 0.0.0.0:7001 [cluster][0x42e77a]

------ INFO OUTPUT ------

Server

redis_version:6.0.9 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:8abd1a4bfb8621e4 redis_mode:cluster os:Linux 3.10.0-1062.12.1.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:9.3.1 process_id:13528 run_id:095207ee622c92dde4873f6f7287552e034689ad tcp_port:7001 uptime_in_seconds:229 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:14694608 executable:/usr/local/redis/bin/redis-server config_file:/u01/redis/conf/redis_7001.conf io_threads_active:1

Clients

connected_clients:10 client_recent_max_input_buffer:8 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:17603768 used_memory_human:16.79M used_memory_rss:49446912 used_memory_rss_human:47.16M used_memory_peak:19816736 used_memory_peak_human:18.90M used_memory_peak_perc:88.83% used_memory_overhead:6459064 used_memory_startup:3100040 used_memory_dataset:11144704 used_memory_dataset_perc:76.84% allocator_allocated:19630672 allocator_active:21721088 allocator_resident:28258304 total_system_memory:16260386816 total_system_memory_human:15.14G used_memory_lua:35840 used_memory_lua_human:35.00K used_memory_scripts:216 used_memory_scripts_human:216B number_of_cached_scripts:1 mem_not_counted_for_evict:2046 mem_replication_backlog:1048576 mem_clients_slaves:20512 mem_clients_normal:143544 mem_aof_buffer:2560 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:229693 rdb_bgsave_in_progress:0 rdb_last_save_time:1608529901 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:8896512 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0 aof_current_size:84640586 aof_base_size:75045018 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0

Stats

total_connections_received:390 total_commands_processed:473195 instantaneous_ops_per_sec:56490 total_net_input_bytes:13664977 total_net_output_bytes:19485920 instantaneous_input_kbps:2326.73 instantaneous_output_kbps:1575.72 rejected_connections:0 sync_full:1 sync_partial_ok:0 sync_partial_err:1 expired_keys:3 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:7 evicted_keys:0 keyspace_hits:31793 keyspace_misses:275 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:760 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_reads_processed:363916 total_writes_processed:405152 io_threaded_reads_processed:96793 io_threaded_writes_processed:140861

Replication

role:master connected_slaves:1 slave0:ip=172.16.223.196,port=7002,state=online,offset=9587207,lag=0 master_replid:7da68b8721ff04a4ea5581207526b704d1bc7978 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:9589484 master_repl_offset:9589484 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:8540909 repl_backlog_histlen:1048576

CPU

used_cpu_sys:4.654237 used_cpu_user:12.970479 used_cpu_sys_children:0.005984 used_cpu_user_children:0.064829

Modules

Commandstats

cmdstat_exists:calls=33,usec=49,usec_per_call=1.48 cmdstat_hscan:calls=46,usec=292,usec_per_call=6.35 cmdstat_psync:calls=1,usec=953,usec_per_call=953.00 cmdstat_incr:calls=36896,usec=29090,usec_per_call=0.79 cmdstat_lpop:calls=37376,usec=57597,usec_per_call=1.54 cmdstat_del:calls=58,usec=216,usec_per_call=3.72 cmdstat_auth:calls=390,usec=1237,usec_per_call=3.17 cmdstat_rpop:calls=37728,usec=44021,usec_per_call=1.17 cmdstat_set:calls=38250,usec=36520,usec_per_call=0.95 cmdstat_sadd:calls=34135,usec=24426,usec_per_call=0.72 cmdstat_client:calls=15,usec=22,usec_per_call=1.47 cmdstat_info:calls=219,usec=8951,usec_per_call=40.87 cmdstat_replconf:calls=233,usec=283,usec_per_call=1.21 cmdstat_eval:calls=58,usec=1494,usec_per_call=25.76 cmdstat_get:calls=31989,usec=24716,usec_per_call=0.77 cmdstat_config:calls=2,usec=26,usec_per_call=13.00 cmdstat_hset:calls=33,usec=40,usec_per_call=1.21 cmdstat_ping:calls=67185,usec=20614,usec_per_call=0.31 cmdstat_setex:calls=23,usec=165,usec_per_call=7.17 cmdstat_rpush:calls=41716,usec=41186,usec_per_call=0.99 cmdstat_lpush:calls=37612,usec=50521,usec_per_call=1.34 cmdstat_command:calls=3,usec=1948,usec_per_call=649.33 cmdstat_cluster:calls=16,usec=3445,usec_per_call=215.31

Cluster

cluster_enabled:1

Keyspace

db0:keys=25368,expires=25192,avg_ttl=4517562974

------ CLIENT LIST OUTPUT ------ id=27 addr=172.16.65.238:52984 fd=28 name= age=102 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20544 events=r cmd=hscan user=default id=19 addr=172.16.92.249:35922 fd=23 name= age=174 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default id=20 addr=172.16.67.24:50888 fd=25 name= age=173 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default id=28 addr=172.16.34.186:43286 fd=29 name= age=102 idle=72 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default id=25 addr=172.16.88.250:37496 fd=26 name= age=103 idle=71 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default id=26 addr=172.16.60.159:34578 fd=27 name= age=102 idle=72 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default id=9 addr=172.16.80.114:53616 fd=24 name= age=205 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=info user=default id=6 addr=172.16.223.196:19730 fd=22 name= age=227 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=replconf user=default id=365 addr=172.31.48.10:38800 fd=34 name= age=0 idle=0 flags=cA db=0 sub=0 psub=0 multi=-1 qbuf=9 qbuf-free=32759 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default id=368 addr=172.31.48.10:38812 fd=37 name= age=0 idle=0 flags=cA db=0 sub=0 psub=0 multi=-1 qbuf=9 qbuf-free=32759 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default id=387 addr=172.31.48.10:38926 fd=56 name= age=0 idle=0 flags=cA db=0 sub=0 psub=0 multi=-1 qbuf=9 qbuf-free=32759 argv-mem=36 obl=44 oll=0 omem=0 tot-mem=61508 events=r cmd=hset user=default

------ REGISTERS ------ 13528:M 21 Dec 2020 05:55:28.803 # RAX:0000000000000000 RBX:0000000000000464 RCX:0000000000bc02c0 RDX:00007f67c4f2f7b8 RDI:00007f67c4f2f760 RSI:0000000000000000 RBP:000000000054bd6f RSP:00007ffde4276ef0 R8 :0000000000000001 R9 :00007f67c5a76f80 R10:7562206568742074 R11:0000000000000206 R12:00000000005545a2 R13:0000000000000002 R14:0000000000000001 R15:0000000000bb1ba0 RIP:0000000000478125 EFL:0000000000010202 CSGSFS:0000000000000033 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276eff) -> 00007ffd00000000 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276efe) -> 00007f67b8d6d4d0 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276efd) -> 00000000004455c5 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276efc) -> 0000000000000001 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276efb) -> 00007f67b8d6eec8 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276efa) -> 000000000000000a 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef9) -> 0000000000444f21 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef8) -> 00007f67b9111b80 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef7) -> 00007ffde4276ff0 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef6) -> 0000000000000002 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef5) -> 8601000000000000 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef5) -> 8601000000000000 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef4) -> 0000000000bb1ba0 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef3) -> 0000000000444c44 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef2) -> 0000000000000001 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef1) -> 00007f67b9111b80 13528:M 21 Dec 2020 05:55:28.803 # (00007ffde4276ef0) -> 00007ffde4276ff0

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------ 13528:M 21 Dec 2020 05:55:28.804 # Bio thread for job type #0 terminated 13528:M 21 Dec 2020 05:55:28.804 # Bio thread for job type #1 terminated 13528:M 21 Dec 2020 05:55:28.804 # Bio thread for job type #2 terminated 13528:M 21 Dec 2020 05:55:28.804 # IO thread(tid:140083505121024) terminated 13528:M 21 Dec 2020 05:55:28.804 # IO thread(tid:140083496728320) terminated 13528:M 21 Dec 2020 05:55:28.804 # IO thread(tid:140083488335616) terminated *** Preparing to test memory region 5b8000 (2277376 bytes) *** Preparing to test memory region bb1000 (135168 bytes) *** Preparing to test memory region 7f679c000000 (135168 bytes) *** Preparing to test memory region 7f67a4000000 (135168 bytes) *** Preparing to test memory region 7f67ac000000 (135168 bytes) *** Preparing to test memory region 7f67b1dff000 (8388608 bytes) *** Preparing to test memory region 7f67b2600000 (8388608 bytes) *** Preparing to test memory region 7f67b2e00000 (19009536 bytes) *** Preparing to test memory region 7f67b8200000 (8388608 bytes) *** Preparing to test memory region 7f67b8a00000 (2097152 bytes) *** Preparing to test memory region 7f67b8c79000 (13107200 bytes) *** Preparing to test memory region 7f67b98fa000 (8388608 bytes) *** Preparing to test memory region 7f67ba0fb000 (8388608 bytes) *** Preparing to test memory region 7f67ba8fc000 (8388608 bytes) *** Preparing to test memory region 7f67bb0fd000 (8388608 bytes) *** Preparing to test memory region 7f67bb8fe000 (8388608 bytes) *** Preparing to test memory region 7f67bd100000 (9437184 bytes) *** Preparing to test memory region 7f67bda3d000 (2621440 bytes) *** Preparing to test memory region 7f67c4200000 (8388608 bytes) *** Preparing to test memory region 7f67c4f31000 (20480 bytes) *** Preparing to test memory region 7f67c514e000 (16384 bytes) *** Preparing to test memory region 7f67c5a75000 (24576 bytes) *** Preparing to test memory region 7f67c5a7f000 (4096 bytes) ** Preparing to test memory region 7f67c4f31000 (20480 bytes) *** Preparing to test memory region 7f67c514e000 (16384 bytes) *** Preparing to test memory region 7f67c5a75000 (24576 bytes) *** Preparing to test memory region 7f67c5a7f000 (4096 bytes) *** Preparing to test memory region 7f67c5a80000 (4096 bytes) *** Preparing to test memory region 7f67c5a83000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: _serverAssert (base: 0x4780c0) Module: bin/redis-server 0.0.0.0:7001 [cluster] (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x4780c0 -D -b binary -m i386:x86-64 /tmp/dump.bin

13528:M 21 Dec 2020 05:55:29.528 # dump of function (hexdump of 229 bytes): 8b059666140041544989fc554889f55389d385c07462bec8945500bf0300000031c0e8e9cffbff4d89e089d94889eabee1945500bf0300000031c0e8d0cffbffbe907c550031c0bf030000004c89253566140048892d36661400891d38661400e8abcffbffc60425ffffffff785b5d415cc3660f1f440000e8d3faffffeb97904154554889fd534881eca00000008b150866140085d20f84f4000000bec07c5500bf0300000031c031dbe861cffbff488b95b8000000befc94550031c0bf03000000e849cffbff488b7d08ba200000004889e6e838ce0400be11955500bf030000004889c2

Aditional information

OS distribution and version Linux ip-172-16-194-129.ap-south-1.compute.internal 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.7.1908 (Core)
Steps to reproduce (if any) 3.redis-version 6.0.9 redis-cluster 3master 4slave step: (1): this is redis new version,we want test the cluster ops and tps on the aws ec2 instance。we used 3 machines (each 4C 16G) but once we execute " redis-benchmark -q -h ... -p 7001 -c 50 -n 10000 --cluster" command they will crash like above logs. (2): we use same config to execute on different machines (not aws ec2),they would be ok,There will be no such problems。

Comment From: yossigo

Seems like an IO threading bug related to io-threads-do-reads and handling of protocol errors, where readQueryFromClient() actually ends up producing a reply from an IO thread. In the mean time a reasonable workaround is to disable io-threads-do-reads.

Comment From: bluerfish

thanks , i disabled parameter "io-threads-do-reads" , The above mistakes have never happened again。

Redis [CRASH] networking.c:1124 'ln != NULL' is not true