Crash report See Attachment redis_crash_report.log

Additional information

Operating System: Debian GNU/Linux 10 (buster) Kernel: Linux 4.19.0-9-2-amd64 Redis Version: 6.0.6-1~bpo10+1

The Redis server has been busy processing requests for 50days after which it crashed spitting out the above report. Issue was seen only once till date. redis_crash_report.log

Comment From: oranagra

@rajendra-dendukuri thanks. anything else you can tell us about what's leading to this crash? anything special about the system on which it runs? the redis binary the one supplied by the OS? maybe you can upgrade to a more recent version?

Comment From: rajendra-dendukuri

anything else you can tell us about what's leading to this crash?

There were a lot of transactions happening and that info can be seen in the stats section of the crash report. This issue was seen only once.

anything special about the system on which it runs?

It is a typical 4 core intel x86 CPU powered network switch. The redis DB is being used to store network traffic counters collected from the hardware. The redis-server is running inside a Debian 10 container running on top of a Debian10 host o/s.

the redis binary the one supplied by the OS?

Yes. The redis-server package is obtained from the Debian buster repository.

maybe you can upgrade to a more recent version?

We will most likely do this. But it we are interested to know what caused the crash. This will help us to be doubly sure that the latest version fixes it. Also because this issue was seen only once, we may not be in a position to confirm if the upgraded version fixes the crash.

@rajendra-dendukuri thanks. anything else you can tell us about what's leading to this crash? anything special about the system on which it runs? the redis binary the one supplied by the OS? maybe you can upgrade to a more recent version?

Comment From: oranagra

well, i don't have much to work with, so i kinda doubt we'll get to the bottom of it.

what i do see is that the crash log says:

EIP:
/usr/bin/redis-server 127.0.0.1:63796(processMultibulkBuffer+0x142)[0x556245c0f902]

the disassembly from the crash log:

    556245c0f8fc:       41 83 e4 07             and    $0x7,%r12d
    556245c0f900:       48 83 fe ff             cmp    $0xffffffffffffffff,%rsi
    556245c0f904:       0f 84 d6 01 00 00       je     0x556245c0fae0
    556245c0f90a:       41 80 fc 04             cmp    $0x4,%r12b
    556245c0f90e:       0f 87 76 03 00 00       ja     0x556245c0fc8a
    556245c0f914:       48 8d 0d 09 50 09 00    lea    0x95009(%rip),%rcx        # 0x556245ca4924

note that xxx902 is in the middle of a a multi-byte instruction, so somehow we jumped there :shrug:

Comment From: rajendra-dendukuri

@oranagra

I observed two more instances of crashes in other nodes running same application. See attachment.

  1. SIGSEGV (see attached redis_crash_report_2.log) redis_crash_report_2.log
#0  0x00007fa689187bc7 in kill () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055ecb807cb69 in sigsegvHandler ()
#2  <signal handler called>
#3  0x000055ecb8045980 in processMultibulkBuffer ()
#4  0x000055ecb804853a in processInputBuffer ()
#5  0x000055ecb80c91dc in ?? ()
#6  0x000055ecb802e13f in aeProcessEvents ()
#7  0x000055ecb802e4ad in aeMain ()
#8  0x000055ecb8028669 in main ()
  1. Stack smashing
[New LWP 116]
[New LWP 117]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/redis-server 127.0.0.1:63796                                         '.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f0d23df78eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f0d23c1bf40 (LWP 61))]
(gdb) bt
#0  0x00007f0d23df78eb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f0d23de2535 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f0d23e39648 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f0d23eca3ad in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f0d23eca362 in __stack_chk_fail ()
   from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00005587d5592e0c in beforeSleep ()
#6  0x00007fff579b34e0 in ?? ()
#7  0x00005587d558f001 in aeProcessEvents ()
#8  0x00005587d558f4ad in aeMain ()
#9  0x00005587d5589669 in main ()
(gdb) quit
root@sonic:/#

Oct 29 08:00:32.311227+00:00 2023 testhost INFO database#/supervisord: redis6 *** stack smashing detected ***: terminated

Comment From: rajendra-dendukuri

@oranagra any observations?

Comment From: oranagra

i'm sorry, i don't see anything i can help with. i suspect some environmental issue, like bad build, or an issue with the platform / OS / configuration.