redis部署模式为sentinel，系统为中标麒麟，版本为V10-sp3（2212），redis部署版本7.0.12，在某一时间点，redis主节点master突然内存崩溃，redis进程被系统直接kill掉，排查服务器内存、cpu、io均未达到瓶颈，以下是redis报错日志截取如下：
=== REDIS BUG REPORT START: Cut & paste starting from here === 141955:M 28 Aug 2024 09:55:01.401 # Redis 7.0.12 crashed by signal: 11, si_code: 1 141955:M 28 Aug 2024 09:55:01.401 # Accessing address: 0x10 141955:M 28 Aug 2024 09:55:01.401 # Crashed running the instruction at: 0x4622dc
------ STACK TRACE ------ EIP: ./redis-server *:36379[0x4622dc]
Backtrace: /usr/lib64/libpthread.so.0(+0x134c0)[0x7fec81ebd4c0] ./redis-server :36379[0x4622dc] ./redis-server :36379(_writeToClient+0xab)[0x46400b] ./redis-server :36379(writeToClient+0x45)[0x464195] ./redis-server :36379(handleClientsWithPendingWrites+0x5c)[0x4643bc] ./redis-server :36379(handleClientsWithPendingWritesUsingThreads+0x1f5)[0x46ac35] ./redis-server :36379(beforeSleep+0x13c)[0x4481bc] ./redis-server :36379(aeProcessEvents+0x82)[0x444162] ./redis-server :36379(aeMain+0x1d)[0x44466d] ./redis-server :36379(main+0x32b)[0x44043b] /usr/lib64/libc.so.6(__libc_start_main+0xe7)[0x7fec81d17b27] ./redis-server :36379(_start+0x2a)[0x440b0a]
------ REGISTERS ------ 141955:M 28 Aug 2024 09:55:01.402 # RAX:0000000000000000 RBX:00007fec817fb000 RCX:0000000000000006 RDX:ffffffffffffe768 RDI:00007ffda3dc1940 RSI:00007fec81a5c8d8 RBP:00000000000084db RSP:00007ffda3dc1940 R8 :00007fec81800900 R9 :00007fec816012f8 R10:0000000000000002 R11:0000000000000002 R12:00007ffda3dc59b8 R13:0000000000000002 R14:0000000000000000 R15:0000000000000001 RIP:00000000004622dc EFL:0000000000010246 CSGSFS:002b000000000033 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194f) -> 0000000000005977 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194e) -> 00007fec6b30fed0 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194d) -> 0000000000018127 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194c) -> 00007fec6b2db750 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194b) -> 00000000000544b7 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194a) -> 00007fec6b3696d0 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1949) -> 0000000000000ada 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1948) -> 00007fec6fd677d0 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1947) -> 0000000000006e09 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1946) -> 00007fec704bfe90 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1945) -> 00000000000084db 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1944) -> 00007fec70636ed0 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1943) -> 0000000000000800 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1942) -> 00007fec817cc000 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1941) -> 0000000000000000 141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1940) -> 0000000000000000 通过GBD排查，发现崩溃点如下： (gdb)
0 0x00007fec81d2b757 in ?? ()
https://github.com/redis/docs/pull/1 0x00000000004a574c in sdslen (s=0x1
) at sds.h:89 https://github.com/redis/docs/pull/2 logCurrentClient () at debug.c:1818 https://github.com/redis/docs/pull/3 0x0000000000000000 in ?? () (gdb) info frame 0 Stack frame at 0x7ffda3dc12c0: rip = 0x7fec81d2b757; saved rip 0x4a574c called by frame at 0x7ffda3dc1300 Arglist at 0x7ffda3dc12b0, args: Locals at 0x7ffda3dc12b0, Previous frame's sp is 0x7ffda3dc12c0 Saved registers: rip at 0x7ffda3dc12b8 (gdb) list 11 * notice, this list of conditions and the following disclaimer in the 12 * documentation and/or other materials provided with the distribution. 13 * * Neither the name of Redis nor the names of its contributors may be used 14 * to endorse or promote products derived from this software without 15 * specific prior written permission. 16 * 17 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 18 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 19 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE 20 * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE (gdb) info frame 1 Stack frame at 0x7ffda3dc1300: rip = 0x4a574c in sdslen (sds.h:89); saved rip 0x0 inlined into frame 2, caller of frame at 0x7ffda3dc12c0 source language c. Arglist at unknown address. Locals at unknown address, Previous frame's sp is 0x7ffda3dc12c0 Saved registers: rip at 0x7ffda3dc12b8 (gdb) list 21 * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 22 * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 23 * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 24 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 25 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 26 * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 27 * POSSIBILITY OF SUCH DAMAGE. 28 */ 29 30 #include "server.h" (gdb) info frame 2 Stack frame at 0x7ffda3dc1300: rip = 0x4a574c in logCurrentClient (debug.c:1818); saved rip 0x0 called by frame at 0x7ffda3dc1308, caller of frame at 0x7ffda3dc1300 source language c. Arglist at 0x7ffda3dc12b8, args: Locals at 0x7ffda3dc12b8, Previous frame's sp is 0x7ffda3dc1300 Saved registers: rbx at 0x7ffda3dc12c8, rbp at 0x7ffda3dc12d0, r12 at 0x7ffda3dc12d8, r13 at 0x7ffda3dc12e0, r14 at 0x7ffda3dc12e8, r15 at 0x7ffda3dc12f0, rip at 0x7ffda3dc12f8 (gdb) list 31 #include "monotonic.h" 32 #include "cluster.h" 33 #include "slowlog.h" 34 #include "bio.h" 35 #include "latency.h" 36 #include "atomicvar.h" 37 #include "mt19937-64.h" 38 #include "functions.h" 39 #include "syscheck.h" 40 (gdb) info frame 3 Stack frame at 0x7ffda3dc1308: rip = 0x0; saved rip 0x0 caller of frame at 0x7ffda3dc1300 Arglist at 0x7ffda3dc12f8, args: Locals at 0x7ffda3dc12f8, Previous frame's sp is 0x7ffda3dc1308 Saved registers: rip at 0x7ffda3dc1300 (gdb) list 41 #includeComment From: sundb
@Kevin-zhao326 thanks for your report, i'll take a look. btw, it's better to use english to describe your question, thanks.
Comment From: Kevin-zhao326
@Kevin-zhao326 thanks for your report, i'll take a look. btw, it's better to use english to describe your question, thanks.
Okay, we will ask questions in English later. Thank you！
Comment From: sundb
@Kevin-zhao326 can you share the fully crash report?
Comment From: Kevin-zhao326
@Kevin-zhao326 can you share the fully crash report? @sundb There is a system message log that I have pasted below, and there is also a core file. Do you need it? If so, I can also send it to you！
message Aug 28 09:55:02 xhxRedissjjdA1 systemd[1]: session-14151.scope: Succeeded. Aug 28 09:55:03 xhxRedissjjdA1 systemd[1]: Started Process Core Dump (PID 1788019/UID 0). Aug 28 09:55:04 xhxRedissjjdA1 systemd-coredump[1788020]: Process 141955 (redis-server) of user 1001 dumped core.#012#012Stack trace of thread 141955:#012#0 0x00007fec81d2b757 kill (libc.so.6)#012#1 0x00000000004a574c bugReportEnd (redis-server)#012#2 0x00007fec81ebd4c0 n/a (libpthread.so.0)#012#3 0x00000000004622dc _writevToClient (redis-server)#012#4 0x000000000046400b _writeToClient (redis-server)#012#5 0x0000000000464195 writeToClient (redis-server)#012#6 0x00000000004643bc handleClientsWithPendingWrites (redis-server)#012#7 0x000000000046ac35 handleClientsWithPendingWritesUsingThreads (redis-server)#012#8 0x00000000004481bc beforeSleep (redis-server)#012#9 0x0000000000444162 aeProcessEvents (redis-server)#012#10 0x000000000044466d aeMain (redis-server)#012#11 0x000000000044043b main (redis-server)#012#12 0x00007fec81d17b27 __libc_start_main (libc.so.6)#012#13 0x0000000000440b0a _start (redis-server)#012#012Stack trace of thread 141959:#012#0 0x00007fec81eb8a0c pthread_cond_wait (libpthread.so.0)#012#1 0x000000000054c5ff background_thread_sleep (redis-server)#012#2 0x00007fec81eb2f1b n/a (libpthread.so.0)#012#3 0x00007fec81dea1c0 __clone (libc.so.6) Aug 28 09:55:04 xhxRedissjjdA1 systemd[1]: systemd-coredump@1-1788019-0.service: Succeeded. Aug 28 09:55:04 xhxRedissjjdA1 systemd[1]: session-c8.scope: Succeeded. Aug 28 09:55:04 xhxRedissjjdA1 systemd-logind[1002]: Removed session c8. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1024... Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped target Main User Target. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped target Basic System. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped target Paths. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped target Sockets. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped target Timers. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: grub-boot-success.timer: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: dbus.socket: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Closed D-Bus User Message Bus Socket. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: pipewire.socket: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Closed Multimedia System. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Reached target Shutdown. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: systemd-exit.service: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Started Exit the Session. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787829]: Reached target Exit the Session. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: user@1024.service: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1024. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1024... Aug 28 09:55:10 xhxRedissjjdA1 systemd[159089]: run-user-1024.mount: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1744]: run-user-1024.mount: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: run-user-1024.mount: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1787919]: run-user-1024.mount: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1024.service: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[141812]: run-user-1024.mount: Succeeded. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1024. Aug 28 09:55:10 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1024. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Stopped target Main User Target. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Stopped target Basic System. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Stopped target Paths. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Stopped target Sockets. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Stopped target Timers. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: dbus.socket: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Closed D-Bus User Message Bus Socket. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: pipewire.socket: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Closed Multimedia System. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Reached target Shutdown. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 995... Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: systemd-exit.service: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Started Exit the Session. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1787919]: Reached target Exit the Session. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: user@995.service: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 995. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/995... Aug 28 09:55:12 xhxRedissjjdA1 systemd[1744]: run-user-995.mount: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: run-user-995.mount: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[159089]: run-user-995.mount: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: user-runtime-dir@995.service: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/995. Aug 28 09:55:12 xhxRedissjjdA1 systemd[141812]: run-user-995.mount: Succeeded. Aug 28 09:55:12 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 995. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1002... Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped target Main User Target. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped target Basic System. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped target Paths. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped target Sockets. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped target Timers. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: grub-boot-success.timer: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: dbus.socket: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Closed D-Bus User Message Bus Socket. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: pipewire.socket: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Closed Multimedia System. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Reached target Shutdown. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: systemd-exit.service: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Started Exit the Session. Aug 28 09:55:14 xhxRedissjjdA1 systemd[141812]: Reached target Exit the Session. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: user@1002.service: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1002. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1002... Aug 28 09:55:14 xhxRedissjjdA1 systemd[1744]: run-user-1002.mount: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: run-user-1002.mount: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1002.service: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[159089]: run-user-1002.mount: Succeeded. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1002. Aug 28 09:55:14 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1002. Aug 28 09:55:18 xhxRedissjjdA1 systemd[1]: systemd-hostnamed.service: Succeeded. Aug 28 09:55:26 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Main process exited, code=exited, status=255/EXCEPTION Aug 28 09:55:26 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Failed with result 'exit-code'. Aug 28 09:55:46 xhxRedissjjdA1 sshd[1788064]: Accepted keyboard-interactive/pam for check from 10.1.228.232 port 21195 ssh2 Aug 28 09:55:46 xhxRedissjjdA1 systemd[1]: Created slice User Slice of UID 1024. Aug 28 09:55:46 xhxRedissjjdA1 systemd[1]: Starting User Runtime Directory /run/user/1024... Aug 28 09:55:46 xhxRedissjjdA1 systemd-logind[1002]: New session c12430 of user check. Aug 28 09:55:46 xhxRedissjjdA1 systemd[1]: Started User Runtime Directory /run/user/1024. Aug 28 09:55:46 xhxRedissjjdA1 systemd[1]: Starting User Manager for UID 1024... Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Configuration file /usr/lib/systemd/user/grub-boot-success.timer is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Configuration file /usr/lib/systemd/user/grub-boot-success.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Started Mark boot as successful after the user session has run 2 minutes. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Reached target Paths. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Reached target Timers. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Starting D-Bus User Message Bus Socket. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Listening on Multimedia System. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Listening on D-Bus User Message Bus Socket. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Reached target Sockets. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Reached target Basic System. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Reached target Main User Target. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1788074]: Startup finished in 71ms. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1]: Started User Manager for UID 1024. Aug 28 09:55:47 xhxRedissjjdA1 systemd[1]: Started Session c12430 of user check. Aug 28 09:55:47 xhxRedissjjdA1 dbus-daemon[979]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.286952' (uid=1024 pid=1788089 comm="/usr/bin/hostnamectl --transient ") Aug 28 09:55:47 xhxRedissjjdA1 systemd[1]: Starting Hostname Service... Aug 28 09:55:47 xhxRedissjjdA1 dbus-daemon[979]: [system] Successfully activated service 'org.freedesktop.hostname1' Aug 28 09:55:47 xhxRedissjjdA1 systemd[1]: Started Hostname Service. Aug 28 09:55:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Service RestartSec=30s expired, scheduling restart. Aug 28 09:55:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Scheduled restart job, restart counter is at 11173. Aug 28 09:55:57 xhxRedissjjdA1 systemd[1]: Stopped run kylin_kms_daemon at boot time. Aug 28 09:55:57 xhxRedissjjdA1 systemd[1]: Started run kylin_kms_daemon at boot time. Aug 28 09:55:58 xhxRedissjjdA1 systemd[1]: session-c12430.scope: Succeeded. Aug 28 09:55:58 xhxRedissjjdA1 systemd-logind[1002]: Session c12430 logged out. Waiting for processes to exit. Aug 28 09:55:58 xhxRedissjjdA1 systemd-logind[1002]: Removed session c12430. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1024... Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped target Main User Target. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped target Basic System. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped target Paths. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped target Sockets. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped target Timers. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: grub-boot-success.timer: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: dbus.socket: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Closed D-Bus User Message Bus Socket. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: pipewire.socket: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Closed Multimedia System. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Reached target Shutdown. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: systemd-exit.service: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Started Exit the Session. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1788074]: Reached target Exit the Session. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: user@1024.service: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1024. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1024... Aug 28 09:56:08 xhxRedissjjdA1 systemd[1744]: run-user-1024.mount: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: run-user-1024.mount: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[159089]: run-user-1024.mount: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1024.service: Succeeded. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1024. Aug 28 09:56:08 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1024. Aug 28 09:56:17 xhxRedissjjdA1 systemd[1]: systemd-hostnamed.service: Succeeded. Aug 28 09:56:55 xhxRedissjjdA1 sshd[1788184]: Accepted keyboard-interactive/pam for check from 10.1.228.232 port 44687 ssh2 Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Created slice User Slice of UID 1024. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Starting User Runtime Directory /run/user/1024... Aug 28 09:56:55 xhxRedissjjdA1 systemd-logind[1002]: New session c12431 of user check. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Started User Runtime Directory /run/user/1024. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Starting User Manager for UID 1024... Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Configuration file /usr/lib/systemd/user/grub-boot-success.timer is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Configuration file /usr/lib/systemd/user/grub-boot-success.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Started Mark boot as successful after the user session has run 2 minutes. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Reached target Paths. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Reached target Timers. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Starting D-Bus User Message Bus Socket. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Listening on Multimedia System. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Listening on D-Bus User Message Bus Socket. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Reached target Sockets. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Reached target Basic System. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Reached target Main User Target. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1788193]: Startup finished in 72ms. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Started User Manager for UID 1024. Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Started Session c12431 of user check. Aug 28 09:56:55 xhxRedissjjdA1 dbus-daemon[979]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.286974' (uid=1024 pid=1788208 comm="/usr/bin/hostnamectl --transient ") Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Starting Hostname Service... Aug 28 09:56:55 xhxRedissjjdA1 dbus-daemon[979]: [system] Successfully activated service 'org.freedesktop.hostname1' Aug 28 09:56:55 xhxRedissjjdA1 systemd[1]: Started Hostname Service. Aug 28 09:56:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Main process exited, code=exited, status=255/EXCEPTION Aug 28 09:56:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Failed with result 'exit-code'. Aug 28 09:57:07 xhxRedissjjdA1 lvm[1788267]: ppid=1788205, cmdline:vgdisplay -c Aug 28 09:57:07 xhxRedissjjdA1 lvm[1788267]: /run/lock/lvm/P_global:aux: open failed: Permission denied Aug 28 09:57:08 xhxRedissjjdA1 systemd[1]: session-c12431.scope: Succeeded. Aug 28 09:57:08 xhxRedissjjdA1 systemd-logind[1002]: Session c12431 logged out. Waiting for processes to exit. Aug 28 09:57:08 xhxRedissjjdA1 systemd-logind[1002]: Removed session c12431. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1024... Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped target Main User Target. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped target Basic System. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped target Paths. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped target Sockets. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped target Timers. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: grub-boot-success.timer: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: dbus.socket: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Closed D-Bus User Message Bus Socket. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: pipewire.socket: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Closed Multimedia System. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Reached target Shutdown. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: systemd-exit.service: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Started Exit the Session. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1788193]: Reached target Exit the Session. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: user@1024.service: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1024. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1024... Aug 28 09:57:18 xhxRedissjjdA1 systemd[1744]: run-user-1024.mount: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: run-user-1024.mount: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[159089]: run-user-1024.mount: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1024.service: Succeeded. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1024. Aug 28 09:57:18 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1024. Aug 28 09:57:25 xhxRedissjjdA1 systemd[1]: systemd-hostnamed.service: Succeeded. Aug 28 09:57:27 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Service RestartSec=30s expired, scheduling restart. Aug 28 09:57:27 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Scheduled restart job, restart counter is at 11174. Aug 28 09:57:27 xhxRedissjjdA1 systemd[1]: Stopped run kylin_kms_daemon at boot time. Aug 28 09:57:27 xhxRedissjjdA1 systemd[1]: Started run kylin_kms_daemon at boot time. Aug 28 09:57:48 xhxRedissjjdA1 sshd[1788303]: Accepted keyboard-interactive/pam for check from 10.1.228.232 port 43121 ssh2 Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Created slice User Slice of UID 1024. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Starting User Runtime Directory /run/user/1024... Aug 28 09:57:48 xhxRedissjjdA1 systemd-logind[1002]: New session c12432 of user check. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Started User Runtime Directory /run/user/1024. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Starting User Manager for UID 1024... Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Configuration file /usr/lib/systemd/user/grub-boot-success.timer is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Configuration file /usr/lib/systemd/user/grub-boot-success.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Started Mark boot as successful after the user session has run 2 minutes. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Reached target Paths. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Reached target Timers. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Starting D-Bus User Message Bus Socket. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Listening on Multimedia System. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Listening on D-Bus User Message Bus Socket. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Reached target Sockets. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Reached target Basic System. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Reached target Main User Target. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Started User Manager for UID 1024. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1788312]: Startup finished in 74ms. Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Started Session c12432 of user check. Aug 28 09:57:48 xhxRedissjjdA1 dbus-daemon[979]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.286996' (uid=1024 pid=1788329 comm="/usr/bin/hostnamectl --transient ") Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Starting Hostname Service... Aug 28 09:57:48 xhxRedissjjdA1 dbus-daemon[979]: [system] Successfully activated service 'org.freedesktop.hostname1' Aug 28 09:57:48 xhxRedissjjdA1 systemd[1]: Started Hostname Service. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Created slice User Slice of UID 995. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Starting User Runtime Directory /run/user/995... Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Started User Runtime Directory /run/user/995. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Starting User Manager for UID 995... Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Configuration file /usr/lib/systemd/user/grub-boot-success.timer is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Configuration file /usr/lib/systemd/user/grub-boot-success.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Condition check resulted in Mark boot as successful after the user session has run 2 minutes being skipped. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Reached target Paths. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Reached target Timers. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Starting D-Bus User Message Bus Socket. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Listening on Multimedia System. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Listening on D-Bus User Message Bus Socket. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Reached target Sockets. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Reached target Basic System. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Reached target Main User Target. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1788396]: Startup finished in 72ms. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Started User Manager for UID 995. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: Started Session 14156 of user pcp. Aug 28 09:58:01 xhxRedissjjdA1 systemd[1]: session-14156.scope: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 995... Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Stopped target Main User Target. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Stopped target Basic System. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Stopped target Paths. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Stopped target Sockets. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Stopped target Timers. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: dbus.socket: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Closed D-Bus User Message Bus Socket. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: pipewire.socket: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Closed Multimedia System. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Reached target Shutdown. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: systemd-exit.service: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Started Exit the Session. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788396]: Reached target Exit the Session. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: user@995.service: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 995. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/995... Aug 28 09:58:11 xhxRedissjjdA1 systemd[1788312]: run-user-995.mount: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: run-user-995.mount: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[159089]: run-user-995.mount: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: user-runtime-dir@995.service: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/995. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1744]: run-user-995.mount: Succeeded. Aug 28 09:58:11 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 995. Aug 28 09:58:13 xhxRedissjjdA1 systemd[1]: session-c12432.scope: Succeeded. Aug 28 09:58:13 xhxRedissjjdA1 systemd-logind[1002]: Session c12432 logged out. Waiting for processes to exit. Aug 28 09:58:13 xhxRedissjjdA1 systemd-logind[1002]: Removed session c12432. Aug 28 09:58:18 xhxRedissjjdA1 systemd[1]: systemd-hostnamed.service: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1024... Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped target Main User Target. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped target Basic System. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped target Paths. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped target Sockets. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped target Timers. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: grub-boot-success.timer: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: dbus.socket: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Closed D-Bus User Message Bus Socket. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: pipewire.socket: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Closed Multimedia System. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Reached target Shutdown. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: systemd-exit.service: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Started Exit the Session. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1788312]: Reached target Exit the Session. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: user@1024.service: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1024. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1024... Aug 28 09:58:23 xhxRedissjjdA1 systemd[159089]: run-user-1024.mount: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: run-user-1024.mount: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1744]: run-user-1024.mount: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1024.service: Succeeded. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1024. Aug 28 09:58:23 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1024. Aug 28 09:58:27 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Main process exited, code=exited, status=255/EXCEPTION Aug 28 09:58:27 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Failed with result 'exit-code'. Aug 28 09:58:55 xhxRedissjjdA1 sshd[1788513]: Accepted keyboard-interactive/pam for check from 10.1.228.232 port 32746 ssh2 Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Created slice User Slice of UID 1024. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Starting User Runtime Directory /run/user/1024... Aug 28 09:58:55 xhxRedissjjdA1 systemd-logind[1002]: New session c12433 of user check. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Started User Runtime Directory /run/user/1024. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Starting User Manager for UID 1024... Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Configuration file /usr/lib/systemd/user/grub-boot-success.timer is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Configuration file /usr/lib/systemd/user/grub-boot-success.service is marked executable. Please remove executable permission bits. Proceeding anyway. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Started Mark boot as successful after the user session has run 2 minutes. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Reached target Paths. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Reached target Timers. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Starting D-Bus User Message Bus Socket. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Listening on Multimedia System. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Listening on D-Bus User Message Bus Socket. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Reached target Sockets. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Reached target Basic System. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Reached target Main User Target. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Started User Manager for UID 1024. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1788524]: Startup finished in 71ms. Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Started Session c12433 of user check. Aug 28 09:58:55 xhxRedissjjdA1 dbus-daemon[979]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.287026' (uid=1024 pid=1788539 comm="/usr/bin/hostnamectl --transient ") Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Starting Hostname Service... Aug 28 09:58:55 xhxRedissjjdA1 dbus-daemon[979]: [system] Successfully activated service 'org.freedesktop.hostname1' Aug 28 09:58:55 xhxRedissjjdA1 systemd[1]: Started Hostname Service. Aug 28 09:58:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Service RestartSec=30s expired, scheduling restart. Aug 28 09:58:57 xhxRedissjjdA1 systemd[1]: kylin-kms-activation.service: Scheduled restart job, restart counter is at 11175. Aug 28 09:58:57 xhxRedissjjdA1 systemd[1]: Stopped run kylin_kms_daemon at boot time. Aug 28 09:58:57 xhxRedissjjdA1 systemd[1]: Started run kylin_kms_daemon at boot time. Aug 28 09:59:06 xhxRedissjjdA1 lvm[1788598]: ppid=1788536, cmdline:vgdisplay -c Aug 28 09:59:06 xhxRedissjjdA1 lvm[1788598]: /run/lock/lvm/P_global:aux: open failed: Permission denied Aug 28 09:59:06 xhxRedissjjdA1 systemd[1]: session-c12433.scope: Succeeded. Aug 28 09:59:06 xhxRedissjjdA1 systemd-logind[1002]: Session c12433 logged out. Waiting for processes to exit. Aug 28 09:59:06 xhxRedissjjdA1 systemd-logind[1002]: Removed session c12433. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: Stopping User Manager for UID 1024... Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped target Main User Target. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped target Basic System. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped target Paths. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped target Sockets. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped target Timers. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: grub-boot-success.timer: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: dbus.socket: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Closed D-Bus User Message Bus Socket. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: pipewire.socket: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Closed Multimedia System. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Reached target Shutdown. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: systemd-exit.service: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Started Exit the Session. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1788524]: Reached target Exit the Session. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: user@1024.service: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: Stopped User Manager for UID 1024. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: Stopping User Runtime Directory /run/user/1024... Aug 28 09:59:16 xhxRedissjjdA1 systemd[159089]: run-user-1024.mount: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1744]: run-user-1024.mount: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: run-user-1024.mount: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: user-runtime-dir@1024.service: Succeeded. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: Stopped User Runtime Directory /run/user/1024. Aug 28 09:59:16 xhxRedissjjdA1 systemd[1]: Removed slice User Slice of UID 1024. Aug 28 09:59:25 xhxRedissjjdA1 systemd[1]: systemd-hostnamed.service: Succeeded.
Comment From: sundb
@Kevin-zhao326 no, i mean the part after
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194f) -> 0000000000005977
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194e) -> 00007fec6b30fed0
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194d) -> 0000000000018127
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194c) -> 00007fec6b2db750
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194b) -> 00000000000544b7
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc194a) -> 00007fec6b3696d0
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1949) -> 0000000000000ada
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1948) -> 00007fec6fd677d0
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1947) -> 0000000000006e09
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1946) -> 00007fec704bfe90
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1945) -> 00000000000084db
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1944) -> 00007fec70636ed0
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1943) -> 0000000000000800
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1942) -> 00007fec817cc000
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1941) -> 0000000000000000
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1940) -> 0000000000000000
Comment From: Kevin-zhao326
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1940) -> 0000000000000000
@sundb The configuration information for Redis is now available. I will forward it to your email and you will receive it later！
Comment From: Kevin-zhao326
141955:M 28 Aug 2024 09:55:01.402 # (00007ffda3dc1940) -> 0000000000000000
@sundb The configuration information for Redis is now available. I will forward it to your email and you will receive it later！
@sundb The email has been sent, please check it！
Comment From: sundb
@Kevin-zhao326 thanks.
Comment From: Kevin-zhao326
@Kevin-zhao326 thanks.
@sundb Hello, the issue I sent last time has reappeared on my end. Did I analyze the reason for the dump file I sent you last time?
Comment From: sundb
@Kevin-zhao326 i'm sorry i haven't had a chance to look at it, did you send it to my email(debing.sun@redis.com), i didn't see it in my box.
Comment From: Kevin-zhao326
@sundb I didn't send it this time. Last time on August 29th, I sent you the relevant logs and configuration information, but there was no final result. Today, a similar problem occurred again, both times on the master node and on different servers
Comment From: sundb
@Kevin-zhao326 still crashed in _writeToClient() this time?
Comment From: Kevin-zhao326
@sundb Yes, the same problem occurred again at 23:34 last night, the same memory crash。
Comment From: Kevin-zhao326
@sundb The problem has reappeared again. It went down around 6am this morning, and the master node crashes approximately every two weeks. Do you have any good suggestions on your end?
Comment From: sundb
@Kevin-zhao326 can you share the redis executable file Redis-server with me (debing.sun@redis.com)?
Comment From: Kevin-zhao326
@sundb The file has been sent to your email. This issue has occurred three times in the past month and a half. Please assist in determining the cause of the problem as soon as possible. Thank you very much!
Comment From: sundb
@Kevin-zhao326 ~~got it~~, thanks. EDIT: I can't find it in my mailbox. was it intercepted?
Comment From: Kevin-zhao326
@sundb The letter was returned and sent again. Please take a look
Comment From: sundb
@Kevin-zhao326 got it now, thx.
Comment From: sundb
after simple analysis, it crashes in _writevToClient() but i don't find out the root cause, need more time.
while (remaining > 0) {
next = listNext(&iter);
o = listNodeValue(next);
if (remaining < (ssize_t)(o->used - c->sentlen)) {
c->sentlen += remaining;
break;
}
remaining -= (ssize_t)(o->used - c->sentlen);
c->reply_bytes -= o->size;
listDelNode(c->reply, next);
c->sentlen = 0;
}
0x00000000004622d7 <+407>: call 0x440f40 <listNext>
0x00000000004622dc <+412>: mov 0x10(%rax),%rcx <- crash here, `o = listNodeValue(next);`
0x00000000004622e0 <+416>: mov 0xb8(%rbx),%rsi
0x00000000004622e7 <+423>: mov 0x8(%rcx),%rdx