Crash report

168148:C 06 Sep 2024 09:45:17.256 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 168148:C 06 Sep 2024 09:45:17.256 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=168148, just started 168148:C 06 Sep 2024 09:45:17.256 # Configuration loaded 168149:M 06 Sep 2024 09:45:17.260 * Node configuration loaded, I'm 0bcc37e8c03b529f97ac0a84dce6d0734df21de1 168149:M 06 Sep 2024 09:45:17.261 * Running mode=cluster, port=3000. 168149:M 06 Sep 2024 09:45:17.261 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 168149:M 06 Sep 2024 09:45:17.261 # Server initialized 168149:M 06 Sep 2024 09:45:17.261 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command ' sysctl vm.overcommit_memory=1' for this to take effect. 168149:M 06 Sep 2024 09:45:17.261 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kerne l/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

=== REDIS BUG REPORT START: Cut & paste starting from here === 168149:M 06 Sep 2024 09:48:12.959 # === ASSERTION FAILED OBJECT CONTEXT === 168149:M 06 Sep 2024 09:48:12.959 # Object type: 0 168149:M 06 Sep 2024 09:48:12.959 # Object encoding: 0 168149:M 06 Sep 2024 09:48:12.959 # Object refcount: 1 168149:M 06 Sep 2024 09:48:12.960 # Object raw string len: 3 168149:M 06 Sep 2024 09:48:12.960 # Object raw string content: "\x00\x00\x00" 168149:M 06 Sep 2024 09:48:12.960 # === ASSERTION FAILED === 168149:M 06 Sep 2024 09:48:12.960 # ==> db.c:177 'retval == DICT_OK' is not true 168149:M 06 Sep 2024 09:48:12.960 # (forcing SIGSEGV to print the bug report.) 168149:M 06 Sep 2024 09:48:12.960 # Redis 5.0.3 crashed by signal: 11 168149:M 06 Sep 2024 09:48:12.960 # Crashed running the instruction at: 0x46e73a 168149:M 06 Sep 2024 09:48:12.960 # Accessing address: 0xffffffffffffffff 168149:M 06 Sep 2024 09:48:12.960 # Failed assertion: retval == DICT_OK (db.c:177)

------ STACK TRACE ------ EIP: /redis/bin/redis-server 10.10.10.45:3000 cluster[0x46e73a]

Backtrace: /redis/bin/redis-server 10.10.10.45:3000 cluster[0x470919] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x470fbc] /lib64/libpthread.so.0(+0xf630)[0x7f741cb35630] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x46e73a] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x4466d7] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x4511d8] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x45182b] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x43461b] /redis/bin/redis-server 10.10.10.45:3000 cluster[0x427bf5] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f741c77a555] /redis/bin/redis-server 10.10.10.45:3000 [cluster][0x427eda]

------ INFO OUTPUT ------

Server

redis_version:5.0.3 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:de2580f8d1dffac2 redis_mode:cluster os:Linux 3.10.0-1127.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:4.8.5 process_id:168149 run_id:a54cb1a50a5d5c4d612ec77547aaabe17dc4edb4 tcp_port:3000 uptime_in_seconds:175 uptime_in_days:0 hz:100 configured_hz:100 lru_clock:14311085 executable:/redis/bin/redis-server config_file:/redis/conf/3000.conf

Clients

connected_clients:44 client_recent_max_input_buffer:0 client_recent_max_output_buffer:0 blocked_clients:0

Memory

used_memory:11410157048 used_memory_human:10.63G used_memory_rss:0 used_memory_rss_human:0B used_memory_peak:11410157048 used_memory_peak_human:10.63G used_memory_peak_perc:100.05% used_memory_overhead:495631376 used_memory_startup:1330672 used_memory_dataset:10914525672 used_memory_dataset_perc:95.67% allocator_allocated:0 allocator_active:0 allocator_resident:0 total_system_memory:540820533248 total_system_memory_human:503.68G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:21474836480 maxmemory_human:20.00G maxmemory_policy:volatile-lru allocator_frag_ratio:-nan allocator_frag_bytes:0 allocator_rss_ratio:-nan allocator_rss_bytes:0 rss_overhead_ratio:-nan rss_overhead_bytes:0 mem_fragmentation_ratio:-nan mem_fragmentation_bytes:0 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:2186536 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:1 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1725587117 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 loading_start_time:1725587117 loading_total_bytes:4272770107 loading_loaded_bytes:3124756466 loading_loaded_perc:73.13 loading_eta_seconds:64

Stats

total_connections_received:143 total_commands_processed:3943 instantaneous_ops_per_sec:0 total_net_input_bytes:243454 total_net_output_bytes:379023 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:master connected_slaves:0 master_replid:b8cfd3a3500021456f8e804c9330bb735dc7c78e master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:29.489370 used_cpu_user:146.180966 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000

Commandstats

cmdstat_auth:calls=3937,usec=4274,usec_per_call=1.09 cmdstat_info:calls=6,usec=172,usec_per_call=28.67

Cluster

cluster_enabled:1

Keyspace

db0:keys=7989780,expires=897001,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=45 addr=10.10.10.35:41772 fd=51 name= age=149 idle=149 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=101 addr=10.10.10.169:43534 fd=15 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=102 addr=10.10.10.169:43536 fd=16 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=103 addr=10.10.10.169:43538 fd=17 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=104 addr=10.10.10.169:43540 fd=18 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=105 addr=10.10.10.169:43542 fd=19 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=106 addr=10.10.10.169:43546 fd=20 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=107 addr=10.10.10.169:43544 fd=21 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=108 addr=10.10.10.169:43548 fd=22 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=109 addr=10.10.10.169:43552 fd=23 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=110 addr=10.10.10.169:43550 fd=24 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=111 addr=10.10.10.169:43554 fd=25 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=112 addr=10.10.10.169:43556 fd=26 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=113 addr=10.10.10.169:43558 fd=27 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=114 addr=10.10.10.169:43560 fd=28 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=115 addr=10.10.10.169:43562 fd=29 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=116 addr=10.10.10.169:43564 fd=30 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=117 addr=10.10.10.169:43566 fd=31 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=118 addr=10.10.10.169:43568 fd=32 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=119 addr=10.10.10.169:43570 fd=33 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=120 addr=10.10.10.169:43572 fd=34 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=121 addr=10.10.10.169:43574 fd=35 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=122 addr=10.10.10.169:43578 fd=36 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=123 addr=10.10.10.169:43580 fd=37 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=124 addr=10.10.10.169:43576 fd=38 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=125 addr=10.10.10.169:43582 fd=39 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=126 addr=10.10.10.169:43584 fd=40 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=127 addr=10.10.10.169:43586 fd=41 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=128 addr=10.10.10.239:51780 fd=42 name= age=42 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping id=130 addr=10.10.10.85:40867 fd=43 name= age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=131 addr=10.10.10.85:40868 fd=44 name= age=29 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=132 addr=10.10.10.85:40870 fd=45 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=133 addr=10.10.10.85:40869 fd=46 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=134 addr=10.10.10.85:40872 fd=47 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=135 addr=10.10.10.85:40871 fd=48 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=136 addr=10.10.10.85:40873 fd=49 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=137 addr=10.10.10.85:40874 fd=50 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=138 addr=10.10.10.85:40875 fd=52 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=139 addr=10.10.10.85:40876 fd=53 name= age=28 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=140 addr=10.10.10.33:51938 fd=10 name= age=15 idle=15 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=141 addr=10.10.10.33:51940 fd=11 name= age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=142 addr=10.10.10.33:51942 fd=12 name= age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=143 addr=10.10.10.33:51944 fd=13 name= age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth id=144 addr=10.10.10.33:51946 fd=14 name= age=14 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=auth

------ REGISTERS ------ 168149:M 06 Sep 2024 09:48:12.961 # RAX:0000000000000000 RBX:00000000000000b1 RCX:0000000001b94280 RDX:0000000000012b40 RDI:00007f741cb1f760 RSI:0000000000000000 RBP:0000000000536f56 RSP:00007ffc03b45460 R8 :0000000000000001 R9 :00007f741d666f80 R10:7562206568742074 R11:0000000000000206 R12:00000000005343fc R13:00007f7369ff1440 R14:00007ffc03b45930 R15:00007f7116717f10 RIP:000000000046e73a EFL:0000000000010206 CSGSFS:0000000000000033 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546f) -> 0000000000000009 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546e) -> 00000191c501d4cd 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546d) -> 0000000000000000 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546c) -> 00007ffc03b459d0 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546b) -> 00007f741c2835d1 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b4546a) -> 0000000000da5ead 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b45469) -> 000000000055f5c1 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b45468) -> 0000000000000000 168149:M 06 Sep 2024 09:48:12.961 # (00007ffc03b45467) -> 00000000004511d8 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45466) -> 00007f741c21ac00 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45465) -> ffffffffffffffff 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45464) -> ffffffffffffffff 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45463) -> 00000000004466d7 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45462) -> 00007f741c21ac00 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45461) -> 00007f7116717f10 168149:M 06 Sep 2024 09:48:12.962 # (00007ffc03b45460) -> 00007f7369ff1440

------ FAST MEMORY TEST ------ 168149:M 06 Sep 2024 09:48:12.962 # Bio thread for job type #0 terminated 168149:M 06 Sep 2024 09:48:12.962 # Bio thread for job type #1 terminated 168149:M 06 Sep 2024 09:48:12.962 # Bio thread for job type #2 terminated *** Preparing to test memory region 78d000 (2252800 bytes) *** Preparing to test memory region 1b86000 (135168 bytes) *** Preparing to test memory region 7f70faa00000 (12937330688 bytes) *** Preparing to test memory region 7f73fddba000 (369098752 bytes) *** Preparing to test memory region 7f7413dbb000 (8388608 bytes) *** Preparing to test memory region 7f74145bc000 (8388608 bytes) *** Preparing to test memory region 7f7414dbd000 (8388608 bytes) *** Preparing to test memory region 7f74155bd000 (3145728 bytes) *** Preparing to test memory region 7f741be00000 (8388608 bytes) *** Preparing to test memory region 7f741cb21000 (20480 bytes) *** Preparing to test memory region 7f741cd3e000 (16384 bytes) *** Preparing to test memory region 7f741d665000 (20480 bytes) *** Preparing to test memory region 7f741d66e000 (8192 bytes) *** Preparing to test memory region 7f741d670000 (4096 bytes) *** Preparing to test memory region 7f741d673000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: _serverAssert (base: 0x46e6d0) Module: /redis/bin/redis-server 10.10.10.45:3000 [cluster] (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x46e6d0 -D -b binary -m i386:x86-64 /tmp/dump.bin

168149:M 06 Sep 2024 09:49:11.330 # dump of function (hexdump of 234 bytes): 41548b05940b32004989fc554889f585c05389d37505e8f5fdffffbeaec65300bf0300000031c0e874fafbff4d89e089d94889eabec7c65300bf0300000031c0e85bfafbffbe48b65300bf0300000031c04c8925300b320048892d310b3200891d330b3200e836fafbffc60425ffffffff785b5d4 15cc3660f1f84000000000041544989fc55534883c4808b150b0b320085d20f84e700000031c0be78b65300bf03000000e8f6f9fbff418b9424a000000031c0bee2c65300bf0300000031ed31dbe8d9f9fbff418b54240831c0bef5c65300bf03000000e8c3f9fbff418b54244031c0be05c75300 bf

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/antirez/redis/issues

Suspect RAM error? Use redis-server --test-memory to verify it.

Additional information

Linux 3.10.0-1127.el7.x86_64 x86_64

Comment From: sundb

@a296961081 i saw that THP was enabled, which may cause OOM, eventually causing _dictKeyIndex() fail and trigger this assertion. please try to disable THP and try again.

Comment From: Wang-DuoYou

Issue Description: In our production environment, we encountered a problem where one of the slave nodes in our Redis cluster, which was offline for a prolonged period, crashed when we attempted to start it today. The error message was Failed assertion: retval == DICT_OK.

System Details:

System Memory: 64GB Redis Version: 5.0.3 Solution: We resolved the issue by setting the system parameter vm.overcommit_memory to 1. However, I do not fully understand the reason behind this solution and hope that this information will be helpful to others who encounter the same issue.

Crash report

113628:M 16 Oct 2024 14:05:17.837 # Server initialized 113628:M 16 Oct 2024 14:05:17.837 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 113628:M 16 Oct 2024 14:05:17.837 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 113628:M 16 Oct 2024 14:05:17.838 * Reading RDB preamble from AOF file... 113628:M 16 Oct 2024 14:08:47.725 * Reading the remaining AOF tail... 113628:M 16 Oct 2024 14:11:49.560 * DB loaded from append only file: 391.723 seconds 113628:M 16 Oct 2024 14:11:49.560 * Ready to accept connections 113628:M 16 Oct 2024 14:11:49.596 * 10 changes in 300 seconds. Saving... 113628:M 16 Oct 2024 14:11:49.778 * Background saving started by pid 113787 113628:S 16 Oct 2024 14:11:49.785 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer. 113628:S 16 Oct 2024 14:11:49.785 # Cluster state changed: ok 113628:S 16 Oct 2024 14:11:49.822 * FAIL message received from 3be067887fe0b54e57f97dd854eccc928339cbd7 about 74e5371618dac1fc786320c2c4e5d380e1106b3b 113628:S 16 Oct 2024 14:11:49.881 * Clear FAIL state for node 74e5371618dac1fc786320c2c4e5d380e1106b3b: replica is reachable again. 113628:S 16 Oct 2024 14:11:50.864 * Connecting to MASTER 10.4.146.141:6379 113628:S 16 Oct 2024 14:11:50.864 * MASTER <-> REPLICA sync started 113628:S 16 Oct 2024 14:11:50.864 * Non blocking connect for SYNC fired the event. 113628:S 16 Oct 2024 14:11:50.864 * Master replied to PING, replication can continue... 113628:S 16 Oct 2024 14:11:50.866 * Trying a partial resynchronization (request a6feb0267b4d301420e233c03152ef137b04c787:1). 113628:S 16 Oct 2024 14:12:50.391 * FAIL message received from 3be067887fe0b54e57f97dd854eccc928339cbd7 about 74e5371618dac1fc786320c2c4e5d380e1106b3b 113628:S 16 Oct 2024 14:12:59.504 * Clear FAIL state for node 74e5371618dac1fc786320c2c4e5d380e1106b3b: replica is reachable again. 113628:S 16 Oct 2024 14:13:44.562 * Full resync from master: c8ef829dd7bd327ef1794d1847b73dfb38aa3c09:2135459834413 113628:S 16 Oct 2024 14:13:44.562 * Discarding previously cached master state. 113787:C 16 Oct 2024 14:15:42.844 * DB saved on disk 113787:C 16 Oct 2024 14:15:43.010 * RDB: 660 MB of memory used by copy-on-write 113628:S 16 Oct 2024 14:15:43.314 * Background saving terminated with success 113628:S 16 Oct 2024 14:18:34.165 * MASTER <-> REPLICA sync: receiving 14290978707 bytes from master 113628:S 16 Oct 2024 14:19:23.693 * MASTER <-> REPLICA sync: Flushing old data 113628:S 16 Oct 2024 14:21:56.560 * MASTER <-> REPLICA sync: Loading DB in memory

=== REDIS BUG REPORT START: Cut & paste starting from here === 113628:S 16 Oct 2024 14:25:10.047 # === ASSERTION FAILED OBJECT CONTEXT === 113628:S 16 Oct 2024 14:25:10.047 # Object type: 0 113628:S 16 Oct 2024 14:25:10.047 # Object encoding: 0 113628:S 16 Oct 2024 14:25:10.047 # Object refcount: 1 113628:S 16 Oct 2024 14:25:10.047 # Object raw string len: 25 113628:S 16 Oct 2024 14:25:10.047 # Object raw string content: "jis:flogin:zwfw0028161369" 113628:S 16 Oct 2024 14:25:10.047 # === ASSERTION FAILED === 113628:S 16 Oct 2024 14:25:10.047 # ==> db.c:177 'retval == DICT_OK' is not true 113628:S 16 Oct 2024 14:25:10.047 # (forcing SIGSEGV to print the bug report.) 113628:S 16 Oct 2024 14:25:10.047 # Redis 5.0.3 crashed by signal: 11 113628:S 16 Oct 2024 14:25:10.047 # Crashed running the instruction at: 0x46e71a 113628:S 16 Oct 2024 14:25:10.047 # Accessing address: 0xffffffffffffffff 113628:S 16 Oct 2024 14:25:10.047 # Failed assertion: retval == DICT_OK (db.c:177)

------ INFO OUTPUT ------

Server

redis_version:5.0.3 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:f434073981798e6 redis_mode:cluster os:Linux 3.10.0-957.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:4.8.5 process_id:113628 run_id:e11bbc2db311f1a4b30486a9b10a753564800e12 tcp_port:6379 uptime_in_seconds:1193 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:1006314 executable:/data/redis/src/redis-server

Clients

connected_clients:0 client_recent_max_input_buffer:2 client_recent_max_output_buffer:0 blocked_clients:0

Memory

used_memory:18297958736 used_memory_human:17.04G used_memory_rss:23717244928 used_memory_rss_human:22.09G used_memory_peak:23270154768 used_memory_peak_human:21.67G used_memory_peak_perc:78.63% used_memory_overhead:1733663464 used_memory_startup:1462848 used_memory_dataset:16564295272 used_memory_dataset_perc:90.53% allocator_allocated:19891963656 allocator_active:23268388864 allocator_resident:23723397120 total_system_memory:67632943104 total_system_memory_human:62.99G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:42949672960 maxmemory_human:40.00G maxmemory_policy:volatile-lru allocator_frag_ratio:1.17 allocator_frag_bytes:3376425208 allocator_rss_ratio:1.02 allocator_rss_bytes:455008256 rss_overhead_ratio:1.00 rss_overhead_bytes:-6152192 mem_fragmentation_ratio:1.19 mem_fragmentation_bytes:3825452176 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:0 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:1 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1729059343 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:234 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:692699136 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 loading_start_time:1729059716 loading_total_bytes:14290978707 loading_loaded_bytes:11962154605 loading_loaded_perc:83.70 loading_eta_seconds:37

Stats

total_connections_received:537 total_commands_processed:331 instantaneous_ops_per_sec:0 total_net_input_bytes:14290990735 total_net_output_bytes:154427 instantaneous_input_kbps:296655.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:2858 expired_stale_perc:4.96 expired_time_cap_reached_count:1 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:181843 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0

Replication

role:slave master_host: master_port:6379 master_link_status:down master_last_io_seconds_ago:-1 master_sync_in_progress:1 slave_repl_offset:1 master_sync_left_bytes:0 master_sync_last_io_seconds_ago:348 master_link_down_since_seconds:1729059910 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:a6feb0267b4d301420e233c03152ef137b04c787 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:268435456 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:102.844732 used_cpu_user:685.280556 used_cpu_sys_children:17.525354 used_cpu_user_children:212.352050

Commandstats

cmdstat_info:calls=17,usec=1984,usec_per_call=116.71 cmdstat_readonly:calls=23,usec=7,usec_per_call=0.30 cmdstat_config:calls=17,usec=1456,usec_per_call=85.65 cmdstat_cluster:calls=31,usec=3370,usec_per_call=108.71 cmdstat_latency:calls=17,usec=41,usec_per_call=2.41 cmdstat_slowlog:calls=16,usec=28,usec_per_call=1.75 cmdstat_auth:calls=179,usec=259,usec_per_call=1.45 cmdstat_client:calls=31,usec=46,usec_per_call=1.48

Cluster

cluster_enabled:1

Keyspace

db0:keys=18694574,expires=18647781,avg_ttl=13784880

------ CLIENT LIST OUTPUT ------

------ REGISTERS ------ 113628:S 16 Oct 2024 14:25:10.048 # RAX:0000000000000000 RBX:00000000000000b1 RCX:00000000010211e0 RDX:0000000000012be0 RDI:00007fbd124fc760 RSI:0000000000000000 RBP:0000000000536f56 RSP:00007fffb2626710 R8 :0000000000000001 R9 :00007fbd13042f80 R10:7562206568742074 R11:0000000000000206 R12:00000000005343fc R13:00007fb7dda52d10 R14:00007fffb2626be0 R15:00007fb7dce205f0 RIP:000000000046e71a EFL:0000000000010206 CSGSFS:0000000000000033 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671f) -> 0000000000000009 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671e) -> 0000019293fd7dd0 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671d) -> 0000000000000000 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671c) -> 00007fffb2626c70 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671b) -> 00007fbc79bd4531 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb262671a) -> 00000000000f5aea 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626719) -> 00000000000167c9 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626718) -> 00007fbd11c28558 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626717) -> 00000000004511b8 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626716) -> 00007fbd11c1a800 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626715) -> 00000192d68f8597 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626714) -> ffffffffffffffff 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626713) -> 00000000004466b7 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626712) -> 00007fbd11c1a800 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626711) -> 00007fb7dce205f0 113628:S 16 Oct 2024 14:25:10.048 # (00007fffb2626710) -> 00007fb7dda52d10

------ FAST MEMORY TEST ------ 113628:S 16 Oct 2024 14:25:10.048 # Bio thread for job type #0 terminated 113628:S 16 Oct 2024 14:25:10.049 # Bio thread for job type #1 terminated 113628:S 16 Oct 2024 14:25:10.049 # Bio thread for job type #2 terminated *** Preparing to test memory region 78d000 (2248704 bytes) *** Preparing to test memory region 1013000 (135168 bytes) *** Preparing to test memory region 7fb6d4c00000 (26317160448 bytes) *** Preparing to test memory region 7fbcf57d3000 (335544320 bytes) *** Preparing to test memory region 7fbd097d4000 (8388608 bytes) *** Preparing to test memory region 7fbd09fd5000 (8388608 bytes) *** Preparing to test memory region 7fbd0a7d6000 (8388608 bytes) *** Preparing to test memory region 7fbd0afd6000 (3145728 bytes) *** Preparing to test memory region 7fbd11800000 (8388608 bytes) *** Preparing to test memory region 7fbd124fe000 (20480 bytes) *** Preparing to test memory region 7fbd1271b000 (16384 bytes) *** Preparing to test memory region 7fbd13041000 (20480 bytes) *** Preparing to test memory region 7fbd1304b000 (8192 bytes) *** Preparing to test memory region 7fbd1304d000 (4096 bytes) *** Preparing to test memory region 7fbd13050000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: _serverAssert (base: 0x46e6b0) Module: /data/redis/src/redis-server 10.4.146.144:6379 [cluster] (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x46e6b0 -D -b binary -m i386:x86-64 /tmp/dump.bin

113628:S 16 Oct 2024 14:26:39.091 # dump of function (hexdump of 234 bytes): 41548b05740b32004989fc554889f585c05389d37505e8f5fdffffbeaec65300bf0300000031c0e874fafbff4d89e089d94889eabec7c65300bf0300000031c0e85bfafbffbe48b65300bf0300000031c04c8925100b320048892d110b3200891d130b3200e836fafbffc60425ffffffff785b5d415cc3660f1f84000000000041544989fc55534883c4808b15eb0a320085d20f84e700000031c0be78b65300bf03000000e8f6f9fbff418b9424a000000031c0bee2c65300bf0300000031ed31dbe8d9f9fbff418b54240831c0bef5c65300bf03000000e8c3f9fbff418b54244031c0be05c75300bf

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

   Please report the crash by opening an issue on github:

       http://github.com/antirez/redis/issues

Suspect RAM error? Use redis-server --test-memory to verify it.

Redis Restarting the slave node redis, an error occurred when importing rdb

Server

Clients

Memory

Persistence

Stats

Replication

CPU

Commandstats

Cluster

Keyspace

Crash report

Server

Clients

Memory

Persistence

Stats

Replication

CPU

Commandstats

Cluster

Keyspace