=== REDIS BUG REPORT START: Cut & paste starting from here ===
11361:M 29 Aug 2024 19:35:05.570 # Redis 5.0.8.1 crashed by signal: 11
11361:M 29 Aug 2024 19:35:05.570 # Crashed running the instruction at: 0x4a48ae
11361:M 29 Aug 2024 19:35:05.570 # Accessing address: (nil)
11361:M 29 Aug 2024 19:35:05.570 # Failed assertion:
------ STACK TRACE ------ EIP: /home/zyb/apps/redis/src/redis-server *:7000 cluster[0x4a48ae]
Backtrace: /home/zyb/apps/redis/src/redis-server :7000 cluster[0x4712e9] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x47198c] /lib64/libpthread.so.0(+0xf630)[0x7f903cdfc630] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x4a48ae] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x445ebd] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x445ff2] /home/zyb/apps/redis/src/redis-server :7000 [cluster][0x49feb5] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x4a005a] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x42e0bf] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x430307] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x429fdf] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x42a21b] /home/zyb/apps/redis/src/redis-server :7000 cluster[0x4270ce] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f903ca41555] /home/zyb/apps/redis/src/redis-server *:7000 [cluster][0x427313]
------ INFO OUTPUT ------
Server
redis_version:5.0.8.1 auth_center_password_version:2 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:7cbb5836a8fd1ab7 redis_mode:cluster os:Linux 3.10.0-1160.el7.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:4.8.5 process_id:11361 run_id:ff3a07b3eb8f00d20af450afaaef9f5ee964c976 tcp_port:7000 uptime_in_seconds:66417087 uptime_in_days:768 hz:10 configured_hz:10 lru_clock:13655273 executable:/home/zyb/apps/redis/src/redis-server config_file:/home/zyb/apps/redis/redis-7000.conf
Clients
connected_clients:13 client_recent_max_input_buffer:69985 client_recent_max_output_buffer:0 blocked_clients:0
Memory
used_memory:16208380648 used_memory_human:15.10G used_memory_rss:17744711680 used_memory_rss_human:16.53G used_memory_peak:18141666680 used_memory_peak_human:16.90G used_memory_peak_perc:89.34% used_memory_overhead:1029195072 used_memory_startup:1533160 used_memory_dataset:15179185576 used_memory_dataset_perc:93.66% allocator_allocated:16208581744 allocator_active:17753706496 allocator_resident:18154328064 total_system_memory:33562468352 total_system_memory_human:31.26G used_memory_lua:46080 used_memory_lua_human:45.00K used_memory_scripts:216 used_memory_scripts_human:216B number_of_cached_scripts:1 maxmemory:25769803776 maxmemory_human:24.00G maxmemory_policy:volatile-lru allocator_frag_ratio:1.10 allocator_frag_bytes:1545124752 allocator_rss_ratio:1.02 allocator_rss_bytes:400621568 rss_overhead_ratio:0.98 rss_overhead_bytes:-409616384 mem_fragmentation_ratio:1.09 mem_fragmentation_bytes:1536290768 mem_not_counted_for_evict:768 mem_replication_backlog:5242880 mem_clients_slaves:16922 mem_clients_normal:383846 mem_aof_buffer:768 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0
Persistence
loading:0 rdb_changes_since_last_save:6529866061 rdb_bgsave_in_progress:0 rdb_last_save_time:1658514218 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:28979200 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:196 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:416260096 aof_current_size:10275580269 aof_base_size:6957885077 aof_pending_rewrite:0 aof_buffer_length:83 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:40
Stats
total_connections_received:7332530 total_commands_processed:96373440565 instantaneous_ops_per_sec:175 total_net_input_bytes:13062876428502 total_net_output_bytes:52420437687812 instantaneous_input_kbps:25.60 instantaneous_output_kbps:36.86 rejected_connections:0 sync_full:1 sync_partial_ok:1 sync_partial_err:1 expired_keys:122399236 expired_stale_perc:0.93 expired_time_cap_reached_count:0 evicted_keys:0 keyspace_hits:86509989384 keyspace_misses:2282869838 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:531521 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:52160630 active_defrag_misses:202798485 active_defrag_key_hits:7696416 active_defrag_key_misses:3844927
Replication
role:master connected_slaves:1 slave0:ip=192.168.5.14,port=7000,state=online,offset=5524181706908,lag=0 master_replid:61bce7676e01ace30f30f01c96b68c97c96bc207 master_replid2:61ac2002c8aa8082ed2b2f04db43ed83b4ab38b4 master_repl_offset:5524181715183 second_repl_offset:5969344878 repl_backlog_active:1 repl_backlog_size:5242880 repl_backlog_first_byte_offset:5524176472304 repl_backlog_histlen:5242880
CPU
used_cpu_sys:1589925.030564 used_cpu_user:1298348.425759 used_cpu_sys_children:9779.600317 used_cpu_user_children:107500.142751
Commandstats
cmdstat_info:calls=2529164,usec=288969273,usec_per_call=114.25 cmdstat_del:calls=1400274434,usec=6296191611,usec_per_call=4.50 cmdstat_eval:calls=1393461695,usec=30886655137,usec_per_call=22.17 cmdstat_dbsize:calls=367,usec=206,usec_per_call=0.56 cmdstat_pexpire:calls=1440762186,usec=2345455566,usec_per_call=1.63 cmdstat_type:calls=27703594845,usec=82663338877,usec_per_call=2.98 cmdstat_exec:calls=331,usec=3789,usec_per_call=11.45 cmdstat_scan:calls=10269255,usec=39591107722,usec_per_call=3855.30 cmdstat_config:calls=1116000,usec=17581598,usec_per_call=15.75 cmdstat_hdel:calls=413012727,usec=2563139999,usec_per_call=6.21 cmdstat_replconf:calls=66251915,usec=82630856,usec_per_call=1.25 cmdstat_command:calls=1,usec=2399,usec_per_call=2399.00 cmdstat_select:calls=1,usec=0,usec_per_call=0.00 cmdstat_cluster:calls=3327771,usec=1338862735,usec_per_call=402.33 cmdstat_get:calls=1393461695,usec=1982006400,usec_per_call=1.42 cmdstat_slowlog:calls=110808,usec=87633457,usec_per_call=790.86 cmdstat_client:calls=24812,usec=34535,usec_per_call=1.39 cmdstat_hmget:calls=4294271749,usec=40072805695,usec_per_call=9.33 cmdstat_hgetall:calls=55401530933,usec=260478257744,usec_per_call=4.70 cmdstat_hmset:calls=1440762186,usec=20647600313,usec_per_call=14.33 cmdstat_ping:calls=222380,usec=87845,usec_per_call=0.40 cmdstat_set:calls=1398442802,usec=11582965237,usec_per_call=8.28 cmdstat_auth:calls=10008919,usec=9562067,usec_per_call=0.96 cmdstat_multi:calls=331,usec=35,usec_per_call=0.11 cmdstat_psync:calls=2,usec=223,usec_per_call=111.50
Cluster
cluster_enabled:1
Keyspace
db0:keys=11774716,expires=11774716,avg_ttl=489998431049
------ CLIENT LIST OUTPUT ------ id=358 addr=192.168.5.14:26621 fd=43 name= age=66415693 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=83 oll=0 omem=0 events=r cmd=replconf id=7332417 addr=192.168.5.13:30036 fd=52 name=EAF-EAF age=2298 idle=2298 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7332434 addr=192.168.5.12:37532 fd=55 name=EAF-EAF age=1968 idle=1968 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7306364 addr=192.168.5.11:32522 fd=8 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=eval id=7332228 addr=192.168.4.2:17442 fd=63 name=EAF-EAF age=5929 idle=5929 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7306365 addr=192.168.4.3:40678 fd=14 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hmget id=7306366 addr=192.168.4.5:62374 fd=16 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hmget id=7306367 addr=192.168.4.7:44868 fd=36 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hmget id=7332346 addr=192.168.4.9:42222 fd=61 name=EAF-EAF age=3689 idle=3689 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7306368 addr=192.168.5.16:31162 fd=37 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=eval id=7306369 addr=192.168.5.18:49308 fd=38 name=EAF-EAF age=507121 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=pexpire id=7332004 addr=192.168.4.9:7098 fd=62 name=EAF-EAF age=9987 idle=9987 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7332272 addr=192.168.5.11:44784 fd=64 name=EAF-EAF age=5101 idle=5101 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=7332093 addr=192.168.4.8:23952 fd=65 name=EAF-EAF age=8376 idle=8376 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client
------ REGISTERS ------ 11361:M 29 Aug 2024 19:35:05.572 # RAX:0000000000000ce0 RBX:0000000000000ce0 RCX:0000000000000034 RDX:00007f8ce59003e1 RDI:000000000000002f RSI:000000000000001a RBP:00007f8fe28cf4a0 RSP:00007fff52ced940 R8 :0000000000000015 R9 :745fb78fe5b788e6 R10:0000000000000030 R11:000000000000005a R12:00007fff52ced988 R13:0000000000000041 R14:4100007f8ce32909 R15:0000000000000032 RIP:00000000004a48ae EFL:0000000000010246 CSGSFS:0000000000000033 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94f) -> 00007f90248efed8 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94e) -> 00007f8f418e8e10 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94d) -> 00007f8f6cdc3cf0 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94c) -> 00007f8f91cc4530 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94b) -> 00007f8cba877fe8 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced94a) -> 00007f9031868400 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced949) -> 00007f903c632170 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced948) -> 0000000000000020 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced947) -> 000000000000001a 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced946) -> 00007fff52ced988 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced945) -> 00007f8cd6c57a11 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced944) -> 00007f8ba37e8700 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced943) -> 0000000000000000 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced942) -> 00007f903c628660 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced941) -> 00007f903c628660 11361:M 29 Aug 2024 19:35:05.572 # (00007fff52ced940) -> 00007f903d92b730
Additional information
- VM ，redhat7.9 ，redis-cluster (16 master 16slave)
- Not reproduced
Comment From: sundb
@0379xiaomao It's hard to figure out why without a clear stack. i saw the auth_center_password_version in the INFO, did you change anything?
Comment From: 0379xiaomao
Configuration parameters are not modified.nothing is modified.Nothing has been modified. 20 minutes after the virtual machine hot migration of the Redis instance, the instance experienced a crash.
Comment From: 0379xiaomao
compile Redis use jemalloc.
Comment From: sundb
@0379xiaomao but info auth_center_password_version doesn't belong to Redis.
Comment From: 0379xiaomao
redis-server is adapted to the authentication center.The authentication center is a small function developed by us the auth_center_password_version is part of the authentication center and does not belong to the configuration parameters of Redis itself.