Crash report
=== REDIS BUG REPORT START: Cut & paste starting from here ===
21:M 18 Jan 2024 01:13:46.577 # Redis 7.0.4 crashed by signal: 11, si_code: 128
21:M 18 Jan 2024 01:13:46.577 # Accessing address: (nil)
I0118 01:13:46.577425 1 log.go:16] 21:M 18 Jan 2024 01:13:46.577 # Crashed running the instruction at: 0x55be7f98ce20
I0118 01:13:46.577556 1 log.go:16]
------ STACK TRACE ------
EIP:
I0118 01:13:46.577733 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](+0xdce20)[0x55be7f98ce20]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7f602255e140]
I0118 01:13:46.577857 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](+0xdce20)[0x55be7f98ce20]
I0118 01:13:46.577887 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](clusterProcessPacket+0x64c)[0x55be7f999dcc]
I0118 01:13:46.577964 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](clusterReadHandler+0xd5)[0x55be7f99a7a5]
I0118 01:13:46.578041 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](+0x13dfa8)[0x55be7f9edfa8]
I0118 01:13:46.578091 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](aeProcessEvents+0x1ca)[0x55be7f92186a]
I0118 01:13:46.578162 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](aeMain+0x1d)[0x55be7f921c0d]
I0118 01:13:46.578207 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](main+0x311)[0x55be7f91d591]
I0118 01:13:46.578231 1 log.go:16] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f6022399d0a]
I0118 01:13:46.578278 1 log.go:16] redis-server 0.0.0.0:6379 [cluster](_start+0x2a)[0x55be7f91dbea]
------ REGISTERS ------
I0118 01:13:46.578287 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 #
RAX:0000000000000041 RBX:00007f601eb94050
RCX:000000000000000f RDX:0000000000000000
RDI:00007f6021fb2200 RSI:0000000000000000
RBP:00007f601eba8800 RSP:00007ffe20e53780
R8 :0000000000000000 R9 :0000000000000071
R10:00007ffe20e53280 R11:0000000000000246
R12:ccf0d2f2fe556be6 R13:00007f601eba4240
R14:00007f601eb94000 R15:0000018d1a219b51
RIP:000055be7f98ce20 EFL:0000000000010246
CSGSFS:002b000000000033
I0118 01:13:46.578313 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378f) -> 0000000000000ad8
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378e) -> 00000000000015b0
I0118 01:13:46.578320 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378d) -> 0000000000000ad0
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378c) -> 0000000000000400
I0118 01:13:46.578325 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378b) -> 000055be7f937308
I0118 01:13:46.578331 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e5378a) -> 00007f601eb94000
I0118 01:13:46.578336 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53789) -> 0000000000000000
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53788) -> 0000000000000400
I0118 01:13:46.578350 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53787) -> 00000000000015b0
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53786) -> 000000000000003c
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53785) -> 0000120100000000
I0118 01:13:46.578356 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53784) -> 0000000000000034
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53783) -> 000055be7f999dcc
I0118 01:13:46.578361 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53782) -> 0000000000000001
I0118 01:13:46.578367 1 log.go:16] 21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53781) -> 00007f601eba8800
21:M 18 Jan 2024 01:13:46.578 # (00007ffe20e53780) -> 00007f601eb94050
I0118 01:13:46.578372 1 log.go:16]
------ INFO OUTPUT ------
I0118 01:13:46.578767 1 log.go:16] # Server
redis_version:7.0.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:4f8875cbf3281026
redis_mode:cluster
os:Linux 4.18.0-372.36.1.el8_6.x86_64 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:10.2.1
process_id:21
process_supervised:no
run_id:d0995ddedeec514d971b5fbf454e351213258cbb
tcp_port:6379
server_time_usec:1705540426577083
uptime_in_seconds:562
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:11041610
executable:/data/redis-server
config_file:/redis-conf/redis.conf
io_threads_active:0
# Clients
connected_clients:364
cluster_connections:78
maxclients:50000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:20704
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:39447248
used_memory_human:37.62M
used_memory_rss:46174208
used_memory_rss_human:44.04M
used_memory_peak:39476576
used_memory_peak_human:37.65M
used_memory_peak_perc:99.93%
used_memory_overhead:17695928
used_memory_startup:4068896
used_memory_dataset:21751320
used_memory_dataset_perc:61.48%
allocator_allocated:39544120
allocator_active:42696704
allocator_resident:50479104
total_system_memory:160268918784
total_system_memory_human:149.26G
used_memory_lua:79872
used_memory_vm_eval:79872
used_memory_lua_human:78.00K
used_memory_scripts_eval:504
number_of_cached_scripts:1
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:112640
used_memory_vm_total_human:110.00K
used_memory_functions:184
used_memory_scripts:688
used_memory_scripts_human:688B
maxmemory:12025908428
maxmemory_human:11.20G
maxmemory_policy:noeviction
allocator_frag_ratio:1.08
allocator_frag_bytes:3152584
allocator_rss_ratio:1.18
allocator_rss_bytes:7782400
rss_overhead_ratio:0.91
rss_overhead_bytes:-4304896
mem_fragmentation_ratio:1.17
mem_fragmentation_bytes:6735400
mem_not_counted_for_evict:0
mem_replication_backlog:5105552
mem_total_replication_buffers:5105496
mem_clients_slaves:0
mem_clients_normal:8025168
mem_cluster_links:52224
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4288
rdb_bgsave_in_progress:0
rdb_last_save_time:1705539864
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:5621
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:732
total_commands_processed:156927
instantaneous_ops_per_sec:2201
total_net_input_bytes:37280714
total_net_output_bytes:19055700
total_net_repl_input_bytes:3766103
total_net_repl_output_bytes:10147102
instantaneous_input_kbps:361.14
instantaneous_output_kbps:223.75
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:76.29
rejected_connections:0
sync_full:0
sync_partial_ok:21
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:7
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:121
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:1
dump_payload_sanitizations:0
total_reads_processed:129386
total_writes_processed:136446
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:506
reply_buffer_expands:3
# Replication
role:master
connected_slaves:2
slave0:ip=192.168.70.113,port=6379,state=online,offset=787713026935,lag=1
slave1:ip=192.168.103.141,port=6379,state=online,offset=787713035666,lag=0
master_failover_state:no-failover
master_replid:b6d5bca110dd2b222a26aedc1db670bdcd1b24e7
master_replid2:1999c615a2a94daa14080ce0830a4223625cb144
master_repl_offset:787713041709
second_repl_offset:787707968159
repl_backlog_active:1
repl_backlog_size:209715200
repl_backlog_first_byte_offset:787707968159
repl_backlog_histlen:5073551
# CPU
used_cpu_sys:4.474596
used_cpu_user:3.784102
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:4.469360
used_cpu_user_main_thread:3.782161
# Modules
module:name=udsf,ver=1,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_psync:calls=21,usec=1112,usec_per_call=52.95,rejected_calls=0,failed_calls=0
cmdstat_udsf.getrec:calls=1901,usec=43781,usec_per_call=23.03,rejected_calls=0,failed_calls=0
cmdstat_udsf.setblk:calls=510,usec=6775,usec_per_call=13.28,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=1136,usec=2021,usec_per_call=1.78,rejected_calls=0,failed_calls=0
cmdstat_cluster|slots:calls=487,usec=33378,usec_per_call=68.54,rejected_calls=0,failed_calls=0
cmdstat_cluster|info:calls=42,usec=2247,usec_per_call=53.50,rejected_calls=0,failed_calls=0
cmdstat_cluster|myid:calls=39,usec=32,usec_per_call=0.82,rejected_calls=0,failed_calls=0
cmdstat_cluster|nodes:calls=132,usec=16358,usec_per_call=123.92,rejected_calls=0,failed_calls=0
cmdstat_cluster|failover:calls=1,usec=33,usec_per_call=33.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|forget:calls=13,usec=1684,usec_per_call=129.54,rejected_calls=0,failed_calls=1
cmdstat_cluster|replicate:calls=1,usec=247,usec_per_call=247.00,rejected_calls=0,failed_calls=0
cmdstat_cluster|meet:calls=23,usec=436,usec_per_call=18.96,rejected_calls=0,failed_calls=0
cmdstat_udsf.init:calls=1,usec=1734,usec_per_call=1734.00,rejected_calls=0,failed_calls=0
cmdstat_udsf.lock:calls=88,usec=720,usec_per_call=8.18,rejected_calls=0,failed_calls=0
cmdstat_udsf.setms:calls=26,usec=190,usec_per_call=7.31,rejected_calls=0,failed_calls=0
cmdstat_latency|latest:calls=38,usec=60,usec_per_call=1.58,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=1,usec=8,usec_per_call=8.00,rejected_calls=0,failed_calls=0
cmdstat_get:calls=27,usec=95,usec_per_call=3.52,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=23,usec=79,usec_per_call=3.43,rejected_calls=0,failed_calls=0
cmdstat_client|setname:calls=38,usec=142,usec_per_call=3.74,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=38,usec=41,usec_per_call=1.08,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=38,usec=76,usec_per_call=2.00,rejected_calls=0,failed_calls=0
cmdstat_udsf.setmeta:calls=492,usec=34404,usec_per_call=69.93,rejected_calls=0,failed_calls=0
cmdstat_udsf.getreclk:calls=1506,usec=22516,usec_per_call=14.95,rejected_calls=0,failed_calls=0
cmdstat_udsf.setrec:calls=3985,usec=202623,usec_per_call=50.85,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=38,usec=8380,usec_per_call=220.53,rejected_calls=0,failed_calls=0
cmdstat_config|set:calls=40,usec=341,usec_per_call=8.52,rejected_calls=0,failed_calls=0
cmdstat_info:calls=155,usec=14846,usec_per_call=95.78,rejected_calls=0,failed_calls=0
cmdstat_eval:calls=38,usec=7843,usec_per_call=206.39,rejected_calls=0,failed_calls=0
cmdstat_udsf.search:calls=145954,usec=2110120,usec_per_call=14.46,rejected_calls=0,failed_calls=0
cmdstat_udsf.delrec:calls=95,usec=3818,usec_per_call=40.19,rejected_calls=0,failed_calls=0
# Errorstats
errorstat_ERR:count=1
# Latencystats
latency_percentiles_usec_psync:p50=51.199,p99=74.239,p99.9=74.239
latency_percentiles_usec_udsf.getrec:p50=22.015,p99=45.055,p99.9=57.087
latency_percentiles_usec_udsf.setblk:p50=12.031,p99=32.127,p99.9=45.055
latency_percentiles_usec_replconf:p50=2.007,p99=3.007,p99.9=18.047
latency_percentiles_usec_cluster|slots:p50=67.071,p99=123.391,p99.9=191.487
latency_percentiles_usec_cluster|info:p50=46.079,p99=102.399,p99.9=102.399
latency_percentiles_usec_cluster|myid:p50=1.003,p99=5.023,p99.9=5.023
latency_percentiles_usec_cluster|nodes:p50=119.295,p99=221.183,p99.9=309.247
latency_percentiles_usec_cluster|failover:p50=33.023,p99=33.023,p99.9=33.023
latency_percentiles_usec_cluster|forget:p50=115.199,p99=385.023,p99.9=385.023
latency_percentiles_usec_cluster|replicate:p50=247.807,p99=247.807,p99.9=247.807
latency_percentiles_usec_cluster|meet:p50=18.047,p99=37.119,p99.9=37.119
latency_percentiles_usec_udsf.init:p50=1736.703,p99=1736.703,p99.9=1736.703
latency_percentiles_usec_udsf.lock:p50=7.007,p99=29.055,p99.9=36.095
latency_percentiles_usec_udsf.setms:p50=3.007,p99=52.223,p99.9=52.223
latency_percentiles_usec_latency|latest:p50=2.007,p99=3.007,p99.9=3.007
latency_percentiles_usec_hello:p50=8.031,p99=8.031,p99.9=8.031
latency_percentiles_usec_get:p50=3.007,p99=6.015,p99.9=6.015
latency_percentiles_usec_ping:p50=1.003,p99=12.031,p99.9=12.031
latency_percentiles_usec_client|setname:p50=2.007,p99=15.039,p99.9=15.039
latency_percentiles_usec_slowlog|len:p50=1.003,p99=2.007,p99.9=2.007
latency_percentiles_usec_slowlog|get:p50=2.007,p99=19.071,p99.9=19.071
latency_percentiles_usec_udsf.setmeta:p50=68.095,p99=123.391,p99.9=325.631
latency_percentiles_usec_udsf.getreclk:p50=12.031,p99=46.079,p99.9=61.183
latency_percentiles_usec_udsf.setrec:p50=50.175,p99=127.487,p99.9=176.127
latency_percentiles_usec_config|get:p50=202.751,p99=313.343,p99.9=313.343
latency_percentiles_usec_config|set:p50=7.007,p99=51.199,p99.9=51.199
latency_percentiles_usec_info:p50=15.039,p99=489.471,p99.9=561.151
latency_percentiles_usec_eval:p50=178.175,p99=413.695,p99.9=413.695
latency_percentiles_usec_udsf.search:p50=9.023,p99=63.231,p99.9=87.039
latency_percentiles_usec_udsf.delrec:p50=38.143,p99=62.207,p99.9=80.383
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=6747,expires=0,avg_ttl=0
------ MODULES INFO OUTPUT ------
------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-expire no
lazyfree-lazy-user-del yes
sanitize-dump-payload no
list-compress-depth 0
repl-diskless-sync yes
proto-max-bulk-len 512mb
lazyfree-lazy-user-flush no
client-query-buffer-limit 1gb
replica-read-only yes
lazyfree-lazy-server-del yes
activedefrag no
slave-read-only yes
repl-diskless-load on-empty-db
io-threads 1
lazyfree-lazy-eviction no
io-threads-do-reads no
I0118 01:13:46.581136 1 log.go:16]
------ FAST MEMORY TEST ------
I0118 01:13:46.581310 1 log.go:16] 21:M 18 Jan 2024 01:13:46.581 # Bio thread for job type #0 terminated
I0118 01:13:46.581448 1 log.go:16] 21:M 18 Jan 2024 01:13:46.581 # Bio thread for job type #1 terminated
I0118 01:13:46.581626 1 log.go:16] 21:M 18 Jan 2024 01:13:46.581 # Bio thread for job type #2 terminated
I0118 01:13:46.581686 1 log.go:16] *** Preparing to test memory region 55be7fb80000 (2301952 bytes)
*** Preparing to test memory region 55be80baa000 (557056 bytes)
*** Preparing to test memory region 7f601bf7f000 (48758784 bytes)
I0118 01:13:46.581697 1 log.go:16] *** Preparing to test memory region 7f601ee00000 (8388608 bytes)
*** Preparing to test memory region 7f601f600000 (6291456 bytes)
*** Preparing to test memory region 7f601fdb1000 (8388608 bytes)
I0118 01:13:46.581718 1 log.go:16] *** Preparing to test memory region 7f60205b2000 (8388608 bytes)
*** Preparing to test memory region 7f6020db3000 (8388608 bytes)
*** Preparing to test memory region 7f602177d000 (12288 bytes)
I0118 01:13:46.581730 1 log.go:16] *** Preparing to test memory region 7f6021780000 (11010048 bytes)
I0118 01:13:46.581739 1 log.go:16] *** Preparing to test memory region 7f602236f000 (28672 bytes)
I0118 01:13:46.581757 1 log.go:16] *** Preparing to test memory region 7f6022547000 (16384 bytes)
I0118 01:13:46.581779 1 log.go:16] *** Preparing to test memory region 7f6022569000 (16384 bytes)
*** Preparing to test memory region 7f602285d000 (16384 bytes)
I0118 01:13:46.581824 1 log.go:16] *** Preparing to test memory region 7f6022a3e000 (8192 bytes)
*** Preparing to test memory region 7f6022a6e000 (4096 bytes)
I0118 01:13:46.581836 1 log.go:16] .
I0118 01:13:46.591659 1 log.go:16] O.
I0118 01:13:46.593200 1 log.go:16] O.
I0118 01:13:46.749386 1 log.go:16] O.
I0118 01:13:46.782354 1 log.go:16] O.
I0118 01:13:46.806523 1 log.go:16] O.
I0118 01:13:46.841982 1 log.go:16] O.
I0118 01:13:46.876773 1 log.go:16] O.
I0118 01:13:46.911541 1 log.go:16] O.
I0118 01:13:46.911610 1 log.go:16] O.
I0118 01:13:46.950635 1 log.go:16] O.
I0118 01:13:46.950672 1 log.go:16] O.
I0118 01:13:46.950745 1 log.go:16] O.
I0118 01:13:46.950824 1 log.go:16] O.
I0118 01:13:46.950868 1 log.go:16] O.
I0118 01:13:46.950887 1 log.go:16] O.O
I0118 01:13:46.950896 1 log.go:16]
I0118 01:13:46.950935 1 log.go:16] Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
I0118 01:13:46.951046 1 log.go:16]
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: redis-server 0.0.0.0:6379 [cluster] (base 0x55be7f8b0000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
- OS distribution and version
- Steps to reproduce (if any)