Trying to upgrade our Redis Lab Cluster from 7.0 to 7.2 but nodes keep crashing on startup Have also tried to set up a completely new replica, there redis 7.2 works fine until i try to add the node to the cluster, then it also crashes.
After downgrading to 7.0.12 the nodes works fine again - so seems to something specific to 7.2
Additional information OS: Ubuntu 20.04 HW: Virtual (VmWare)
Bug report:
=== REDIS BUG REPORT START: Cut & paste starting from here ===
Redis 7.2.0 crashed by signal: 11, si_code: 1
Accessing address: (nil)
Crashed running the instruction at: 0x562d0d6623fb
------ REGISTERS ------
RAX:3966386237303330 RBX:00007f241e76fa00
RCX:0000000000000001 RDX:3235306133326465
RDI:00007f241e76fa00 RSI:0000000000000000
RBP:0000000000000000 RSP:00007ffcb3f46590
R8 :0000000064dec2ad R9 :000000000ff7e624
R10:00007ffcb3fdf080 R11:00007f241e770400
R12:00007f241e76fa30 R13:0000018a06287589
R14:00000000000000b9 R15:00007f241e76fa00
RIP:0000562d0d6623fb EFL:0000000000010206
CSGSFS:002b000000000033
(00007ffcb3f4659f) -> 0000000000000400
(00007ffcb3f4659e) -> 00007ffcb3f466b0
(00007ffcb3f4659d) -> ddd7f1f3918dd200
(00007ffcb3f4659c) -> 0000562d0dfd90e0
(00007ffcb3f4659b) -> ddd7f1f3918dd200
(00007ffcb3f4659a) -> 00007ffcb3f466b0
(00007ffcb3f46599) -> 00007f241f05caad
(00007ffcb3f46598) -> 00007f241ea1bb60
(00007ffcb3f46597) -> 0000562d0d8660e0
(00007ffcb3f46596) -> 0000562d0d8660e0
(00007ffcb3f46595) -> 0000000000000001
(00007ffcb3f46594) -> 0000562d0d8660e0
(00007ffcb3f46593) -> 0000562d0d65cbd6
(00007ffcb3f46592) -> 00007f24060cd000
(00007ffcb3f46591) -> 00007f240b41c7d0
(00007ffcb3f46590) -> 00007f24060cd050
------ INFO OUTPUT ------
# Server
redis_version:7.2.0
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:3a1b7000f8371834
redis_mode:cluster
os:Linux 5.4.0-155-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.4.0
process_id:3540560
process_supervised:no
run_id:66c7f64c1fd1a8e265b162e3a810759c6cea4385
tcp_port:6379
server_time_usec:1692320429449515
uptime_in_seconds:1
uptime_in_days:0
hz:100
configured_hz:100
lru_clock:14598829
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
listener2:name=tls,bind=127.0.0.1,bind=10.3.48.124,port=6379
# Clients
connected_clients:0
cluster_connections:16
maxclients:1000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
# Memory
used_memory:305765264
used_memory_human:291.60M
used_memory_rss:318922752
used_memory_rss_human:304.15M
used_memory_peak:305765264
used_memory_peak_human:291.60M
used_memory_peak_perc:100.00%
used_memory_overhead:34522872
used_memory_startup:1268264
used_memory_dataset:271242392
used_memory_dataset_perc:89.08%
allocator_allocated:302066304
allocator_active:302301184
allocator_resident:311668736
total_system_memory:16742162432
total_system_memory_human:15.59G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:234880
allocator_rss_ratio:1.03
allocator_rss_bytes:9367552
rss_overhead_ratio:1.02
rss_overhead_bytes:7254016
mem_fragmentation_ratio:1.06
mem_fragmentation_bytes:17047752
mem_not_counted_for_evict:8
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:16008
mem_aof_buffer:8
mem_allocator:jemalloc-5.3.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1692320428
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:518644
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:197591285
aof_base_size:197591285
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:0
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:518777
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:2
eventloop_duration_sum:58115
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:0
instantaneous_eventloop_duration_usec:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:d52983f364f9364b19f2c10cd846de059fd39830
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.153794
used_cpu_user:1.183030
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.153781
used_cpu_user_main_thread:1.182937
# Modules
module:name=search,ver=20804,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_info:calls=1,usec=21,usec_per_call=21.00,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=1,usec=8,usec_per_call=8.00,rejected_calls=0,failed_calls=0
# Errorstats
# Latencystats
latency_percentiles_usec_info:p50=21.119,p99=21.119,p99.9=21.119
latency_percentiles_usec_config|get:p50=8.031,p99=8.031,p99.9=8.031
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=518644,expires=0,avg_ttl=0
# Cluster info
cluster_state:fail
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:1
cluster_current_epoch:187
cluster_my_epoch:185
cluster_stats_messages_pong_sent:1
cluster_stats_messages_sent:1
cluster_stats_messages_ping_received:1
cluster_stats_messages_received:1
total_cluster_links_buffer_limit_exceeded:0
------ CLUSTER NODES OUTPUT ------
3b4c34db7a673629d5acafa3fd2f2d6de0212ade 10.3.48.128:0@16379,,tls-port=6379,shard-id=6d5d1e470c2147de434f691196176fc95e50ca75 slave
de9123b19a3250036e53e65da8cd03e2e8ac7932 0 1692320428064 186 disconnected
e2163f560a2e89955dd222d940f32d2917ed1a30 10.3.48.134:0@16379,,tls-port=6379,shard-id=1351b2c3cee680e1cdb199d4ab7364211b7a32c2 slave
5c7101d30e77b341d923539492998fcadaca6392 0 1692320428064 187 disconnected
8544cf7802fb5fc751c02029f553af7737c12f39 10.3.48.124:0@16379,,tls-port=6379,shard-id=031b66292ffa165413bed5d2ed51459ca2cf0ad7 myself,slave
12ec951aeceeb2803feece639410231615b3dac3 0 1692320428064 185 connected
5c7101d30e77b341d923539492998fcadaca6392 10.3.48.112:0@16379,,tls-port=6379,shard-id=a69d91dcac934e26e4562902e67cf43312964d2a master - 0 1692320428064 1
10923-16383
9c25abd4846d829426b1c439e28148654a5fd5bd 10.3.48.63:0@16379,,tls-port=6379,shard-id=0307b8f9ed23a052f0b87fba06270e9ab23fc8f2 slave 12ec951aeceeb2803feec
0 1692320428064 185 disconnected
5e93950692ef5dcc1f492e661f16880eeb127471 10.3.48.127:0@16379,,tls-port=6379,shard-id=44f1d80663b0b248978219969da897543080a7c5 slave
de9123b19a3250036e53e65da8cd03e2e8ac7932 0 1692320428064 186 disconnected
4f3ebae03f901cfb4a48f2fee01be2b99ee277d4 10.3.48.69:0@16379,,tls-port=6379,shard-id=427ae9b57255bd7e2b067887e73eac362d12345f slave 5c7101d30e77b341d9235
0 1692320428064 187 disconnected
12ec951aeceeb2803feece639410231615b3dac3 10.3.48.125:0@16379,,tls-port=6379,shard-id=f8fda916f49e56e8b866b8b874e4b66a38aeccf6 master - 0 1692320428064 1
0-5460
de9123b19a3250036e53e65da8cd03e2e8ac7932 10.3.48.66:0@16379,,tls-port=6379,shard-id=673c6db8a7d82eba1374b8d0bcab881605df5cdc master - 0 1692320428064 18
5461-10922
------ CLIENT LIST OUTPUT ------
------ MODULES INFO OUTPUT ------
# search_version
search_version:2.8.4
search_redis_version:7.2.0 - oss
# search_index
search_number_of_indexes:2
# search_fields_statistics
search_fields_numeric:Numeric=10,Sortable=2
search_fields_tag:Tag=2
# search_dialect_statistics
search_dialect_1:0
search_dialect_2:0
search_dialect_3:0
search_dialect_4:0
# search_runtime_configurations
search_concurrent_mode:OFF
search_enableGC:ON
search_minimal_term_prefix:2
search_maximal_prefix_expansions:200
search_query_timeout_ms:500
search_timeout_policy:return
search_cursor_read_size:1000
search_cursor_max_idle_time:300000
search_max_doc_table_size:1000000
search_max_search_results:1000000
search_max_aggregate_results:-1
search_search_pool_size:20
search_index_pool_size:8
search_gc_scan_size:100
search_min_phonetic_term_length:3
------ CONFIG DEBUG OUTPUT ------
lazyfree-lazy-user-flush no
repl-diskless-load disabled
slave-read-only yes
list-compress-depth 0
sanitize-dump-payload no
activedefrag no
replica-read-only yes
lazyfree-lazy-user-del no
repl-diskless-sync no
proto-max-bulk-len 512mb
io-threads 1
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
io-threads-do-reads no
lazyfree-lazy-server-del no
lazyfree-lazy-expire no
------ FAST MEMORY TEST ------
Bio worker thread #0 terminated
Bio worker thread #1 terminated
Bio worker thread #2 terminated
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/bin/redis-server 127.0.0.1:6379 [cluster] (base 0x562d0d568000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: Believemenot
i have same problem on updating 7.0.12 to 7.2 on working sharded cluster. on non sharded cluster upgrade from 7.0.12 to 7.2 works well