Hi!

Try cluster reset hard on new nodes, no success. After power outage instance is also lost and cannot work Ports client and data bus available. Tcpdump sees packets from cluster nodes, but after receiving a packet with a list of nodes, the new node does not send a response to the cluster node Number master instance: 25 Nodes: 3 Workload: 150Mb/s Tell me how to get around the problem?

Erorr in log new instance: 8961:M 10 Jun 2021 12:16:12.937 . --- Processing packet of type 2, 2624 bytes 8961:M 10 Jun 2021 12:16:14.648 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 10 Jun 2021 12:16:17.939 . I/O error reading from node link: connection closed 8961:M 10 Jun 2021 12:16:18.813 - Client closed connection 8961:M 10 Jun 2021 12:16:19.657 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 10 Jun 2021 12:16:21.535 - Client closed connection

redis version: 6.0.6

config: port 6001 bind 0.0.0.0 cluster-enabled yes cluster-config-file /var/lib/redis/6001/nodes.conf dir /var/lib/redis/6001 pidfile /var/run/redis/node_6001.pid logfile /var/log/redis/node_6001.log daemonize no activerehashing yes appendonly no client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 0 0 0 client-output-buffer-limit pubsub 33554432 8388608 60 cluster-migration-barrier 1 cluster-node-timeout 5000 cluster-require-full-coverage no cluster-slave-validity-factor 0 hash-max-ziplist-entries 1 hash-max-ziplist-value 3000000 list-max-ziplist-entries 1 list-max-ziplist-value 3000000 loglevel debug maxclients 100000 maxmemory 6GB maxmemory-policy volatile-lru maxmemory-samples 5 rdbcompression no repl-disable-tcp-nodelay no repl-ping-slave-period 10 repl-timeout 60 repl-diskless-sync yes repl-diskless-sync-delay 60 repl-backlog-ttl 0 repl-backlog-size 128MB set-max-intset-entries 512 slave-priority 100 slave-read-only yes slave-serve-stale-data yes timeout 0 zset-max-ziplist-entries 1 zset-max-ziplist-value 3000000

Comment From: huangzhw

How do you add a node?

Comment From: nevgin

run command on cluster node: cluster meet ip_new_node port

Comment From: huangzhw

Are this full logs? After Processing packet of type2 should be

GOSSIP c603b8650e59e45f3dcdfee6d325f5fa9cc3ff5f 127.0.0.1:7003@17003 slave

Comment From: nevgin

After Processing packet of type 2 - I/O error reading from node link: connection closed

8961:M 11 Jun 2021 14:48:43.972 . 1 clients connected (0 replicas), 6841672 bytes in use 8961:M 11 Jun 2021 14:48:46.342 . --- Processing packet of type 2, 2624 bytes 8961:M 11 Jun 2021 14:48:48.980 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 11 Jun 2021 14:48:53.991 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 11 Jun 2021 14:48:58.998 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 11 Jun 2021 14:49:01.349 . I/O error reading from node link: connection closed 8961:M 11 Jun 2021 14:49:04.012 . 0 clients connected (0 replicas), 6820736 bytes in use

Comment From: huangzhw

It's like your new node is already part of the cluster. You send cluster meet, new node reply PONG, then old node found this node already exist, so close the connection. Then new node found connection closed. Can you confirm it?

Comment From: nevgin

It's like your new node is already part of the cluster. You send cluster meet, new node reply PONG, then old node found this node already exist, so close the connection. Then new node found connection closed. Can you confirm it?

No. I am run on new node cluster reset hard before add node. In output cluster nodes on cluster node, also no new node ip.

Comment From: huangzhw

It's weird. Can you post the logs of old and new nodes with no reduced and the CLUSTER NODES output of the old and new nodes before and after.

Comment From: nevgin

It's weird. Can you post the logs of old and new nodes with no reduced and the CLUSTER NODES output of the old and new nodes before and after.

Before adding: cluster nodes of new node: 58aa5df1b2339fe6bac86a7118ed2af3a66b8c90 :6901@16901 myself,master - 0 0 0 connected cluster nodes of old node: d5bb7fb353ceb9ff73439fbb92c45c53af863b22 192.168.38.26:6910 slave 07a3709636342e0fc80feb29767edf76d615e636 0 1623832120885 103 connected d92feca0c6ab39b156808b6b66717bb857169d77 192.168.214.2:6909 slave cbcbc13160bf179c239802c2c6b91a3f18d9dc90 0 1623832119386 125 connected 24f30dc1f8aa9d157b8f90744655b3619a133b21 192.168.149.9:6910 master - 0 1623832116384 92 connected 6554-7208 e68e20cd592569027b3ca6bd6d72c6c14bad22d4 192.168.38.26:6908 slave 9b86439e6aac5f835d1956f5cd2c8751a003eaa3 0 1623832120385 106 connected 26f6a4cabc86dd9ffb18edb6a4c5abb20de8adb5 192.168.214.2:6914 master - 0 1623832120885 123 connected 13763-14417 eea81c4f2a119a53f5855be3265150d716a44b48 192.168.214.2:6912 slave 945129e3e5f71829926d56dbeeec0d5d53052ca8 0 1623832122986 119 connected 9b86439e6aac5f835d1956f5cd2c8751a003eaa3 192.168.214.2:6908 master - 0 1623832118584 106 connected 3932-4587 cc229eb1b9d508138554a0d04d7c8ebba8b4eea0 192.168.38.26:6902 master - 0 1623832119985 115 connected 5243-5897 6b9c06949599e66070fb34dff7fc4a223151b9f2 192.168.38.26:6911 master - 0 1623832118985 118 connected 9175-9829 63a4be5d8868af5f69ca6461298de43943dd50fc 192.168.214.2:6903 slave 6b9c06949599e66070fb34dff7fc4a223151b9f2 0 1623832118584 118 connected 0f914f67275620d824274dd5dc7c6cd87f9f55d2 192.168.149.9:6914 master - 0 1623832116884 82 connected 13107-13762 5d521ee4d56a68c66d83adcd3a8b8ae72520a1fa 192.168.149.9:6911 slave 8d47de04ee5c37e6597d43ea622c57db119feb0a 0 1623832115984 104 connected 4ba15af3161a2c52e5ae06fa555f6550e62c8b08 192.168.38.26:6903 slave 565b6e6130f240f060c6d708f1fc1854cbf2900f 0 1623832116884 124 connected cbcbc13160bf179c239802c2c6b91a3f18d9dc90 192.168.149.9:6912 master - 0 1623832118884 125 connected 9830-10485 d1f4f9552e81bd734c4b30de3fad2ef230768bfb 192.168.149.9:6908 master - 0 1623832121385 126 connected 3277-3931 945129e3e5f71829926d56dbeeec0d5d53052ca8 192.168.38.26:6912 master - 0 1623832121485 119 connected 10486-11140 2c3af1506f4f02da5c7919d3412b3815aa4d31dc 192.168.214.2:6901 master - 0 1623832118484 109 connected 2621-3276 9b171df4e30b37ed31256779999b59ac3a6edb8e 192.168.214.2:6902 master - 0 1623832122986 102 connected 5898-6553 a572b43bd8bcd092cee1ae9395de79dd26f9b994 192.168.149.9:6905 master - 0 1623832117884 87 connected 14418-15072 cca21a71d2eb6ff4b654176855dc72fcf680ea8f 192.168.214.2:6915 slave f861b97a944a88b10e4634671252ef442dff4f93 0 1623832115883 117 connected a83052f763dc894770852eca3b24c3855a20fb88 192.168.38.26:6904 master - 0 1623832118384 116 connected 11796-12451 870b9a6c37f86086f0e562f536f5883554ed616b 192.168.38.26:6913 master - 0 1623832118484 120 connected 12452-13106 a9642eaafb07414e21fab4a8dcf8c514287e6830 192.168.38.26:6901 master - 0 1623832121385 114 connected 1966-2620 dd4e675d02989f2f71083ac58f4340d8a7846095 192.168.149.9:6909 slave cc229eb1b9d508138554a0d04d7c8ebba8b4eea0 0 1623832120885 115 connected 9702c1da4c623e9f0561fc56e7fd4a3f9726f02f 192.168.149.9:6902 master - 0 1623832120385 93 connected 4588-5242 40bfbf17dd580884ae3970f330da11b9c0480e51 192.168.38.26:6914 slave 26f6a4cabc86dd9ffb18edb6a4c5abb20de8adb5 0 1623832118884 123 connected e0d5c28d72b334bc2e91f94d548882ca497b6c4d 192.168.38.26:6906 slave fface65f6d01449b2a8aed49bbb1aaed6722c700 0 1623832122485 121 connected 569a45ec30923e7707af7cc9af58232126d23441 192.168.149.9:6915 slave befd44f8eaa207648b411a74c827bd13e0a7a677 0 1623832122485 117 connected 0a00da1d7c7d6182f7a9f301ca703300326505e5 192.168.214.2:6913 slave 9702c1da4c623e9f0561fc56e7fd4a3f9726f02f 0 1623832120386 116 connected 376e62daa3581e56e8f0850d66879ff04c3a8fdc 192.168.38.26:6909 slave 9b171df4e30b37ed31256779999b59ac3a6edb8e 0 1623832118985 102 connected aa4eb6a315cf900c28d996d8d0d4dadf61b39b3e 192.168.214.2:6907 slave a9642eaafb07414e21fab4a8dcf8c514287e6830 0 1623832117384 114 connected fface65f6d01449b2a8aed49bbb1aaed6722c700 192.168.214.2:6906 master - 0 1623832117384 121 connected 655-1310 565b6e6130f240f060c6d708f1fc1854cbf2900f 192.168.149.9:6904 master - 0 1623832116384 124 connected 11141-11795 18d7013b289a1caa1ace87d7fb9652f8d5d24937 192.168.214.2:6904 slave 870b9a6c37f86086f0e562f536f5883554ed616b 0 1623832116885 120 connected f861b97a944a88b10e4634671252ef442dff4f93 192.168.38.26:6905 master - 0 1623832116385 117 connected 15073-15728 bddc0df6edff386b0bf01b4c0c39f99f93504dc0 192.168.149.9:6913 slave a83052f763dc894770852eca3b24c3855a20fb88 0 1623832122385 116 connected 95cca49ba05f65c4faa7d43d7f59d44e726be982 192.168.149.9:6906 master - 0 1623832115984 122 connected 0-654 b4f16e05ea7be1f4ae0d4e08c0e2d996065b5562 192.168.214.2:6905 master - 0 1623832121585 110 connected 15729-16383 4db0dd49532994a6e27b3b54cc62ee67684c1cc9 192.168.149.9:6907 slave 95cca49ba05f65c4faa7d43d7f59d44e726be982 0 1623832122385 122 connected fe372553eaead0595f041bc265149ecbd0c619df 192.168.38.26:6915 slave b4f16e05ea7be1f4ae0d4e08c0e2d996065b5562 0 1623832122886 110 connected 8d47de04ee5c37e6597d43ea622c57db119feb0a 192.168.214.2:6911 master - 0 1623832119385 104 connected 8520-9174 befd44f8eaa207648b411a74c827bd13e0a7a677 192.168.149.9:6901 myself,master - 0 0 88 connected 1311-1965 07a3709636342e0fc80feb29767edf76d615e636 192.168.214.2:6910 master - 0 1623832119985 103 connected 7209-7863 c8731591d0271d622181ec83072a2ba1b8055776 192.168.149.9:6903 master - 0 1623832121885 86 connected 7864-8519 f38cd3e183b32c20a152fdcc29e3f91624827d6b 192.168.38.26:6907 slave 2c3af1506f4f02da5c7919d3412b3815aa4d31dc 0 1623832120885 109 connected

After adding new node: 58aa5df1b2339fe6bac86a7118ed2af3a66b8c90 :6901@16901 myself,master - 0 0 0 connected old node: d5bb7fb353ceb9ff73439fbb92c45c53af863b22 192.168.38.26:6910 slave 07a3709636342e0fc80feb29767edf76d615e636 0 1623832451606 103 connected d92feca0c6ab39b156808b6b66717bb857169d77 192.168.214.2:6909 slave cbcbc13160bf179c239802c2c6b91a3f18d9dc90 0 1623832449606 125 connected 24f30dc1f8aa9d157b8f90744655b3619a133b21 192.168.149.9:6910 master - 0 1623832450606 92 connected 6554-7208 e68e20cd592569027b3ca6bd6d72c6c14bad22d4 192.168.38.26:6908 slave 9b86439e6aac5f835d1956f5cd2c8751a003eaa3 0 1623832454710 106 connected 26f6a4cabc86dd9ffb18edb6a4c5abb20de8adb5 192.168.214.2:6914 master - 0 1623832448606 123 connected 13763-14417 eea81c4f2a119a53f5855be3265150d716a44b48 192.168.214.2:6912 slave 945129e3e5f71829926d56dbeeec0d5d53052ca8 0 1623832453609 119 connected 9b86439e6aac5f835d1956f5cd2c8751a003eaa3 192.168.214.2:6908 master - 0 1623832452709 106 connected 3932-4587 cc229eb1b9d508138554a0d04d7c8ebba8b4eea0 192.168.38.26:6902 master - 0 1623832452107 115 connected 5243-5897 6b9c06949599e66070fb34dff7fc4a223151b9f2 192.168.38.26:6911 master - 0 1623832449606 118 connected 9175-9829 63a4be5d8868af5f69ca6461298de43943dd50fc 192.168.214.2:6903 slave 6b9c06949599e66070fb34dff7fc4a223151b9f2 0 1623832453610 118 connected 0f914f67275620d824274dd5dc7c6cd87f9f55d2 192.168.149.9:6914 master - 0 1623832450106 82 connected 13107-13762 5d521ee4d56a68c66d83adcd3a8b8ae72520a1fa 192.168.149.9:6911 slave 8d47de04ee5c37e6597d43ea622c57db119feb0a 0 1623832449706 104 connected 4ba15af3161a2c52e5ae06fa555f6550e62c8b08 192.168.38.26:6903 slave 565b6e6130f240f060c6d708f1fc1854cbf2900f 0 1623832452709 124 connected cbcbc13160bf179c239802c2c6b91a3f18d9dc90 192.168.149.9:6912 master - 0 1623832450606 125 connected 9830-10485 d1f4f9552e81bd734c4b30de3fad2ef230768bfb 192.168.149.9:6908 master - 0 1623832448606 126 connected 3277-3931 3fd5a8e2fe112deceb9f443d7884714738e69031 192.168.161.4:6901 handshake - 0 0 0 connected 945129e3e5f71829926d56dbeeec0d5d53052ca8 192.168.38.26:6912 master - 0 1623832448306 119 connected 10486-11140 2c3af1506f4f02da5c7919d3412b3815aa4d31dc 192.168.214.2:6901 master - 0 1623832451707 109 connected 2621-3276 9b171df4e30b37ed31256779999b59ac3a6edb8e 192.168.214.2:6902 master - 0 1623832453209 102 connected 5898-6553 a572b43bd8bcd092cee1ae9395de79dd26f9b994 192.168.149.9:6905 master - 0 1623832448106 87 connected 14418-15072 cca21a71d2eb6ff4b654176855dc72fcf680ea8f 192.168.214.2:6915 slave f861b97a944a88b10e4634671252ef442dff4f93 0 1623832449206 117 connected a83052f763dc894770852eca3b24c3855a20fb88 192.168.38.26:6904 master - 0 1623832450206 116 connected 11796-12451 870b9a6c37f86086f0e562f536f5883554ed616b 192.168.38.26:6913 master - 0 1623832448906 120 connected 12452-13106 a9642eaafb07414e21fab4a8dcf8c514287e6830 192.168.38.26:6901 master - 0 1623832450206 114 connected 1966-2620 dd4e675d02989f2f71083ac58f4340d8a7846095 192.168.149.9:6909 slave cc229eb1b9d508138554a0d04d7c8ebba8b4eea0 0 1623832451606 115 connected 9702c1da4c623e9f0561fc56e7fd4a3f9726f02f 192.168.149.9:6902 master - 0 1623832450206 93 connected 4588-5242 40bfbf17dd580884ae3970f330da11b9c0480e51 192.168.38.26:6914 slave 26f6a4cabc86dd9ffb18edb6a4c5abb20de8adb5 0 1623832454710 123 connected e0d5c28d72b334bc2e91f94d548882ca497b6c4d 192.168.38.26:6906 slave fface65f6d01449b2a8aed49bbb1aaed6722c700 0 1623832451107 121 connected 569a45ec30923e7707af7cc9af58232126d23441 192.168.149.9:6915 slave befd44f8eaa207648b411a74c827bd13e0a7a677 0 1623832451606 117 connected 0a00da1d7c7d6182f7a9f301ca703300326505e5 192.168.214.2:6913 slave 9702c1da4c623e9f0561fc56e7fd4a3f9726f02f 0 1623832448207 116 connected 376e62daa3581e56e8f0850d66879ff04c3a8fdc 192.168.38.26:6909 slave 9b171df4e30b37ed31256779999b59ac3a6edb8e 0 1623832450707 102 connected aa4eb6a315cf900c28d996d8d0d4dadf61b39b3e 192.168.214.2:6907 slave a9642eaafb07414e21fab4a8dcf8c514287e6830 0 1623832452209 114 connected fface65f6d01449b2a8aed49bbb1aaed6722c700 192.168.214.2:6906 master - 0 1623832447607 121 connected 655-1310 565b6e6130f240f060c6d708f1fc1854cbf2900f 192.168.149.9:6904 master - 0 1623832454409 124 connected 11141-11795 18d7013b289a1caa1ace87d7fb9652f8d5d24937 192.168.214.2:6904 slave 870b9a6c37f86086f0e562f536f5883554ed616b 0 1623832452610 120 connected f861b97a944a88b10e4634671252ef442dff4f93 192.168.38.26:6905 master - 0 1623832452208 117 connected 15073-15728 bddc0df6edff386b0bf01b4c0c39f99f93504dc0 192.168.149.9:6913 slave a83052f763dc894770852eca3b24c3855a20fb88 0 1623832450606 116 connected 95cca49ba05f65c4faa7d43d7f59d44e726be982 192.168.149.9:6906 master - 0 1623832448206 122 connected 0-654 b4f16e05ea7be1f4ae0d4e08c0e2d996065b5562 192.168.214.2:6905 master - 0 1623832454609 110 connected 15729-16383 4db0dd49532994a6e27b3b54cc62ee67684c1cc9 192.168.149.9:6907 slave 95cca49ba05f65c4faa7d43d7f59d44e726be982 0 1623832452609 122 connected fe372553eaead0595f041bc265149ecbd0c619df 192.168.38.26:6915 slave b4f16e05ea7be1f4ae0d4e08c0e2d996065b5562 0 1623832449606 110 connected 8d47de04ee5c37e6597d43ea622c57db119feb0a 192.168.214.2:6911 master - 0 1623832454609 104 connected 8520-9174 befd44f8eaa207648b411a74c827bd13e0a7a677 192.168.149.9:6901 myself,master - 0 0 88 connected 1311-1965 07a3709636342e0fc80feb29767edf76d615e636 192.168.214.2:6910 master - 0 1623832451206 103 connected 7209-7863 c8731591d0271d622181ec83072a2ba1b8055776 192.168.149.9:6903 master - 0 1623832450206 86 connected 7864-8519 f38cd3e183b32c20a152fdcc29e3f91624827d6b 192.168.38.26:6907 slave 2c3af1506f4f02da5c7919d3412b3815aa4d31dc 0 1623832448706 109 connected

Logs: new node: 8961:M 16 Jun 2021 12:15:55.495 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:00.509 . 1 clients connected (0 replicas), 6882632 bytes in use 8961:M 16 Jun 2021 12:16:05.515 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:10.528 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:15.538 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:20.551 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:25.557 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:30.562 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:35.570 . 1 clients connected (0 replicas), 6841672 bytes in use 8961:M 16 Jun 2021 12:16:40.579 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:45.586 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:50.593 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:16:55.609 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:00.621 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:05.630 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:10.638 . 1 clients connected (0 replicas), 6882632 bytes in use 8961:M 16 Jun 2021 12:17:15.646 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:20.657 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:25.664 . 1 clients connected (0 replicas), 6841672 bytes in use 8961:M 16 Jun 2021 12:17:30.674 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:35.679 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:40.690 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:45.704 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:17:50.393 - Accepting cluster node connection from 192.168.149.9:13018 8961:M 16 Jun 2021 12:17:50.393 . --- Processing packet of type 2, 2624 bytes 8961:M 16 Jun 2021 12:17:50.714 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 16 Jun 2021 12:17:55.720 . 1 clients connected (0 replicas), 6882752 bytes in use 8961:M 16 Jun 2021 12:17:57.590 - Client closed connection 8961:M 16 Jun 2021 12:18:00.729 . 0 clients connected (0 replicas), 6820856 bytes in use 8961:M 16 Jun 2021 12:18:05.394 . I/O error reading from node link: connection closed 8961:M 16 Jun 2021 12:18:05.738 . 1 clients connected (0 replicas), 6882664 bytes in use 8961:M 16 Jun 2021 12:18:10.747 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:18:15.749 . 1 clients connected (0 replicas), 6882688 bytes in use 8961:M 16 Jun 2021 12:18:20.752 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:18:25.765 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:18:30.778 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:18:35.791 . 1 clients connected (0 replicas), 6882688 bytes in use 8961:M 16 Jun 2021 12:18:40.802 . 0 clients connected (0 replicas), 6820736 bytes in use 8961:M 16 Jun 2021 12:18:45.814 . 0 clients connected (0 replicas), 6820736 bytes in use

Old node: with notice level - empty with debug level - no entry with IP new node

Comment From: huangzhw

I think it's Bug which was fixed by https://github.com/redis/redis/pull/7958. You can update to newest version.