Redis Too many connections request causes very high recv-q & slave never recovers.

``Steps to reproduce :

Redis Infra: using sentinel mode 1 master, 2 slaves

Applications: 2 services with 20 pods(Kubernetes) each

Load tests using relatively higher payloads (~150-200KB), we are performing PUT and GET at almost ~5k/s and 6k/s respectively. After running this load for almost ~20-25 minutes, we restarted one of the slave nodes. Once the slave node is out of sync, we observe that recv-q is quite high at the redis server due to too many connections request Note: We have also observed that this issue only occurs when there are more than 20-25 pods running. When the number of pods is 2-3, the redis slave server recovers and no issues are found on our application level.

When we set max-clients on redis to 1200-1400, we do not face this issue.

Probable reason: Connection leaks issues at the client side: https://github.com/redisson/redisson/issues/4969

Logs : 39045:S 13 Apr 2023 06:18:33.936 * Reconnecting to MASTER 172.16.4.81:6379 39045:S 13 Apr 2023 06:18:33.936 * MASTER <-> REPLICA sync started 39045:S 13 Apr 2023 06:18:33.937 # Error condition on socket for SYNC: (null) 39045:S 13 Apr 2023 06:18:34.573 * Connecting to MASTER 172.16.4.81:6379 39045:S 13 Apr 2023 06:18:34.573 * MASTER <-> REPLICA sync started 39045:S 13 Apr 2023 06:18:35.282 * Non blocking connect for SYNC fired the event. 39045:S 13 Apr 2023 06:18:35.347 * Master replied to PING, replication can continue... 39045:S 13 Apr 2023 06:18:35.354 * Trying a partial resynchronization (request 37191d55918fcc3a594163c04178e1cc84d038ae:33654156680). 39045:S 13 Apr 2023 06:18:35.355 * Full resync from master: da3a310811e940ff3f1b183103bd4ff752864193:0 39045:S 13 Apr 2023 06:18:35.355 * Discarding previously cached master state. 39045:S 13 Apr 2023 06:18:35.410 * MASTER <-> REPLICA sync: receiving 3301 bytes from master to disk 39045:S 13 Apr 2023 06:18:35.410 * MASTER <-> REPLICA sync: Flushing old data 39045:S 13 Apr 2023 06:18:35.410 * MASTER <-> REPLICA sync: Loading DB in memory 39045:S 13 Apr 2023 06:18:35.412 * Loading RDB produced by version 6.2.7 39045:S 13 Apr 2023 06:18:35.412 * RDB age 0 seconds 39045:S 13 Apr 2023 06:18:35.412 * RDB memory usage when created 17.45 Mb 39045:S 13 Apr 2023 06:18:35.412 # Done loading RDB, keys loaded: 4, keys expired: 0. 39045:S 13 Apr 2023 06:18:35.412 * MASTER <-> REPLICA sync: Finished with success 39045:S 13 Apr 2023 06:18:36.724 - Accepted 172.16.0.116:46004 39045:S 13 Apr 2023 06:18:36.730 - Client closed connection 39045:S 13 Apr 2023 06:18:37.591 - DB 0: 4 keys (4 volatile) in 4 slots HT. 39045:S 13 Apr 2023 06:18:37.592 . 281 clients connected (0 replicas), 7715392 bytes in use 39045:S 13 Apr 2023 06:18:42.622 - DB 0: 4 keys (4 volatile) in 4 slots HT. 39045:S 13 Apr 2023 06:18:42.622 . 281 clients connected (0 replicas), 8043088 bytes in use 39045:S 13 Apr 2023 06:18:45.436 - Accepted 172.16.5.1:41842 39045:S 13 Apr 2023 06:18:45.437 - Accepted 172.16.5.1:41848 39045:S 13 Apr 2023 06:18:45.438 - Accepted 172.16.5.1:41852 39045:S 13 Apr 2023 06:18:47.647 - DB 0: 12 keys (12 volatile) in 16 slots HT. 39045:S 13 Apr 2023 06:18:47.647 . 284 clients connected (0 replicas), 8206320 bytes in use 39045:S 13 Apr 2023 06:18:52.678 - DB 0: 29 keys (29 volatile) in 32 slots HT. 39045:S 13 Apr 2023 06:18:52.678 . 284 clients connected (0 replicas), 8649736 bytes in use 39045:S 13 Apr 2023 06:18:57.703 - DB 0: 41 keys (41 volatile) in 64 slots HT. 39045:S 13 Apr 2023 06:18:57.703 . 284 clients connected (0 replicas), 18426392 bytes in use 39045:S 13 Apr 2023 06:19:01.576 - Client closed connection 39045:S 13 Apr 2023 06:19:01.576 - Client closed connection 39045:S 13 Apr 2023 06:19:01.577 - Client closed connection 39045:S 13 Apr 2023 06:19:01.577 - Client closed connection 39045:S 13 Apr 2023 06:19:01.577 - Client closed connection 39045:S 13 Apr 2023 06:19:01.577 - Client closed connection