Redis [BUG] Bad message length or signature received from Cluster bus & quorum reached

Describe the bug

Increased resources for redis-cluster & redis-exporter. Uninstall & install again redis-cluster completely.

Still, I faced some similar issues and it might relates to resources but I've no really idea what is the root cause for getting this error:

1:M 29 Sep 2022 14:52:25.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:25.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:25.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:25.003 # Bad message length or signature received from Cluster bus.

To reproduce

Run redis-cluster of bitnami/redis-cluster:6.2.4.0

Expected behavior

Run like it should be without errors.

Additional information

1:M 29 Sep 2022 14:52:09.997 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 29 Sep 2022 14:52:09.997 # Server initialized
1:M 29 Sep 2022 14:52:10.000 * Reading RDB preamble from AOF file...
1:M 29 Sep 2022 14:52:10.000 * Loading RDB produced by version 6.2.4
1:M 29 Sep 2022 14:52:10.000 * RDB age 13606 seconds
1:M 29 Sep 2022 14:52:10.000 * RDB memory usage when created 27.48 Mb
1:M 29 Sep 2022 14:52:10.000 * RDB has an AOF tail
1:M 29 Sep 2022 14:52:10.077 * Reading the remaining AOF tail...
1:M 29 Sep 2022 14:52:10.287 * DB loaded from append only file: 0.289 seconds
1:M 29 Sep 2022 14:52:10.287 * Ready to accept connections
1:M 29 Sep 2022 14:52:15.001 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:19.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:20.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:25.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:29.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:30.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:35.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:39.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:40.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:40.335 * Clear FAIL state for node 1f36f2dde0ac35466ddf12212cf10a218831b3f6: is reachable again and nobody is serving its slots after some time.
1:M 29 Sep 2022 14:52:40.335 # Cluster state changed: ok
1:M 29 Sep 2022 14:52:45.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:49.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:50.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:55.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:52:59.218 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:00.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:00.427 * Marking node 8679627cdd105319f0c298ed0846c98a6f8b77c8 as failing (quorum reached).
1:M 29 Sep 2022 14:53:00.427 # Cluster state changed: fail
1:M 29 Sep 2022 14:53:05.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:09.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:10.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:10.070 * 10000 changes in 60 seconds. Saving...
1:M 29 Sep 2022 14:53:10.071 * Background saving started by pid 319
319:C 29 Sep 2022 14:53:10.094 * DB saved on disk
319:C 29 Sep 2022 14:53:10.095 * RDB: 0 MB of memory used by copy-on-write
1:M 29 Sep 2022 14:53:10.173 * Background saving terminated with success
1:M 29 Sep 2022 14:53:15.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:19.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:19.209 # Address updated for node 8679627cdd105319f0c298ed0846c98a6f8b77c8, now 240b:c0e0:204:5400:b414:2:0:5752:6379
1:M 29 Sep 2022 14:53:20.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:25.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:29.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:30.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:30.577 * Clear FAIL state for node 8679627cdd105319f0c298ed0846c98a6f8b77c8: is reachable again and nobody is serving its slots after some time.
1:M 29 Sep 2022 14:53:30.577 # Cluster state changed: ok
1:M 29 Sep 2022 14:53:35.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:39.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:40.006 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:43.636 * Marking node 1f36f2dde0ac35466ddf12212cf10a218831b3f6 as failing (quorum reached).
1:M 29 Sep 2022 14:53:43.636 # Cluster state changed: fail
1:M 29 Sep 2022 14:53:45.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:49.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:50.003 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:55.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:53:55.443 # Address updated for node 1f36f2dde0ac35466ddf12212cf10a218831b3f6, now 240b:c0e0:204:5400:b414:2:0:6a24:6379
1:M 29 Sep 2022 14:53:59.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:54:00.004 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:54:05.002 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:54:09.206 # Bad message length or signature received from Cluster bus.
1:M 29 Sep 2022 14:54:10.002 # Bad message length or signature received from Cluster bus.

Please help fellas!

Comment From: madolson

That is a pretty delightfully useless error message.

We almost always see these errors when some external actor is trying to send messages to the clusterbus. All cluster bus messages must start with 'RCmb' and be of a given length. However, the failures of the cluster are kind of weird. Did the cluster eventually stabilize?

Comment From: n1vgabay

@madolson The only change I made is basically enabling the Redis exporter as part of the statefulset. So when metrics are enabled, then I started to notice those messages.

It does affect our platform so I don't think there is no meaning for those messages. redis isn't functioning as it used before.

Comment From: madolson

@n1vgabay This exporter? https://github.com/oliver006/redis_exporter

It does affect our platform so I don't think there is no meaning for those messages. redis isn't functioning as it used before.

My guess is that it has nothing to do with Redis, and for some reason the exporter is sending data to the Redis clusterbus port.

Comment From: madolson

As a follow up to this, I think we should add a print statement here to indicate which IP/port is sending these messages. That way we can at least check to see if it's a node in the cluster or some other nefarious node.

Comment From: enjoy-binbin

@madolson Is this kind of output ok?

15731:S 02 Nov 2022 10:46:21.570 # Bad message length or signature received on the Cluster bus from 198.235.24.18:58868
15731:S 02 Nov 2022 10:53:43.206 # Bad message length or signature received on the Cluster bus from 180.101.244.11:32978
15731:S 02 Nov 2022 10:53:43.774 # Bad message length or signature received on the Cluster bus from 180.101.245.246:37640
15731:S 02 Nov 2022 10:53:43.899 # Bad message length or signature received on the Cluster bus from 220.196.160.117:58744
15731:S 02 Nov 2022 10:53:44.003 # Bad message length or signature received on the Cluster bus from 59.83.208.105:24878
15731:S 02 Nov 2022 10:53:44.104 # Bad message length or signature received on the Cluster bus from 180.101.244.15:33072
15731:S 02 Nov 2022 10:53:44.220 # Bad message length or signature received on the Cluster bus from 180.101.245.251:26738
15731:S 02 Nov 2022 10:53:44.320 # Bad message length or signature received on the Cluster bus from 59.83.208.105:25046
15731:S 02 Nov 2022 10:53:44.417 # Bad message length or signature received on the Cluster bus from 220.196.160.84:53780
15731:S 02 Nov 2022 10:53:44.510 # Bad message length or signature received on the Cluster bus from 59.83.208.104:55362
15731:S 02 Nov 2022 10:53:44.642 # Bad message length or signature received on the Cluster bus from 220.196.160.125:39512

Comment From: madolson

@enjoy-binbin LGTM

Comment From: madolson

As far as I can tell this isn't on us any more, so marking this as to be closed.