Hi,
Today when using Redis Cluster, each and every published message will be broadcasted to every node in the cluster, making it pretty much unusable in large scale.
My request is to add a config option, something like cluster-pubsub-propagate that can be disabled so message will only be published within the same cluster node.
From my very little understanding, its only wrapping this line: https://github.com/antirez/redis/blob/unstable/src/pubsub.c#L320 in another if statement.
Once that will be done, it will be possible for clients to distribute message according to the already existing cluster slots and by that making sure messages go to the correct node.
Comment From: slact
+1. I'm building a redis-cluster based queuing pub/sub system for Nchan, and cannot use a separate cluster for pub/sub because messages must be stored and published atomically. This is done via a lua script that stores the message and runs the PUBLISH command. So a separate pub/sub cluster is not an option if I want to use redis to create a guaranteed delivery pub/sub system, and the way it is built now with auto-propagation to every node is not very scalable. This option seems to me the simplest solution to this problem.
Comment From: slact
I've added this config option (sans tests) in this commit. You can experiment with it by building from my fork's branch cluster-pubsub-propagate.
Comment From: shaharmor
@antirez any chance to get some feedback on this?
Comment From: manigandham
Isn't this now solved by the Streams data type?
Comment From: madolson
@shaharmor I want to circle back here now that we have sharded channels implemented. I want to understand if you still believe that this functionality is worth implementing, or the alternative proposed by https://github.com/redis/redis/issues/8029#issuecomment-875768561 is a better solution?
Comment From: shaharmor
@madolson thanks for getting back to me about this. I haven't finished reading the new changes yet, but if I understand correctly its simply an implementation of PubSub which uses the keyspace to only allow one node owner per channel, with MOVED redirects like in normal commands?
While different than what this issue wanted to implement, it also fits the bill for having a linear scalable PubSub so I think it should be enough for closing this issue.
Comment From: madolson
Sounds good, I'll close this for now. Feel free to comment again if any deficiencies every come up.