Hello, this is in regards to 2672 from awhile ago https://github.com/antirez/redis/issues/2672#issuecomment-164376229

I thought a new issue was in order vs. contining with 2672. Here’s our plan:

Cluster with PubSub has always broadcasted incoming messages to every node. In certain situations this limits the total amount of messages you can push and it would be nice to use the getNodeByQuery method to lookup which node needs the message.

  1. Some clients have joined channels with wildcard * patterns
  2. Existing clients must be able to continue sending just normal publish command and have it reach every node.
  3. Clients will have to be updated to know about new publishnobroadcast command and handle a MOVED reply from it. (Same for subscribenobroadcast)
  4. When new nodes are added, existing connections to channels may be terminated because the key has MOVED but this will only happen to channels that have received at least one publishnobroadcast command.

That last point in #4 means this is 100% backwards compatible with older systems that assumed channel connections would not be terminated from an add node event.

psubscribe is not changed. When using patterns you should still use publish and not this new publishnobroadcast command which is only meant for when you know the full name of your channel.

You can see the commits so far here:

https://github.com/andrewarrow/redis/commits/publishnobroadcast

Would something like this be the right approach? Thanks!

Comment From: mgravell

There's an additional thing that would be needed for the scenario where the cluster is resharded after people have subscribed - it would essentially need to broadcast a MOVED as a special pub/sub-style message (i.e. on the out-of-band protocol).

On Fri, 7 Sep 2018, 23:09 Andrew Arrow, notifications@github.com wrote:

Hello, this is in regards to 2672 from awhile ago #2672 (comment) https://github.com/antirez/redis/issues/2672#issuecomment-164376229

I thought a new issue was in order vs. contining with 2672. Here’s our plan:

Summary: Cluster with PubSub has always broadcasted incoming messages to every node. In certain situations this limits the total amount of messages you can push and it would be nice to use the getNodeByQuery method to lookup which node needs the message.

  1. Some clients have joined channels with wildcard * patterns
  2. Existing clients must be able to continue sending just normal publish command and have it reach every node.
  3. Clients will have to be updated to know about new publishnobroadcast command and handle a MOVED reply from it. (Same for subscribenobroadcast)
  4. When new nodes are added, existing connections to channels may be terminated because the key has MOVED but this will only happen to channels that have received at least one publishnobroadcast command.

That last point in #4 https://github.com/antirez/redis/issues/4 means this is 100% backwards compatible with older systems that assumed channel connections would not be terminated from an add node event.

psubscribe is not changed. When using patterns you should still use publish and not this new publishnobroadcast command which is only meant for when you know the full name of your channel.

You can see the commits so far here:

https://github.com/andrewarrow/redis/commits/publishnobroadcast

Would something like this be the right approach? Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/5336, or mute the thread https://github.com/notifications/unsubscribe-auth/AABDsEfSXHqe7IEonYTWd1cIgKR9mBSgks5uYu6ygaJpZM4Wfix4 .

Comment From: andrewarrow

Well the tcp connection from a subscribed client can always be closed without warning. (From just a normal network error.)

Any client expecting that connection to last forever and never be closed must have already noticed by now, that won’t work, and has logic to re-subscribe.

I.e. closing these connections because they moved is all you need to do. Clients already know to re-connect with a new subscribe or in this case a new subscribenobroadcast command. With this new command it’ll send MOVED if this channel key is now on a different node. And if you send just normal subscribe then you are fine because every node still gets those messages.

Comment From: andrewarrow

@antirez super excited to hear your feedback on this. Was hoping to submit a merge request soon with the end goal of getting these two new commands into official redis. We have a large AWS elasticcache cluster running. Thank you!

Comment From: andrewarrow

Hey @antirez could I get this issue into your queue? Thanks!

Comment From: itamarhaber

+1 @antirez - looks like much needed effort, e.g. https://youtu.be/6G22a5Iooqk

Comment From: cpoole

any update here? this is a definite scaling limit for us

Comment From: chivasblue

Hi @antirez , will something like the new Streams data type be used as a subtitute for the Pub-Sub feature without hitting the scalability limits on a Cluster?

Comment From: madolson

I'm going go close this since https://github.com/redis/redis/pull/8621 was merged. It provides functionality to broadcast a message to a channel that is sharded like keys. If this doesn't meet any concerns, let's follow up to address them.