What’s the problem? The current pubsub implementation works poorly with Redis key space functionality since it was built to exist in its own space. The two notable problems I’ve see are ACLs (https://github.com/redis/redis/issues/7923) and cluster scaling (https://github.com/redis/redis/issues/2672). I saw suggestions that were very tactical, but I think it’s really a fundamental problem with how the API was designed that should be addressed. I propose we implement a new type of publish mechanism that directly and explicitly maps to a key. New commands:
Kpublish key-channel message, publish to a channel key space and optionally a sub channel. Drop in replacement for publish if you already use that. Ksubscribe key-channel [key-channel] … Kunsubscribe key-channel [key-channel] …
Publishing will only be accepted on masters, and replicated normally to replicas, this prevents having two separate mechanisms for replicating the data for cluster mode/cluster disabled. It will not be propagated throughout the cluster, but will do validation to make sure the command was sent to the right shard.
Is this really a key? No, it shouldn’t be a key because: 1. It has no internal state, you can’t touch it, miss it, look it up, etc. 2. Replicas would need to set the state to subscribe, which can cause divergence if we blocked an incoming write on it. 3. How do you copy/move/rename something with no state.
But, we need to treat it kind of like a key for cluster mode, namely: 1. On migration, we need to disconnect all the clients using it. 2. Send re-directions when trying to publish/subscribe to the wrong shard. 3. As a side effect, ACLs can also applied to the channel key normally.
Notable limitations: 1. Invalidations and key space notifications can’t be migrated, since they rely on the notation that channels are entities of the cluster. Publish messages are still emitted locally in those cases too, so the problem is really just ACLs. 2. Patterns are intentionally omitted, since I think they’re very inefficient and scale poorly. If you still want them, you have to use the old mechanism.
We could also solve the issues with pubsub individually, but I think we should kind of ignore the old mechanism and move to a more forward thinking mechanism. I just wanted to post this thought here for feedback.
Comment From: yossigo
@madolson I like this idea, although I don't think about it in terms of keyspace-based pub/sub but in terms of sharded channel-space. Also, the old channels aren't going away, so ACLs would still need a way to specify channel name selectors.
Comment From: madolson
Ok, well, the only that might change is the documentation. I might try to prototype this a bit.
With respect to old channels, I agree they aren't going away. But if we can get almost all of the functionality with this, I think that is better than having to introduce special handling for channels.
Comment From: hpatro
Taking a look at this.
Comment From: hpatro
CHANNELS V2
PUBLISHLOCAL/SUBSCRIBELOCAL
PUBSUBLOCAL is the new form of channels which is bounded to a slot (inside Redis shard). This is essential to scale the pubsub in a Redis cluster mode as the regular PUBSUB channel propagates the messages across the cluster which is not scalable. For Redis non cluster mode, the behaviour is consistent with the earlier pubsub channel. For Redis cluster mode, appropriate redirection will be provided as it will be assigned to a HASH SLOT and the data will be available only within the shard (MASTER/REPLICA) where the slot would exist.
SLOT
PUBSUBLOCAL channel will be allocated to the same slot as a key would be with the same name i.e. both keys and channels share the same algorithm for slot
slot = CRC16(channel_name) % 16384
SUBSCRIBELOCAL
To subscribe and receive messages from pubsublocal channel.
SUBSCRIBELOCAL CHANNEL [CHANNEL ...]
Considerations
- If multiple channels are provided, all channels should be in the same slot, otherwise the following error will be returned.
(error) CROSSSLOT Keys in request don't hash to the same slot
Result
In cluster mode, if a client is not pointed to the correct node with slot and tries to subscribe, the following message will be returned.
Redirected to slot [slot-id] located at <hostname:port>
Once the client connects to the appropriate node, the following message will be returned.
Array Reply:
[SUBSCRIBELOCAL, <channel_name>, <count>]
And on new message being published to the channel, the response will be
Messages as Array Reply:
[<channel_name>, <message>]
With the above API, a client can subscribe to multiple channels and start receiving for any message publish within the same slot.
PUBLISHLOCAL
To publish a message to a pubsublocal channel.
PUBLISHLOCAL CHANNEL MESSAGE
Result
Integer: <count of subscriber receiving the message>
PUBSUB Stats
To get the number of subscribers for a channel
PUBSUB LOCAL NUMSUB [CHANNEL-1 ... CHANNEL-N]
Messages as Array Reply:
[<channel_name_1>, <subscription_count>...<channel_name_n>, <subscription_count>]
To get the local channel(s)
PUBSUB LOCAL CHANNELS [PATTERN]
Glob style pattern is supported as additional parameter to discover channels matching the pattern.
Array Reply:
[<channel_name>, <channel_name_1>...<channel_name_n>]
UNSUBSCRIBELOCAL
To unsubscribe client from a local channel/all local channels.
UNSUBSCRIBELOCAL [CHANNEL]
A notification message is sent for each channel, the client is unsubscribed from.
ACLs
The local channels will work with the existing ACL rule for Pub/Sub channel pattern. (&).
Slot Migration
- CLUSTER SETSLOT
IMPORTING/MIGRATING
During the importing/migrating the channel will still be served from the slot in the source node. Any redirection will point the same.
- CLUSTER SETSLOT
NODE
On execution of the above command on the source node, all the clients subscribed to the channel in the slot, will receive a unsubscribe notification and on execution on the target node, the channel for the slot will be served from the target node to which the slot was moved.
Comment From: zuiderkwast
Don't forget we already have streams (with the fancy feature of consumer groups). Can't we use that instead of adding yet another key type? The problem with streams is that you need a blocking call (or polling) to read from a stream.
How about adding XSUBSCRIBE?
For the use case where you don't want to store anything in Redis, we could allow XADD with MAXLEN 0 to send the entries only to currently blocking or subscribed clients...
Comment From: madolson
Streams have different guarantees than pubsub, namely that you are making sure everything is being cleaned up eventually. I think supporting XSUBSCRIBE is a good idea though. It's something we've mulled about before, and has come up for durable keyspace notifications.
Comment From: zuiderkwast
Since old pubsub isn't going away, how about the optimizations suggested by @antirez in this comment and by @DivineTraube below? TL;DR distribute the messages only to the nodes where there are any subscribers of the channel.
It will work out of the box for old clients. If this is then combined with a recommendation for cluster clients to shard the channels just like keys, the performance is improved in general.
Since we already have pubsub and streams designed for similar-enough use cases, I think we should try harder to optimize and adapt these before adding yet another set of commands.
Comment From: yossigo
@zuiderkwast I agree with your final remark, but I'm not sure about the feasibility.
- It will only work out of the box where we have many channels and a small number of subscribers (less than cluster nodes), otherwise there's statistically no real gain.
- There is an inherent race here between subscription and propagation. It's true that PubSub is not guaranteed to be reliable, but I think this option may guarantee NOT to be reliable, especially with clients that are not able to cluster themselves.
Comment From: zuiderkwast
- It will only work out of the box where we have many channels and a small number of subscribers (less than cluster nodes), otherwise there's statistically no real gain.
@yossigo Can you elaborate the statistical argument? Is that valid also for keyspace based pubsub and streams?
The way I see it: If users start using cluster clients which shard the publish and subscribe channels like keys (by convention), the all-to-all propagation is optimized away and they get plain sharded pubsub behavior in practice, just like if they'd be using #3346, but without config.
If clients libs are not updated though, I agree there is no statistical gain. A more explicit way would be to do #3346 (with PUBLISH and SUBSCRIBE resulting in -MOVED and -ASK redirects when sent to the wrong node, presumably). The config would create two different behaviours though and we still can't get rid of the old default behaviour.
- There is an inherent race here between subscription and propagation. It's true that PubSub is not guaranteed to be reliable, but I think this option may guarantee NOT to be reliable, especially with clients that are not able to cluster themselves.
There's an inherent race between subscribe and publish with a single Redis node too. What exactly do you mean by reliable here?
Do you mean that we support some guarantee like once an OK reply of SUBSCRIBE has been received, any PUBLISH sent after this will be delivered to the subscriber? If that's what you mean, we could accomplish that by making SUBSCRIBE block the client until the cluster has synchronized the subscriber info. But if we don't have any guarantees like that already, I can't see why this would be a deal breaker.
Comment From: madolson
@zuiderkwast Option #3346 is basically what is being proposed here, except instead of a config we just have another command. There is a bit more nuance to the implementation, but you hit the important bits.
I think the separate commands is a better approach since it makes it clearer from the client side whether or not it's implemented. Someone might also want both guarantees. I'm not convinced we should deprecate the old behavior without evidence people don't want it.
Comment From: yossigo
The way I see it: If users start using cluster clients which shard the publish and subscribe channels like keys (by convention), the all-to-all propagation is optimized away and they get plain sharded pubsub behavior in practice, just like if they'd be using #3346, but without config.
I agree and I think that's also what @madolson refers to. My point was that as long as clients don't behave in this way we gain nothing, like you said.
There's an inherent race between subscribe and publish with a single Redis node too. What exactly do you mean by reliable here? Do you mean that we support some guarantee like once an OK reply of SUBSCRIBE has been received, any PUBLISH sent after this will be delivered to the subscriber? If that's what you mean, we could accomplish that by making SUBSCRIBE block the client until the cluster has synchronized the subscriber info. But if we don't have any guarantees like that already, I can't see why this would be a deal breaker.
You're correct and technically there's a race already, but practically this race boils down to what client got served first in the event loop - assuming they've even hit the server in the same iteration.
I don't think we should block clients and turn channel subscription into some kind of a consensus-driven operation, but just pointing out that this race grows significantly and may become a practical issue where the current race isn't.
Comment From: shaharmor
The proposed solution only helps in the case of many channels, where the total number of messages (throughput) that are currently flowing through a single Redis node is lower than the maximum capacity of that node.
However, in the case of either a big single channel, or that there are just many messages currently allocated to a single node, this solution will not help.
What I suggested in https://github.com/redis/redis/issues/3346 was to disable the propagation in Redis Cluster for PubSub messages, so that I could build a "smart" client that will subscribe for the same channel on all cluster nodes, and will publish messages in a round robin fashion to all nodes as well. This way the load will always be evenly distributed among all of the Redis Cluster nodes, regardless of how many messages/channels/publishers/subscribers there are, and which slot they fall into.
Is there a plan to also solve the above issue?
Comment From: hpatro
With the above implementation you would be able to publish a message on any of the node which is handling the slot (master/replica). So, now the traffic would reduce by N times (N is the no. of shard you would have).
What I suggested in #3346 was to disable the propagation in Redis Cluster for PubSub messages, so that I could build a "smart" client that will subscribe for the same channel on all cluster nodes, and will publish messages in a round robin fashion to all nodes as well.
For the approach you're suggesting, it is possible to do with PUBSUBLOCAL if you choose to have master only setup and scale out. So, each of the message stays within a single node/master and our client needs to become smart to read from different channels across shards and aggregate them.
Comment From: shaharmor
So instead of having each client just subscribe to the same "channel-name" on each node, it will have to subscribe to "channel-name-{XYZ}" on each node, where XYZ is a special string that maps to one of the hash slots on each node, and recalculating those channel names and subscriptions on every resharding.
It's possible, just a bit more complex.
What do you think about having: - PUBLISHLOCAL - publish to the node you're connected to without any hash slot calculations and propagation (except for sending messages from master to slave and vice versa). - PUBLISHCLUSTER - publish to the node you're connected to, but using the same hash slot calculations, redirection, etc as listed above
Comment From: hpatro
I definitely understand the benefits with it, each individual node behaving as a PUBSUB server. However, when it's working with tandem with other nodes I believe it's a bit confusing. A client connected to Node-X would be aware of different set of messages and the one connected to Node-Y would be aware of other set of messages based on the way PUBLISH is done. Both PUBLISHERS and SUBSCRIBERS have to be aware of which node they are supposed to connect to irrespective of the state the cluster is in.
Tagging @madolson @zuiderkwast @yossigo to share their thoughts as well.
Comment From: madolson
@shaharmor Thanks for adding more detail about your use case! I'm going to add some context first, and then respond to your use case.
The current cluster implementation is intended to be agnostic to cluster topologies. Commands either operate on the entire cluster, like pubsub does today, or on a specific slot (which is mapped to a specific node) like get/set commands. You want to introduce a new type of command that operates on a shard (or node, not entirely clear how your smart client would work). This seems like a missing primitive within Redis cluster. We've had similar issues like reliable pubusub notifications since we want to store the notifications into a stream on each node generated by that node, but there is no consistent way to name that key.
The solution outlined by @hpatro I would say is a "idiomatic" way to introduce pubsub into cluster mode. It takes advantage of the cluster mode consistency model and maps the commands to slots. I think your ask makes sense, but in some ways it's an anti-pattern in the way redis cluster mode works today. Maybe we want to allow a way to connect to individual nodes in a Redis cluster but execute commands in a non-cluster mode way?
So, I think your problem is worth solving, but I think we need to really think through what is the best way to solve it.