Issue Description
Hello Redis team,
I've been working with Redis Pub/Sub and noticed that it currently supports only "at-most-once" message delivery semantics. This is limiting for certain applications that require a stronger "at-least-once" delivery guarantee, as pointed out by the Dapr team here
Problem Statement
In high-availability applications and distributed systems, it's often crucial to have "at-least-once" message delivery semantics. Without this feature, tools like Dapr cannot use Redis Pub/Sub due to its limitation in not providing message persistence.
Proposed Solution
Would it be possible to add a delivery acknowledgment mechanism for Pub/Sub? Such a feature could work like:
- When a subscriber receives a message, it sends back an acknowledgment.
- If the acknowledgment is not received within a given timeframe, the Redis server could requeue the message for delivery.
Benefits
- This would make Redis Pub/Sub a viable option for systems requiring higher message delivery guarantees.
- It could potentially attract more users to Redis by making it a more versatile message-broker.
Comment From: hpatro
We've discussed about this issue in the past over here https://github.com/redis/redis/issues/5766
Comment From: cilerler
Thanks for the fast response. That discussion has its own jargon and seems to be mainly focused on Redis Stream and Redis KeySpace notifications, rather than Redis Pub/Sub. In my opinion, if Redis offers a Pub/Sub feature, it should at least support basic ACK/NACK functionality (and possibly priority and delay features). However, if you're suggesting that the discussion addresses my question, I'm open to closing this issue in favor of it. My understanding of Redis doesn't align with that interpretation.
Comment From: hpatro
Just to summarize the above issue, we have discussed to utilize the properties of Redis Stream(s) which supports ACK semantics for keyspace notification(s). This would provide the reliability to retry the messages if not processed by the clients.
From the discussion in https://github.com/dapr/components-contrib/issues/3100, it seems like Redis Stream(s) is also not able to fit the requirements. @cilerler Would you be able to summarize the gap in Redis Stream(s) which hinders the usage?
Comment From: cilerler
Thanks for the summary. To answer your question briefly, the issue is that messages aren't removed from the queue after receiving an ACK. This deviates from standard pub/sub behavior, where you'd expect messages to be cleared post-ACK.
Longer Explanation for Pub/Sub: A typical queue service should accept the message, deliver it, and retain it until an ACK is received. If an ACK isn't received within a set time (T) or if the client NACKs it, the message should be moved to the DLQ (Dead Letter Queue). Additionally, the service should support priority features and message deferral. The DAPR team claims that Redis Pub/Sub isn't reliable for DAPR consumers because it doesn't wait for ACKs. This makes it prone to data loss, especially if DAPR crashes. As a result, they've opted for Redis Streams. However, when dealing with millions of messages, it's not ideal to keep processed messages in the queue, even if they aren't being redelivered.
Comment From: trulede
Pub/Sub with Ack ... what happens if you have 1000's of published agents, should they all send an ACK. After that, it's still not clear that the message was actually handled. I think the ACK part can be considered differently.
This could be achieved as follows:
- The sender: publish the message ... 0, 1, or m clients might receive the message.
- The sender: write the message to a sorted list (timestamp as index), message content/value. If the messages are large, store a message id and put the message in a hashmap/string key, for later resending. Set a TTL so it gets deleted automatically.
- The receiver: rather than ack the message, sends a delete command for the sortedlist (and or other keys) and removes the message id.
- The sender: periodically query the sorted list for older messages, resend those. Reset the TTL as required.
But, this way could be easier:
Blocking commands can also be used to realise interesting operations. if at-least-once delivery could be reframed as only-once-delivery, then something around BRPOP would work perfectly. It would also be much less complex (i.e. faster) since all the surrounding operations are gone, you have a simple list, and clients simply call BRPOP.
Streams is doing similar things, but I also agree that it does not really fit what you want to achieve. But I don't think a PubSub does either. I think you can get faster without using either of those command subsets.
Comment From: cilerler
Pub/Sub with Ack ... what happens if you have 1000's of published agents, should they all send an ACK. After that, it's still not clear that the message was actually handled. I think the ACK part can be considered differently.
Yes, an ACK only sent out when the message processed by the client. Also remember that it may send back NACK as well.
I'm using a different queue service with this feature and was considering Redis PubSub as an alternative. That's why I opened this ticket. I believe adding this feature should be a priority.
Comment From: trulede
But what do you really mean by that. Should each subscriber reliably complete the same "work item", at some point in time, or only some subscribers.
What does NACK mean? The message was not received; received but not processed; received, processed and rejected. And then what?
PubSub currently does what it should. The other things you want are probably also easy to achieve, with a few simple Redis Keys. Hashmaps, sets, sorted lists and the blocking commands can be combined to create powerful, and simple, operations.
For example; publish a "work item" to subscribed nodes, and have them supply their response, ACK or NACK, to a Set (for each case, ACK or NACK). Then at an instant, you know which nodes received the "work item", and which of those processed it ACK or rejected it NACK. From set theory, you also can easily determine which nodes did not receive the "work item".
Comment From: cilerler
In my common use cases, I design queues to deliver a message to consumed only one time. Each message is processed by only one client present in the queue at that time. This client will either send 'OK' if the message is retrieved and processed successfully, or 'REJECT' if the message is retrieved but not processed successfully. Collectively, these responses are known as 'ACK' for OK and 'NACK' for REJECT. If a message is rejected, it should either be re-queued or sent to a Dead Letter Queue (DLQ) or Dead Letter Exchange (DLX), depending on the setup. Initially, re-queuing may be sufficient, as DLQ/DLX are more advanced concepts.
Regarding your point about "PubSub currently doing what it should," I both agree and disagree. The expectation is that the provider handles message control while the client focuses on the business logic like we see it in AMQP-based systems (like RabbitMQ and Azure Service Bus). I believe it's better for the queue to offer a controlled mechanism to prevent data loss. Leaving message lifecycle management entirely in the hands of clients can lead to a wide range of implementations.
Consider a scenario where a publisher extracts data from a database and distributes it into queues for processing by individual clients. Initially, it's generally more efficient for a single client to process the data directly from the database, unless techniques like database partitioning are applied. However, this is outside our current focus. The clients are responsible for processing the retrieved data. If a failure occurs, the client should send a REJECT signal to make the data available in the queue again. If the client cannot send this signal and a timeout happens, the data should also become available in the queue. Data is permanently removed from the queue only after successful processing confirmation by the client.
Comment From: manish181192
Any updates here? I am looking for the same solution as @cilerler
Comment From: trulede
Pubsub is a particular design pattern, you can read about it here : https://redis.io/docs/interact/pubsub/ .
If you need deliver acknowledgement you can use other features of Redis, such as the Streams family of commands ... or role your own mechanism using the various Redis commands. It's possible to be quite inventive.
Comment From: cilerler
@trulede This ticket was initiated following a statement by @berndverst from the DAPR team in dapr/components-contrib#3100:
"I'll close this issue for now. If Redis Pubsub ever changes to persist data, feel free to open a new issue to add support for Redis PubSub (instead of Redis Streams). Until then however, there is nothing actionable for us. Thanks!"
Could you provide the necessary information or guidance to integrate REDIS PUBSUB (not REDIS STREAMS) with DAPR PUBSUB as requested? Your expertise might make this seem straightforward, but I'm unclear on the process. Additionally, publishing an article on this topic would be extremely helpful.