Please consider implementation of a dead-letter stream - more details listed below. Thank you.
Source queues can target dead-letter queues for messages that can't be processed (consumed) successfully. Dead-letter queues are useful for debugging an application or messaging system because they let one isolate problematic messages to determine why their processing didn't succeed.
Sometimes, messages can’t be processed because of a variety of possible issues, such as erroneous conditions within the producer or consumer application or an unexpected state change that causes an issue with your application code. For example, if a user places a web order with a particular product ID, but the product ID is deleted, the web store's code fails and displays an error, and the message with the order request is sent to a dead-letter queue.
Occasionally, producers and consumers might fail to interpret aspects of the protocol that they use to communicate, causing message corruption or loss. Also, the consumer’s hardware errors might corrupt message payload.
What are the benefits of dead-letter queues? The main task of a dead-letter queue is handling message failure. A dead-letter queue lets you set aside and isolate messages that can’t be processed correctly to determine why their processing didn’t succeed. Setting up a dead-letter queue allows you to do the following: o Configure an alarm for any messages delivered to a dead-letter queue. o Examine logs for exceptions that might have caused messages to be delivered to a dead-letter queue. o Analyze the contents of messages delivered to a dead-letter queue to diagnose software or the producer’s or consumer’s hardware issues. o Determine whether you have given your consumer sufficient time to process messages.
https://en.wikipedia.org/wiki/Dead_letter_queue
Comment From: hpatro
@nadirdbit I would like to understand a bit more of the problem. Is something hindering the application to create a {stream_key}-dlq stream and push message to it based on the criteria you have mentioned above?
Comment From: piavgh
@hpatro : the problem is how can you get the full message data that will be pushed into the {stream_key}-dlq stream?
A simple scenario is: - We want to push any message that was delivered (and failed to be processed successfully) more than 10 times to the dead letter queue
At the moment, I only know that we can use extended form of XPENDING to get the pending messages with RetryCount field Based on this RetryCount, we can detect which message ID that need to push to the DLQ
But the problem it, we can only know the message ID, not the whole message data
Comment From: hpatro
At the moment, I only know that we can use extended form of XPENDING to get the pending messages with RetryCount field Based on this RetryCount, we can detect which message ID that need to push to the DLQ
After that, the client could do the following
- Get the message via
XREAD COUNT 1 STREAMS mystream <id> - Add the message to DLQ stream
XADD {mystream}-dlq <message> - Remove the message from the original stream
XDEL mystream <id>
@piavgh Would the above work?