Per the XACK documentation:
Once a consumer successfully processes a message, it should call XACK so that such message does not get processed again, and as a side effect, the PEL entry about this message is also purged, releasing memory from the Redis server.
This does not seem to be the case be the when there is a single consumer group, as seen in #6941. I've also run into this problem as one of my services was scaling. Would it be possible to update the XACK docs to include this? Is this intended behavior? If this is a bug, is there an issue tracking it?
Comment From: itamarhaber
Hello @alliefitter
Acknowledging the message does not delete it from the Stream itself, only from the group's PEL. In my opinion, this isn't an issue and per the design.
Comment From: tgrall
Hello @alliefitter
Yes, I believe that it is by design when you work with Streams (event/stream-based architecture) you may have many consumers/clients reading and re-reading the messages. So it is important that the message is not deleted automatically since you do not know if you may need the message again.
For example, you may have a first service that reads the message in real times (as a consumer), and another one that runs on a regular basis with a simple XRANGE.
My understanding of the documentation you mentioned is more about the deletion, it is the delete of the PEL Entry not the message. (I may be wrong)
Comment From: jeevaengg21
Understood that this isn't an issue as per the design.
But for the shake of memory management, is there any documentation to understand when the message will be removed from memory once all the group acknowledged the message from the particular stream.
So far I understand from Forums that few techies advised using MAXLEN when adding a message to the streams to make it capped, but am not sure whether this correct approach to follow, any suggestion and guidance?
Comment From: itamarhaber
@jeevaengg21 messages are never deleted from the stream (i.e. "removed from memory") by read operations, whether in a consumer group or not. On the other hand, XDEL, XTRIM and XADD ... [MAXLEN|MINAGE] can all delete messages.
Improvements to the documentation are always welcome - please feel free to suggest them at redis/redis-doc.
Comment From: maor-rozenfeld
I think that the main issue here is that streams are growing indefinitely , and ignoring the MAXMEMORY POLICY.
The design of not deleting messages after they're consumed, because you don't know who else might consume them later, is perfectly reasonable. However, streams that aren't capped shouldn't be able to fill redis memory until a reset or a manual deletion is required. Instead they should respect the maxmemory policy and truncate old messages if the policy is not noeviction.
For now I just capped my streams to a random limit of 1000 messages but that's not really how I wished to solve memory limits.
Comment From: orsinium
Nats has this behavior configurable when creating a new stream. It has Retention field which specifies when messages will be removed from the stream: when reaching a size limit, when there is at least one ack, or when all known consumers have acknowledged it.