Message entry id still shows up in xpending after xtrim

To reproduce:

  • write to a stream
  • create a consumer group
  • xreadgroup some messages but don't ack them
  • xtrim the stream
  • xpending returns all the entry ids including those have been trimmed

I don't know if this is by design or a bug

Comment From: lucasnetau

I'm having similar issues, however it gets even worse when trying to recover from a crash.

Same reproduction steps as you, if you add in

XREADGROUP with ID set to 0 it will return an entry id but will a (nil) value.

1) 1) "mystreamname" 2) 1) 1) "1563939745227-0" 2) (nil)

vs the expected normal operation when no un-acked events exist

1) 1) "mystreamname" 2) (empty list or set)

Clients like phpredis return boolean false for the unexpected response. This means that consumers cannot recover correctly from crashes in the future as this un-acknowledge non-existing record is returned on each call to XREADGROUP with ID of 0 and not any existing unackowledged records.

Comment From: yaroot

I rewrote my scripts to ack all deleted messages before trimming the stream.

eval is always a workaround for messed up clients, not pleasant but works. @lucasnetau

Comment From: lucasnetau

I don't like workarounds. This smells like a bug, if pending events are trimmed before they are acknowledged they should be cleaned up with the TRIM process.

Comment From: antirez

This is actually by design: there is a worker out there that received the message and is going to ACK it: even if the message is no longer there, we can reply "1" to such worker. Moreover I'm not convinced that it is a great idea to lose completely track of the fact that such message, even if now deleted, was delivered to some consumer with the explicit contract of it being acknowledged. Finally to check every time in every consumer group if there is such message would be expensive when trimming, very expensive AFAIK.

Comment From: lucasnetau

Thanks for the details. If this is by design should I be submitting a bug to phpredis then? The call to XREADGROUP with ID of 0 returns a boolean false (ie failed) for the redis response of

1) 1) "mystreamname"
 2) 1) 1) "1563939745227-0"
         2) (nil)

So the client is attempting to recover but the client is returning that the XREADGROUP failed and we never continue.

Comment From: yaroot

Okay I get that, then it's better to have a simpler (and correct) method for trimming, instead of using MAXLEN which could delete unacked entries. And better docs, eg. code example for recovering correctly.

Comment From: maksutovaynur

I encounter the same problem. XPENDING shows lots of already trimmed entities. I.e. if I call XCLAIM on all of them, it will change ownership on all this ghost items, but will return None values.

Comment From: Yeganloo

This feature could be more useful if data was not lost!!!