Hello,

I lack a possibility to read from stream with XREAD starting with the last available message. It is possible read from concrete ID (or all with ID=0) or wait for new message (with special ID $). But for our usage will be nice have to have possibility start with last message from stream, for example with special ID +: XREAD BLOCK 10000 STREAMS stramA streamB streamC + + +

If I need start from last message then now I must use firstly: XREVRANGE streamA + - COUNT 1 XREVRANGE streamB + - COUNT 1 ... XREVRANGE streamX + - COUNT 1 and then I can continue in loop with: XREAD BLOCK 10000 STREAMS stramA streamB ... streamX idA idB ... idX

Thanks

Comment From: madolson

Status: Small effort but I see two clear implementations possible here: 1. Implement as the feature request suggested 2. Implement an inclusive option on XREAD so that the $ includes the latest item.

Looking for someone to try to figure out what the best option is then implement it.

Comment From: felipou

I very much prefer the first option. I think the second options may add confusion as to the meaning of $.

Just did a quick test, seems the implementation would be simple (it's working): https://github.com/felipou/redis/commit/b43c5104899c9346e21c2bb7a6c0c0596dee0878 If that is the chosen option, I can turn this into a proper PR (add tests, check for better options in the code, update doc, etc).

Comment From: oranagra

@felipou i'm a bit afraid to take a decision here and carve it in stone for 6.2 for the fear of later regretting it after it's already released. Looking at the backlog there are quite a lot of pending improvements to streams (which aren't gonna make it for 6.2), maybe adding some of them will influence on the decision we take here.

Are you particularly interested in this issue, or just searching for another victim after completing the LMOVE effort? in which case can i suggest ZDIFF/ZDIFFSTORE #446 8-)

Comment From: felipou

Are you particularly interested in this issue, or just searching for another victim after completing the LMOVE effort? in which case can i suggest ZDIFF/ZDIFFSTORE #446 8-)

Both, really. I recently implemented something in my company that had the exact same requirement: "subscribe" (starting a loop of XREAD) and get the last element at the same time. I solved by just calling XREVRANGE before to get the last element, didn't even think about the possibility of changing XREAD, but looking through the issues for the next milestone here, I saw this issue and got interested because of the practical application for my case.

But I'll take a look at the issue you mentioned! Thanks for the suggestion!

Comment From: guybe7

maybe XREAD/XREADGROUP should check for [ when parsing the ids? this would work for this use case (XREAD BLOCK 10000 STREAMS key [$) and it's not too confusing (like using +) of course that [> is illegal

Comment From: majklik

From my point of view, the inclusive variant with "[$" is good and similar to the exclusive range for "XRANGE stream (999999 +". By the way, the same construction would be useful for "XGROUP CREATE key groupname [$" and "XGROUP SETID key groupname [$".

Comment From: itamarhaber

The more I think about it, the more I'm leaning towards using a new special character rather than reusing $ and [. The $ is meant to be used only with BLOCK, whereas this makes BLOCK meaningless (unless the stream is empty). So, the + makes more sense to me in terms of reflecting "the latest entry" like with X[REV]RANGE.

Comment From: guybe7

@itamarhaber for what it's worth i think it's better to allow [$ in the non blocking variant than adding yet another "special" char to XREAD[GROUP]

Comment From: itamarhaber

@guybe7 fwiw, I think overloading the $ with a different meaning even with the [ modifier will confuse, but I trust you more than I trust myself. This (the syntax) is definitely not a show-stopper, but we need to make clear (in the docs?) that blocking is futile in this case (just like calling $ without blocking).

Comment From: majklik

@itamarhaber The blocking depends on the state of the requested streams? If at least one stream have any record then XREAD ... stream [$ is always nonblocked, if all requested streams are empty then XREAD can be blocked for specified timeout (in this case is operation the same as using XREAD ... stream $).

Comment From: jfloodnet

To add a concrete use case in the mix. I monitor market price changes in real-time, adding this feature would be quite useful to avoid double handling calls to redis when loading latest prices.

EDIT:

For others treading the same path. XREVRANGE was very slow for me, and attempting this across many streams was unusable.

I instead opted for creating a HSET of streamName to lastMessageId-1.

Every time I XADD to stream, I take the return ID, decrement it, and HSET it to map.

This way all my subscribers can lookup the streams they are interested and know from which id to use to get the last message.

Comment From: psoleckimoj

We use similar for a monitor page, as the queue size isn't huge I do an initial call with 0 and then poll to fetch new messages. It just means that initial load can take 10/15 seconds depending on the number of environments/streams.

Comment From: ronen-kalish

We wish to implement this. The preferred implementation is using + rather than [$, and is based on the suggestion from @felipou. In addition, the use-case of the last entry for XREADGROUP is not clear, neither in terms of why is this required (when > already retrieves new, unassigned entries), nor in what exactly should it return (last pending item for this consumer? last unread item by any consumer?) @guybe7, can you please clarify what is the use-case for XREADGROUP and what is the expected return value?

Comment From: guybe7

I'm trying to remember what this was all about... I have a few questions/observations: 1. how is BLOCK even valid with +? Is it only applicable when the stream is empty? because otherwise, it will not block by definition 2. maybe we don't need it at all in XREADGROUP?

Comment From: oranagra

  1. that's right, IIUC the user will use + only on the first call, and then use $ on later calls. the first call will only block if the stream is empty.
  2. Maybe we don't, if we do, as Ronen said in the previous post, we need to decide which one is the last message.

Comment From: guybe7

i think that we should not touch XREADGROUP unless somebody actually needs it (folks that are involved here, please speak up)

having said that, if we do go for it, i think it should return the last message read by the specific consumer provided

Comment From: ronen-kalish

@guybe7, regarding BLOCK I agree it only makes sense if the stream is empty. I can't think of a good use-case for that, but that doesn't mean it doesn't exist. Can we conclude that XREADGROUP should not be modified? I don't think anybody has raised a request for that, or even defined what is the expected return value.

Comment From: felipou

I agree that XREADGROUP should not be modified, it has its own special meaning for IDs, and already does not support $.

Comment From: felipou

We wish to implement this.

@ronen-kalish Not sure if this code is still relevant, but feel free to use it (or any part of it): https://github.com/felipou/redis/commit/b43c5104899c9346e21c2bb7a6c0c0596dee0878