Purpose
This meta issue consolidates streams related issues. The first comment (this one) will be updated periodically, feel free to discuss below.
- XREAD with ID '$' has odd/undefined behavior when attempting to begin observation on multiple streams #5381
Comment From: j3t
Reading items from multiple streams in the order they happened is not possible or at least very expensive. There are two possibilities at the moment.
- You read all items from the streams you are interested in at once and order them by stream id (e.g.
XREAD STREAMS a b c d 0 0 0 0). - You read the first chunk of items (e.g.
XREAD COUNT 1000 STREAMS a b c d 0 0 0 0) and skip all items where the stream id is greater than the lowest stream id of the last item from stream-results with 1000 items. Now, you can order them by stream id and then you can read the next chunk with the same procedure (last consumed id as ID) and so on and so on.
In my opinion XREAD should accept only one ID instead of multiple and COUNT should be the maximum count of items over all streams not per stream.
What are the use cases for XREAD with multiple streams?
Comment From: youurayy
@j3t
What are the use cases for XREAD with multiple streams?
Implementing a multi-topic pub/sub message queue -- not blocking a connection on waiting for XREAD on a single topic/stream (being able to use single connection to read from any number of streams on a given node).
Additionally, IMO, when interested in the latest data feed, the fact that one has to keep repeating the XREAD call in an infinite loop (and thus adding latency) is a Streams design shortcoming/oversight. (You don't have to do that in Kafka.)
I know PUBSUB can do that, but it doesn't scale on the Cluster (this should change in 7), but Streams is still better for catching up on previous/missed data in a new client connection/reconnecting.
Related: https://github.com/redis/redis/issues/6202 https://github.com/StackExchange/StackExchange.Redis/issues/1155 https://github.com/redis/redis/issues/6280
cc: @antirez
Comment From: j3t
It's almost three years ago but if I remember correctly then for my use case it was important to have the data from multiple topics ordered by creation timestamp. Please correct me if I'm wrong but in my opinion this is not supported by redis.
For me it's still not clear why XREAD expects ID for each topic and COUNT not. If you want specific offsets per topic then you could also just execute XREAD for each topic separatly, right?
Comment From: youurayy
@j3t
multiple topics ordered by creation timestamp
the data in each topic is ordered by the key so not sure what's the use case to order multiple topics by creation date
why XREAD expects ID for each topic and COUNT not
assume the COUNT is there to prevent flooding your client with too much data, in which case a shared COUNT will do just fine.
execute XREAD for each topic separately
the point (for a single client) is to share one single connection (per node) for a blocking XREAD (with blocking you don't have to do periodical polling), while handling multiple topics, so this design is great for that
i'm about to replace Kafka with Redis (we license Redis Enterprise) in the software i architect the comms for, so I don't know if I'll hit any walls yet -- just wish the XREAD with BLOCK could work like SUBSCRIBE (no need to repeat the command), but i can also understand that from the point of Redis this may be much easier to implement (to leave the loop on the client), and also then the client can better regulate the back-pressure, and maybe the latency won't be so dramatic, will see.