The problem/use-case that the feature addresses
I have a sequence of items with timestamp values I'm XADDing to a stream. I wish to use those timestamp values in the message ID so I can query them using the actual timestamps.
I keep an internal sequence number which I use/increment every time I use a message. If my service goes down, I lose the sequence number I'm at, and end up hitting:
ERR The ID specified in XADD is equal or smaller than the target stream top item
Description of the feature
Allow specifying a partial message ID which only contains the timestamp and let redis populate the sequence number portion.
XADD mystream 123123-* foo bar
Alternatives you've considered
I'm currently using XINFO STREAM to find the sequence number of the last entry when starting my service.
Comment From: itamarhaber
Hello @icholy
Thanks for making this feature request. If I rephrase it a little, you're asking for a XADD form that adds a message with the same timestamp as the newest message in the stream. This type of use for streams, as far as I know, isn't common and even in your use case it is only rarely needed (i.e. upon restart) so I'm not convinced it merits implementation.
That said, this is an open discussion and my opinion is just one of many.
Comment From: icholy
The XINFO approach works fine for me. I filed this issue because it's something I assumed would already be supported. If it's not worth the added complexity, that makes sense not to add it.
Comment From: icholy
@itamarhaber ~~I have 2 producers writing events to a single stream. I want the timestamp in the message IDs to reflect the timestamp in the events themselves. This becomes impossible without my requested feature.~~
- nvm this won't work.
Comment From: youurayy
The killer use case for this is when you have multiple producers (as in data redundancy, failover, hot-hot) writing the same messages into the same Redis stream, and we want the messages to be keyed by their exact time (not by the time of their insertion which is usually a few milliseconds later, at least), and we don't want the messages to be deduplicated (as in, rejected by Redis).
(I believe this is a common usage pattern from many message queue systems. Sometimes you don't want to dedup the data with the same key (timestamp) bc their content may be slightly different and you may want be performing merge or diff on their content later.)
It would be marvelous to be able to spec MYTIMESTAMPID-* where Redis would fill out the minor (sub-milli) ID.
Can this be reopened or should I submit a new Issue?
Comment From: icholy
@youurayy the problem is if one producer is ahead of the other.
Comment From: youurayy
@icholy Ah! u mean - what if the producer-to-producer lag is bigger than that one millisecond...
dammit, u right
Comment From: itamarhaber
@icholy please have a look at https://github.com/redis/redis/pull/9217 and let us know if it feats your need.
Comment From: icholy
@itamarhaber looks perfect! I haven't had a chance to build/test it yet though.