Feature Request -> any entry older than X seconds is evicted from the Redis Stream!
I have a bunch of race conditions that can result in duplicate or inverted-in-sequence events that need to be delivered to my app. Currently the cleanest way to solve this is to account for these errors in my app's update processing code.
But! The cleanest way would be to use a Redis Stream, where everything older than 1 hour was deleted, allowing me to double check the correctness of events before pushing them down, without having to manually manage/evict old junk from the stream!
Just a thought :)
Comment From: jaamison
Hm, one thing that comes to mind is given that trimming a stream involves recalculating the radix tree macro nodes, there is the potential that stream entries with an expiry might cause random spurts of 1) a lot of computation, and 2) an abrupt change in the amount of memory used, all at idle.
I'm not sure about the architectual decisions that went into stream api, but it might be intentional that trimming a stream requires an explicit directive. Regardless, this should be scriptable, no?
Comment From: itamarhaber
It is indeed the case that the Stream's underlying data structure is not optimal for exact time-based trimming, and hence this feature's omission until now. It is possible that a "best effort" mechanism will be added (ref: https://stackoverflow.com/a/51174796/3160475) to provide a balanced approach.
That said, time-based trimming is a recurring feature request and as such should be given serious consideration. Currently, in order to implement it, one would need to use additional data structures (e.g. a Sorted Set or a RedisTimeSeries) to keep track of the production's pace and trigger a periodic process (possibly in any language, or as a server-side Lua script, or with RedisGears) to perform the actual trimming.
Comment From: oranagra