The problem/use-case that the feature addresses
Hi!
We have a redis server and we want every key to have a ttl. This will help us avoid issues where someone by mistake writes a key without ttl and then we have a lot of unused data in redis. The mistake we encountered is a dev setting a ttl to a key before actually writing the key to redis. The result was no ttl on key.
Description of the feature
I suggest having a config for redis that will set a default ttl for all keys. Another option would be to have a config which will not allow writing keys without a ttl to redis. (Having both options would be awesome!)
Alternatives you've considered
Make setting a ttl for key work even if key does not exist yet. Make sure devs know to add a ttl to every key they write :)
Comment From: ShooterIT
Actually I also have similar idea that default TTL for all keys, our users also have this feature request. But we should know this config is dangerous because users may lose some keys if they don't persist the keys. I am not sure redis core team could agree
Comment From: madolson
I think a default TTL that applies to all new keys would be fine, it's a bit of a scary configuration but if you know your application well you should be able to pick a sane value. I don't think the proposal "Have a config which will not allow writing keys without a ttl to redis" isn't feasible in Redis without some major rework, so I would vote against that.
Comment From: nmeisels
I don't think the second proposal is feasible in Redis without some major rework.
Do you mean 1 or 2? 1. Not letting you set a ttl without the key existing / making it work? 2. Have a config which will not allow writing keys without a ttl to redis
Comment From: madolson
"Have a config which will not allow writing keys without a ttl to redis"
Comment From: nmeisels
So what about
Not letting you set a ttl without the key existing / making it work?
Is that also possible?
Comment From: madolson
It's semantically hard to express a state where a TTL exists but there is no key there, since they are pretty tightly coupled. I don't think we should work too hard to work around bugs that were written :), since ideally the resolution for your problem was the developer fixed your bug.
Comment From: nmeisels
It's semantically hard to express a state where a TTL exists but there is no key there, since they are pretty tightly coupled. I don't think we should work too hard to work around bugs that were written :), since ideally the resolution for your problem was the developer fixed your bug.
Ok. So should redis not throw an exception if you try to set a ttl for a non existent key?
Comment From: madolson
https://redis.io/commands/expire
It returns 0 if the key does not exist. Ideally the code your developer was writing should have tested against that. We generally don't throw "exceptions" for something that could be valid use cases
Comment From: ShooterIT
It returns 0 if the key does not exist
I also think it is ok, and developers are also used to this behavior
@madolson @nmeisels But if we worry about limited memory, we also can set maxmemory-policy to allkeys-lru to avoid filling redis memory.
Comment From: ShooterIT
One difference I can think of is that default TTL may use less memory instead of waiting key eviction when reached maxmemory.
Comment From: nmeisels
Thank you so much for all the responses. I learnt a lot! What's next?
Comment From: ShooterIT
I can try to implement default TTL configuration if core team approve it
Comment From: yossigo
I'm not convinced either way, but please let me be the devil's advocate here.
A default TTL is dangerous. Yes, it could keep the dataset smaller by expiring before eviction kicks in, but it could also end up taking up more memory due to TTL overhead (e.g. lots of small keys), do the opposite and have eviction kick in sooner. @ShooterIT Do you have some useful use cases for this?
I do recall hearing this request in the past, it came from guys who run Redis for developers in other teams who used volatile eviction policies but regularly forgot to set the TTL on keys. They wrote scripts that manually applied arbitrary TTLs to existing keys, and our conclusion was there's no more value in doing that vs. allkeys-* eviction.
Comment From: nmeisels
A default TTL is dangerous. Yes, it could keep the dataset smaller by expiring before eviction kicks in, but it could also end up taking up more memory due to TTL overhead (e.g. lots of small keys), do the opposite and have eviction kick in sooner.
We use our redis cluster this way. All of the data in our cluster should be TTL'd as it's only relevant for a short time. An example use case is using a redis cluster for rate limiting of user requests. We only need the keys for a short time and after that we have no use for them. We also use the cluster for other types of features which require a longer TTL but they may be accessed infrequently.
I do recall hearing this request in the past, it came from guys who run Redis for developers in other teams who used volatile eviction policies but regularly forgot to set the TTL on keys. They wrote scripts that manually applied arbitrary TTLs to existing keys, and our conclusion was there's no more value in doing that vs. allkeys-* eviction.
We have a volatile eviction policy. Having an allkeys-* eviction policy can cause unexpected clearance of keys which we actually require, compared to new keys which might not be used.
Comment From: ShooterIT
Hi @yossigo FYI, I talked with my colleagues, actually, in our company, this requirements are much more from memcached users, because in memcached set command, they must set expire time. but in redis set command, it is not necessary, so they may forget to set expire time.
set key flags exptime bytes [noreply]
value
But redis is redis, I am not sure we should support default TTL feature, I have a glance at it, i think it's not easy to support it.
Comment From: nmeisels
@ShooterIT any updates? Thanks
Comment From: ShooterIT
@nmeisels Sorry, i didn't do that, IIRC, we didn't reach an agreement for this decision.
Comment From: tauferp
Any further thoughts about this issue? It would be really beneficial to us.