Hi everyone I’m writing this post because I am working on a project that implements caching using Redis and Redisson. We’ve been having a small issue on our production environment in which Redis returns a null value for a key that is present. After doing some testing I have been led to believe our issue is caused by key expiration and removal. I learned about the passive key removal which happens when someone tries to access an expired key, Redis will delete the key and return a null.
I noticed that this issue only happens on days in which many calls are made to Redis, so I came up with a simple test on my local environment: 1. Remove all keys from Redis 2. Insert a key with a 10 second TTL 3. Use a JMeter script to make multiple calls from multiple threads 4. Monitor logs and the Redis Insight Profiler
This testing has consistently caused Redis to return a null value for the tested key. After around 10 seconds of running this test the key will always return a null before an Hset operation is called and the key is put back into the cache.
I decided to run another test as a baseline to see how key removal would normally work when there is less load: 1. Remove all keys from Redis 2. Insert a key with a 10 second TTL 3. Monitor logs and Redis Insight Profiler
I noticed that around 2-5 seconds after the time that the key expired Redis will call the HDel command and remove it. I imagine this is Redis' active key removal doing the job.
After looking at the behaviour from these two tests. Is there any other reason why Redis could be returning null values? As far as I know Redis won't allow you to store a null values and we have a lot of safeguards in place to prevent anyone from storing a null. Is this behaviour normal from Redis?
During my second test I also noticed that even though the TTL is set to 10 seconds, the actual key removal will happen around 2-5 seconds later. I have noticed Redis will log when the key has been removed, but I am unable to find any way of logging when the key is expired and marked for removal. Is there a way to see these logs?
Comment From: hpatro
Active expiry cron runs periodically and tries to expire few items in each run. There is no guarantee that the key will be expired immediately after the TTL has elapsed. However, if you perform any operation on the key, then the key will be looked up and expiration logic will be applied and hence NULL would be returned to the client (after 10 secs in your case). NULL is equal to key doesn't exists and the client should refill the cache which you rightly do.
Did you have any other expectation in this scenario? @senrquez-hw-dev
Comment From: senrquez-hw-dev
Thanks for your response @hpatro.
I had not considered this scenario, I was under the assumption that Redis would remove the key on its own once it expired. I did know the removal wouldn't happen at the exact moment the TTL was exceeded. I thought that if someone tried to access a key that was marked for removal it would return the value, even if stale, until Redis deleted it and then it would have to be retrieved from DB again.
I have one more question, is there any way to get a notification or logs from Redis once a key's TTL has expired, but the cron job hasn't removed it? I don't plan on using this in Production, I just want to use it on my local environment to confirm this behaviour.
Comment From: hpatro
I have one more question, is there any way to get a notification or logs from Redis once a key's TTL has expired, but the cron job hasn't removed it? I don't plan on using this in Production, I just want to use it on my local environment to confirm this behaviour.
I don't think so we generate any keyspace notification for this. One possibility is you run SCAN operation periodically and all the keys would be touched and expired keys would be cleaned up as part of the process. I'm not sure if this is a standard process.
@oranagra any thoughts?
Comment From: oranagra
I have one more question, is there any way to get a notification or logs from Redis once a key's TTL has expired, but the cron job hasn't removed it? I don't plan on using this in Production, I just want to use it on my local environment to confirm this behavior.
you mean get notified right away when the key is logically expired? (before redis detected that, which is obviously not possible) or to get notified when a key is removed by lazy-expire, and not when it is removed by active expire (cron)?
in any case, i don't think we'll add anything to redis to satisfy this request, but maybe it'll resolve itself once we implement accurate expiration (i.e. that redis will keep the volatile keys sorted, and thus is more likely to delete them right after they logically expire). this was discussed many times before (it adds memory and cpu overheads), it's not currently planned.
Comment From: senrquez-hw-dev
you mean get notified right away when the key is logically expired? (before redis detected that, which is obviously not possible) or to get notified when a key is removed by lazy-expire, and not when it is removed by active expire (cron)?
Yeah I wanted to know about detecting the moment a key was logically expired, thanks for confirming it.
Thanks for your response.
Comment From: oranagra
ok.. maybe some day we'll have a more efficient expiration mechanism that will be able to detect them sooner after they logically expire. for now, we can't.