The problem/use-case that the feature addresses
Add a command to scan the expired dictionary, we can get those keys that contain the expiration time, and can actively trigger the elimination.
Description of the feature
the command like SCANEXPIRE cursor [MATCH pattern] [COUNT count] [TYPE type]
By modifying the bottom-level scan commands (sscan and hscan commands are also the bottom-level functions), the outdated dictionary is actively scanned for active elimination. We can reuse these codes and do a small amount of modification to achieve this function.
This command has two main functions：
- Conveniently obtain all keys with expiration time
- Actively trigger overdue elimination
Additional information
Hello, very much look forward to your recovery
Comment From: ShooterIT
Hi @631086083 From redis 6.0, we have better expire key scanning mechanism, change 'random sampling garbage collector' to 'consecutive garbage collector`. I don't know your redis version, if not redis 6.0, you can have a try, i think 6.0 can mitigate your problem that expired keys are still occupy memory.
Comment From: 631086083
Hi @631086083 From redis 6.0, we have better expire key scanning mechanism, change 'random sampling garbage collector' to 'consecutive garbage collector`. I don't know your redis version, if not redis 6.0, you can have a try, i think 6.0 can mitigate your problem that expired keys are still occupy memory.
Thank you very much for your help. Redis6 is indeed very effective for expiration and elimination. The reason I want to add this command is also actually to have a function to obtain keys with expiration time in batches.In this regard, do you think this feature can be added.
Comment From: soloestoy
new command, need more opinion from @redis/core-team
Comment From: itamarhaber
Hello @631086083
I understand the FR, but would like to get more information on its intended use. This is especially important in order to understand the behavior for:
- Keys that expired between calls
- Keys that were added with a TTL between calls
- The order of keys returned
My main concern here is what type of contract the server provides, especially given the time-dependent nature of the data involved. So, put a little differently, assuming Redis replies with 10...100 non-expired key names and their TTL with each call to SCANEXPIRE, what would the user use that information for? I can see how one would use it for admin purposes (house cleaning), but I'm wondering if there's something else I'm missing here.
Also, it would be nice to consider a couple of additional modifiers here:
- KEYSONLY - returns only key names without their TTL
- NOTTL - returns keys that do not have a TTL (different logic in terms of what dicts are scanned, so maybe that should be in regular SCAN
- MINTTL/MAXTTL ttl - returns keys with TTL greater/lower-than-or-equal to the provided value
Comment From: 631086083
Hello, @itamarhaber what I think is not very well, your idea is much better than mine. We can decide whether to return the expiration time through the flag bit.
I think of two scenarios:
One case is that when the CPU is empty for a long time (such as the low peak period in the morning), we can actively eliminate the data through the command. In this case, it is better to use the command that does not return the expired time, which reduces the network transmission.In another case, we can not return the key, and only need to traverse the cursor of the hash bucket and eliminate the number of keys next time
In another scenario, access rights through redis: keys without expiration time represent permanent access rights (keys with expiration time are temporary access rights, and nonexistent keys have no access rights). You can obtain expiration time and keys through this command. It is very cost-effective to upgrade temporary access rights with expiration time.
It's also a very good function to get the key whose expiration time is greater than or less than a TTL. We can support this feature by modifying the underlying common commands of scan, or even adding them to the commands such as scan. radix tree is a good way to achievement it, Which way do you think is better