Redis Build a Redis keys 'schema' for Redis

I want to build a virtual 'schema' for the Redis database. My presumption: all keys contain a text and eventually a number in the end. So I want to detect all text combinations, ignoring the number. Sample: customer:1 customer:2 product:1 My schema would be customer, product, etc. How is this possible? I tried using keys, but because the number of text and number combination can be large, is inefficient to scan all possibilities. It would be nice to have a command like keys, but which replace numbers with a letter, like '*' and return only the unique combinations. Could we make out of this a feature request?

Comment From: madolson

The only way to do this today would be with a lua script today, you could fetch all of the keys with the KEYS "*" command and then loop through everything. This will be pretty slow.

You could also look into https://redis.io/commands/keys to see if there is some pattern that will match just what you want though. You can use that with the SCAN or KEYS command.

Comment From: dbschema-pro

Thank you for your reply. Would it be a native command much faster than an LUA script? I don't know how the LUA script works. I think it is a standard for Redis to combine text with numbers to generate keys. And probably users need to document their databases, therefore a native command to list all unique keys ignoring the numbers inside would help.

Comment From: madolson

We can leave this request open to see if other people have a similar use case. My guess is that this isn't that popular of a request. LUA scripts are usually the solution to sort of odd ball requests.

You can of course also look into building your own function with https://redis.io/topics/modules-api-ref.

Comment From: dbschema-pro

My job is to write a JDBC driver for Redis, and this is going to be included in a database design tool - DbSchema. There are customer requests to make possible a visual design for Redis. And this can be done only by having good key management. Our request is to know the possible keys excluding the number from the key suffix. If I write by myself a Redis function using modules, I have the problem that I cannot ask each customer to install it on its server. So it has to be something that works on any Redis server.

Comment From: mgravell

For info: we have something broadly like this in "opserver" (which can monitor lots of systems, not just redid - SQL, haproxy, elastic, etc). What we ended up doing was using a bunch of user-defined regex queries in the config, and test them while ding a SCAN per database, which allows folks to understand the makeup (in terms of counts and sizes) of their redis data. It recommends to execute against a replica, not a primary - but if works on either. By default, IIRC, the regex pulls out chunks of numerics for the reason you say, but allowing the consumer to supply regex makes it easier to partition. It isn't perfect, but it helps.

Just some thoughts.

Comment From: mgravell

Also: I would probably avoid Lua here: that is going to tie up the worker thread on the server, which could significantly impact competing connections.

Comment From: dbschema-pro

Working with regexp as a solution is not something for large databases. I still think the best approach will be to have a parameter for one of the commands ( like keys ) or a separate command. The requirement to have a visual organization of Redis is high. Real customers need this. And you are the best to provide a solution for it. Maybe we can implement a first a LUA script, maybe you can help us. Then you will see how nice is to have a visual interface to Redis, and probably consider a better implementation.

Comment From: zuiderkwast

@wise-coders Here is a LUA script which returns all keys with all numbers replaced by *:

local ks = redis.call('KEYS', '*')
local s = {}
for _, k in ipairs(ks) do
    s[string.gsub(k, '%d+', '*')] = true
end
ks = {}
for k, _ in pairs(s) do
    table.insert(ks, k)
end
return ks

You can run it using the EVAL command: EVAL "<script content here>" 0.

This is at least better than sending over all the keys to the client using KEYS *, but running this on a replica is probably still a good idea. You could also modify the script to use SCAN instead of KEYS to get the result in chunks.

Since Redis stores the keys in some sort of hash table, there is no efficient way of doing this without traversing the whole set of keys, even if you would implement it as a native Redis command.