Example:

return #redis.call("KEYS", "*");

Even SCAN command works only 1.6 times slower when using lua but do the same:

return #redis.call("SCAN", 0, "COUNT", 1000000000)[2];

That means KEYS is 1.5 times faster than SCAN without lua but when using lua SCAN is faster than KEYS 7 times. Is this a bug in lua or just some feature? Commands replies are almost the same, time for parsing should be the same too, isn't it?

redis 3.0.4

Comment From: itamarhaber

Note that SCAN's 'count' directive is only a hint. That means that your Lua doesn't actually scan the entire keyspace (whereas using KEYS does), which could explain the differences you're measuring. To do a fair comparison, implement (in Lua) a loop on the cursor until it reaches back to zero.

Comment From: vitaliylag

@itamarhaber, I know that. In my case SCAN always returning all elements and they all are unique, I checked that many times.

Also this script has additional checks and wins by speed anyway:

local ans, has, cursor = {}, {}, "0";
repeat
    local t = redis.call("SCAN", cursor, "MATCH", KEYS[1], "COUNT", 1000000000);
    local list = t[2];
    for i = 1, #list do
        local s = list[i];
        if has[s] == nil then has[s] = 1; ans[#ans + 1] = s; end;
    end;
    cursor = t[1];
until cursor == "0";
return #ans; --or return ans;

Adding checks have made script 1.8 times slower than before but it 4.3 times faster than KEYS with lua anyway. Script for KEYS command is the same:

return #redis.call("KEYS", "*");

Comment From: itamarhaber

Yet another reason never to use KEYS :)

BTW, in your script above you should pass the match pattern using the ARGV construct.

How many keys do you have in the keyspace? One possible explanation for the difference is that SCANing requires less memory compared to KEYS (whose reply buffer can get quite big), perhaps avoiding the need for a big buffer reallocing is what makes this tick faster.

Comment From: vitaliylag

@itamarhaber, I have tested it for 100, 1000, 10000, 100000, 1000000, 10000000 keys: 1) KEYS: the fastest method 2) SCAN: ~1.5 times slower 3) SCAN with lua:~2.4 times slower 4) SCAN with lua and additional checks: ~4.3 times slower 5) KEYS with lua: ~18 times slower

Results are a little different but difference is not big.

One possible explanation for the difference is that SCANing requires less memory compared to KEYS

I don't think that's true because without lua KEYS is 1.5 times faster than SCAN.

Big memory using can not be a problem for GC or allocator too because I tested even on 100 keys.

Yet another reason never to use KEYS :)

I think so too but can I be sure about other commands? Maybe it can affect not KEYS only.

Comment From: erikdubbelboer

I did some benchmarking.

As you can see with only 100 keys the KEYS and SCAN command are about the same speed. But when you wrap them in Lua KEYS becomes much slower than SCAN. I can't really explain why, they both use bulk replies so I would expect the Lua code to do the same amount of parsing.

With 1000000000 keys KEYS is a bit faster than SCAN, unless you wrap them in Lua, then KEYS becomes much slower again.

As expected the Lua wrapped version is always slower than the normal command.

$ redis-cli eval "for i=1, 100 do redis.call('set', 'foo' .. i, i) end" 0
nil

$ redis-benchmark keys '*'
12013.45 requests per second

$ redis-benchmark scan 0 match '*' count 1000000000
12137.39 requests per second

$ redis-benchmark eval 'return redis.call("KEYS", "*")' 0
2695.85 requests per second

$ redis-benchmark eval 'return redis.call("SCAN", 0, "MATCH", "*", "COUNT", 1000000000)[2]' 0
6681.37 requests per second

$ redis-cli eval "for i=1, 100000 do redis.call('set', 'foo' .. i, i) end" 0
nil

$ redis-benchmark -n 100 keys '*'
13.39 requests per second

$ redis-benchmark -n 100 scan 0 match '*' count 1000000000
8.83 requests per second

$ redis-benchmark -n 100 eval 'return redis.call("KEYS", "*")' 0
0.73 requests per second

$ redis-benchmark -n 100 eval 'return redis.call("SCAN", 0, "MATCH", "*", "COUNT", 1000000000)[2]' 0
5.17 requests per second

Comment From: vitaliylag

@erikdubbelboer, Yes, that's what I meant. If this affects only KEYS command then it's OK, buf if this affects other commands...

Comment From: itamarhaber

This issue appears to be stale so it'll be closed - please feel free to reopen or create a new one if needed.