Redis Support for newer Lua versions?

It seems that we're stuck with Lua 5.1 for some years now. And it seems an upgrade might not be wanted because of backwards compatibility

Is there any way to make it possible to support newer versions? If not what are the reasons?

Comment From: itamarhaber

Hello @OneFirefly

Is there any way to make it possible to support newer versions?

"Anything is possible" :)

That being the case, what are you finding missing in 5.1, or put differently what do you need from 5.3(4)?

Comment From: OneFirefly

Hello @itamarhaber, thanks for asking.

What I am currently missing from Lua 5.1 is goto (comes with 5.2) Not that I think using goto is in general good practice, but it is valuable inside nested loops

Comment From: itamarhaber

Good to know, thanks for sharing @OneFirefly :)

Comment From: antirez

Something that is potentially feasible is to support also a new version of Lua, using a new form of EVAL taking as input also the Lua version. However a problem with that is that I'm not sure it is possible to bind multiple versions of the Lua interpreter in the same executable. I'm not sure if the Lua library supports that. And even in such case, that would be a bit a mess to handle both versions inside scripting.c maybe? In some way, it could be simpler to selectively port certain Lua 5.3 features to 5.1 maybe, but this requires expertise with the Lua core. It's a complex situation...

Comment From: antirez

I wonder if Lua 5.3 supports a backward compatible mode. So that the same interpreter could be switched to version 5.1/5.3 with some API call. That would be ideal.

Comment From: itamarhaber

I wonder if Lua 5.3 supports a backward compatible mode.

AFAIR no, it doesn't.

Comment From: neomantra

I've spent a lot of time trying to get other Lua runtimes (mostly LuaJIT which is fully 5.1 compatible) into Redis. In my later experiments, I really wanted to get LuaJIT's FFI working in a Redis module (https://github.com/neomantra/redis-mod_luajit). I eventually abandoned it for various reasons.

The only thing that every really worked was building a Lua version in the Redis source tree or hacking the Makefile to link a different liblua.a. If Redis dynamically linked Lua, then it would be easier to swap out. But, I also appreciate why all the Redis dependencies are statically linked.

So, if OP really needs a newer Lua, they can do their own builds and there probably aren't any Redis code changes needed. But, two Lua runtimes cannot coexist in the same Redis process.

For the idea of selectively porting certain newer Lua features into Lua 5.1, LuaJIT does do this. Mike Pall has been very careful about what features to include. As it's a completely different implementation, Redis cannot copy that work directly and it would be a big effort to do this in the plain Lua 5.1 core. Here's what LuaJIT supports (including goto): * https://luajit.org/extensions.html#lua52

Here are the changes between Lua 5.1 and 5.2 and 5.3: * https://www.lua.org/manual/5.2/manual.html#8.1 * https://www.lua.org/manual/5.3/manual.html#8.1

There are projects that help bridge the Lua versions, but some features are intrinsic to the runtime and can't be emulated: * https://github.com/keplerproject/lua-compat-5.3

Comment From: jasonxiaoqin

hello

Comment From: ghost

Hi,

I am also interested in support for lua 5.2 because some other applications need 5.2 and I can't make them coexist. I guess this is the trend to keep things updated and secure? It's becoming more difficult to run Redis on servers where other software requires newer versions of common libs like Lua.

Comment From: TBoshoven

Regarding the question around motivation: I ran into an issue resulting from the lack of integer support in Lua 5.1. LuaJIT seems like it would be a great addition.

Comment From: zuiderkwast

LuaJIT seems like a safer choice than Lua 5.4 (or 5.3, etc.) since LuaJIT is backwards compatible with Lua 5.1 and only adds some candy features on top. Besides, it's fast! Currently we use modules instead of Lua just to speed up things because Lua is too slow (or CPU intensive). If we can get away with a fast Lua, we can probably use a vanilla Redis without modules in many cases.

Comment From: manast

@neomantra what were the reasons you did not finalize the integration of LuaJIT into Redis? According to https://luajit.org/luajit.html it is between 3x-100x faster than Lua, that seems like a non trivial improvement that at least I would really care about for my big lua scripts. Also, regarding having the possibility to swap lua interpreters, I do not think this is a good idea, it is better to have core redis that we can rely upon without having custom instances. If a new major release of Redis would include a better/newer version of Lua that breaks backwards compatibility I do not think this is a huge problem, you either upgrade your scripts for the newer version off Redis or you keep using an older version of Redis. We should not be afraid of a limited amount of breaking changes, otherwise it is quite difficult to progress and improve the software in the long run.

Comment From: manast

It would also be interesting to know if LuaJIT would be an acceptable replacement of current Lua interpreter in Redis, so assuming somebody creates a PR with a working LuaJIT integration, would that PR have any chances of being merged?

Comment From: neomantra

@manast The primary reason I stopped working on the FFI-bindings in redis-mod_luajit is that it required a custom build of Redis. I was already maintaining a simpler custom build for years because my bugfix PRs weren't getting accepted and that was painful. [That situation is much better now and absolutely no hard feelings against the original maintainer, mad respect and he had a lot on his plate.] Anyway, maintaining a custom Redis plus a custom FFI-based module was more than I wanted to do. For my needs, modern C++ worked great for modules. But I'm not even doing that anymore....

That said, LuaJIT is still awesome and I use it in other places. If you watch the LuaJIT repo, you'll see that its maintainer is doing a fantastic job despite the fact that he officially stepped down many years ago.

Comment From: manast

@neomantra I am pretty surprised about how small the LuaJIT module is, I was expecting a much more complex code. Do you think replacing the current Lua integration in Redis by LuaJIT is a reasonable simple work to do? I think the benefit of having LuaJIT would be really great. With 3 to 100 times better performance most Redis modules could be written as lua scripts which would be a huge advantage regarding portability and deployability.

Comment From: zuiderkwast

It's not very hard to hack together a working build. There is a significant overhead in each request so for small scripts, there is no improvement at all, but for larger scripts there is.

@bjosv can you share your results?

Comment From: manast

@zuiderkwast why is there a significant overhead per request? wouldn't the script be precompiled so that the overhead become minimal? Since LuaJIT is being used by games I assume the overhead should be very small, is it not?

Comment From: bjosv

I did an experiment where I added luajit to deps/ so it was possible to build with either lua or luajit. The existing lua in deps contains 4 additional lua libs built-in which needed to be added to luajit aswell ("cjson", "struct", "cmsgpack" and "bit", where "bit" comes from luajit originally, i.e not needed to be added)

When running smaller scripts I couldn't find any greater benefit probably since the overhead noted by @zuiderkwast, but when running more unrealistic scripts it improved, some examples:

Test	Lua	Luajit	Details
For-loop with 1M HSET	1,212s	1,086s
For-loop with 10k PING	0,006s	0,004s
For-loop with 100k PING	0,040s	0,029s
For-loop with 1M PING	0,384s	0,288s
KEYS	1,351s	0,130s	See https://github.com/redis/redis/issues/8077#issuecomment-732343216
KEYS (longer key)	13,208s	0,156s	See https://github.com/redis/redis/issues/8077#issuecomment-732343216

The KEYS results is interesting and it would be interesting to know the reasons. I'll see if I can collect some more results from my notes..

Comment From: bjosv

Here is a comparison for a simple script (best of 5 runs)

Using legacy lua: redis-benchmark -P 10 --threads 2 -n 1000000 eval 'return redis.call("GET", "key")' 0 1.75 seconds 571102.19 requests per second 0.811 avg latency (msec)

Using luajit: redis-benchmark -P 10 --threads 2 -n 1000000 eval 'return redis.call("GET", "key")' 0 1.75 seconds 571102.19 requests per second 0.778 avg latency (msec)

this can be compared to when running the command directly, shows somewhat the overhead when using lua-scripts:

Direct: redis-benchmark -P 10 --threads 2 -n 1000000 GET key 0.75 seconds 1333333.38 requests per second 0.315 avg latency

Comment From: bjosv

..and best of 5 runs using evalsha with redis-cli SCRIPT LOAD 'return redis.call("GET", "key")'

Using legacy lua: redis-benchmark -P 10 --threads 2 -n 1000000 evalsha $SHA 0 1.50 seconds 666222.50 requests per second 0.657 avg latency (msec)

Using luajit: redis-benchmark -P 10 --threads 2 -n 1000000 evalsha $SHA 0 1.50 seconds 666222.50 requests per second 0.659 avg latency (msec)

Comment From: manast

@bjosv that overhead is ugly. I run your test above and then with more redis.calls and I saw that performance drops significantly. In my machine starts with 15k/second, 10k/sec with two calls and 8k/sec with 3 calls:

redis-cli SCRIPT LOAD 'local a = redis.call("GET", "key");local b = redis.call("GET", "key");return redis.call("GET", "key")'

What results do you get with LuaJit?

Comment From: manast

Btw, maybe the way to get rid of the overhead is to implement lua->redis interface using http://luajit.org/ext_ffi.html ?

Comment From: neomantra

For a fairer benchmark, perhaps the Lua test should use EVALSHA and not EVAL... that will remove some repeated Lua parsing (albeit of a simple function) and creation (and GC pressure) of a lua_Function and replace it with a hash-lookup to an existing lua_Function.

Btw, maybe the way to get rid of the overhead is to implement lua->redis interface using http://luajit.org/ext_ffi.html ?

@manast That is what redis-mod_luajit does. It's also why it is so short... it just needs to bind the Redis Module API and add a little scaffolding.

If somebody already has a Redis build with Lua replaced with LuaJIT, it would be a small effort to get that working and run the same similar benchmark.

Comment From: bjosv

@manast when running evalsha using a number of calls in the script I get:

1 call get: lua: 1000000 requests completed in 1.50 seconds, 666222.50 rps, 0.657 (avg latency) luajit: 1000000 requests completed in 1.50 seconds, 666222.50 rps, 0.659

2 call get: lua: 1000000 requests completed in 1.75 seconds, 570125.44 rps, 0.811 luajit: 1000000 requests completed in 1.75 seconds, 571102.19 rps, 0.780

3 call get: lua: 1000000 requests completed in 2.00 seconds, 499500.47 rps, 0.918 luajit: 1000000 requests completed in 2.00 seconds, 499500.47 rps, 0.916

redis-cli SCRIPT FLUSH
SHA=$(redis-cli SCRIPT LOAD 'local a = redis.call("GET", "key");local b = redis.call("GET", "key");return redis.call("GET", "key")')
./src/redis-benchmark -P 10 --threads 2 -n 1000000 EVALSHA $SHA 0

Not the same significant drop as in your runs.

Comment From: manast

@bjosv is it reasonable to conclude from your tests that the 1M calls themselves just takes 0.25s, and that the rest is the overhead of calling EVALSHA? Btw, for such simple tests that just call a C api there is no gain using LuaJit as the tests demonstrates, the kind of scrips I care about looks more like this one: https://github.com/taskforcesh/bullmq/blob/master/src/commands/addJob-8.lua

Comment From: MeirShpilraien

Hey everyone, came across this thread and thought I can contribute to the discussion. I also did a POC with LuaJit before publishing this suggestion: https://github.com/redis/redis/issues/8693. Doing the POC was pretty simple, there were almost no changes needed in the code itself (only changing the makefile). I also reach the same conclusion you did, there is no significant improvement (if any) on small running scripts that is doing basically some redis.call and return. Most of the time on those scripts is spend on executing the commands and parse the replies. I do believe though that for scripts that perform a significant amount of logic we will see improvement, but as far as I know, this is not what Lua scripts was meant for, a script that runs for a long time (even 50-100ms) is considered an antipattern because during the entire execution Redis is blocked.

There were also other problems/concerns that I encounter, such as: 1. The status of the project, the last release was 4 years ago. I did try to ask about releases and got this answer: https://github.com/LuaJIT/LuaJIT/issues/665 2. I found some blogs/comments that raise concerns about how secure LuaJit is (especially when combined with FFI). Jit code is considered highly complicate and with complicated code comes higher risks for vulnerabilities. 3. There were some technical issues, for example, the JIT code does not call Lua hooks and so the scripts turn unkillable.

I do believe that the last concerns/problems should not stop us, the only thing that lead me to drop it is that for the use-case Lua scripts was built for, I saw no significant improvement. If at some point we will decide to use Lua for other use-cases (maybe long-running scripts?) we should definitely consider this.

Comment From: bjosv

@bjosv is it reasonable to conclude from your tests that the 1M calls themselves just takes 0.25s, and that the rest is the overhead of calling EVALSHA?

@manast As a reference to the results, running the simplest command that came up takes takes 0.75s ./src/redis-benchmark -P 10 --threads 2 -n 1000000 PING

the kind of scrips I care about looks more like this one..

Nice, Ill see if this can be used in a test.

Comment From: manast

I do believe that the last concerns/problems should not stop us, the only thing that lead me to drop it is that for the use-case Lua scripts was built for, I saw no significant improvement. If at some point we will decide to use Lua for other use-cases (maybe long-running scripts?) we should definitely consider this.

There are for sure use cases where faster lua scripts is beneficial, Bull/BullMQ (https://github.com/optimalbits/bull) has a huge user base and all functionality is implemented in rather complex lua scripts, still we can handle 5k-10k jobs per second.

Also, the functionality provided by BullMQ is impossible to achieve without lua scripts unless implemented as a module, but as you know modules are not very popular outside of redislabs cloud ecosystem.

Comment From: yossigo

@manast Executing 5k-10k ops/sec means you're still well under 1ms (avg) for a single Lua script, which goes along with what @MeirShpilraien mentioned.

I don't know how much LuaJit could improve that but I suspect that it won't be by much, because as stated above, the overhead of redis.call() probably outweighs anything the Lua VM itself does (unless you're number crunching or something - but then scripts would probably be way slower anyway).

BTW BullMQ looks like a really cool project! I wasn't familiar with it.

Comment From: bjosv

@manast I have the redis luajit branch at https://github.com/Nordix/redis/tree/luajit if you would like to build and test something for BullMQ, its mainly some trivial changes to the makefiles.

Comment From: jeffreydwalter

+1 for goto support.

Comment From: tianshuang

Lua 5.1 lacks support for long integers, see #5261.

Execute the following command through redis-cli: EVAL 'return tonumber("2022081100000000001")' 0, it will return (integer) 2022081100000000000, this is not the result I want.

Comment From: oranagra

sorry for lack of update here, there were several issues on the same topic. current decision posted here a while back: https://github.com/redis/redis/pull/8180#issuecomment-911265500

basically:

We don't currently plan upgrade Lua Lua 5.4 is not backwards compatible with 5.1, so if we wanna support it, we'll need to support both, and we eventually concluded that the shortcomings of 5.1 are not worth that complication. instead we considered to switch to Lua JIT, which is 5.1 compatible and more maintained, but we don't have a plan for doing that anytime soon either.

p.s. we may support Java Script some day soon, see in #8693, and #10126.