In the comments under discussion "Redis and scripting" (http://www.antirez.com/post/redis-and-scripting.html) there were a lot of suggestions of using LuaJIT2 instead of standart Lua interpreter, but no response from antirez about.
Why LuaJIT2 is good for Redis? - API-compatible with Lua 5.1, so you shouldn't face any problems due to replacing - Much faster than original interpreter - Uses less memory - Have built-in Lua BitOp (I've seen It's planned to built it in by default into Redis) - Works everyehere Redis does (see http://luajit.org/install.html)
You'll probably say that the speedup is insignificant, because the time spent in the interpreter much less than time spent to i/o and commands processing, but I disagree. If I'll be able to process command at least 5% faster, It will make me much happier.
So, why don't you use LuaJIT2?
Comment From: jbergstroem
While at it, I would like to see something in style with "USE_LUA_EXTERNAL" instead of statically linking to a bundled version of lua. This would make it easier to switch lua implementation as well.
Comment From: eoranged
@jbergstroem AFAIK, external dependences isn't Redis-way.
And Redis also uses a bit modified version of Lua (check out comits), so you'll need to replace some functions in your dinamicaly linked library by your own.
Comment From: jbergstroem
@utros: It would sure be a nice way to handle the feature requested in this item. Lets not hijack this thread, but do you know if there's more information about why using shared libraries isn't the "Redis way"?
Comment From: eoranged
@jbergstroem It doesn't matter how do you link the library, but as i see, redis is intended to be small, fast and have no external dependences (only computer, supported OS and c compiler).
It is just my point of view and redis developers may think different way.
Comment From: antirez
Hello,
@utros: about Lua JIT, sure since it is a drop in replacement I've a long term plan of testing it if we'll incur into speed issues with the current interpreter. It is a switch that we can always do and I'm taking this in mind. However as long as there is no evidence that it is a big win for us (probably not, as we never execute algorithmic-style code in scripts) otherwise having the standard Lua interpreter that compiles fast and clean with an "ANSI" target is better.
@jbergstoem: we'll never support linking against a different version of Lua, as from our point of view scripting is a feature of Redis, so you can't exclude scripting nor you can modify it. A redis version 2.6.5 (for instance) will be able to run a set of scripts, in all the supported architectures, without any differences from a computer to another. So we consider the behavior and functions available in a specific Lua interpreter as our feature and specification. For the same reason we'll probably deny the possibility of loading external code into the interpreter.
Thanks, closing the issue for now since it is a long term business ;)
Comment From: neomantra
I made of fork of Redis which uses the LuaJIT VM instead of the vanilla ("PUC-Lua") VM. You can find it here: https://github.com/neomantra/redis/tree/luajit
It passes all the tests, but doesn't include the libraries that Redis bundles (cjson, struct, and cmsgpack). In simple benchmarks, it does not perform better than the Lua VM: https://github.com/neomantra/redis/wiki/LuaJIT-Performance
I'm working on moving more of the implementation from C to Lua which does help some (~15% in those tests)... I'll post more when it is ready (if ever). My personal goal is to store C structures in Redis and then manipulate them in-server using the LuaJIT FFI. This is fraught with security/stability issues (e.g. you can very easily deref a null pointer and crash the server), so I don't expect any of this to ever converge with mainline Redis. But it's a fun experiment!
Comment From: Suor
Here is my use case for this. I use redis for cache and store invalidation information using redis sets. So I need to update both cache and invalidation information atomically. I used transactions before, but moved to lua scripts, which made my code both simpler and faster. However, now I have some non-trivial lua scripts and the majoriy of time they take are not spent in redis calls.
Comment From: antirez
Hey, I want to be honest: currently it is very unlikely we move away from the standard Lua interpreter, but moreover without benchmarks the ability of this idea to move forward is near-zero... To propose a speed related change without good quality benchmarks is mostly pointless.
Comment From: antirez
p.s and @neomantra initial attempts at testing it shows no performance gains for simple scripts. Another evidence of problems with this approach. Btw also make sure to test stuff with latest Redis since Lua scripting execution script was improved. It is mostly dominated by cache misses and alloc/dealloc patterns.
Comment From: Suor
I understand it's unlikely. Just wanted to show that there are some interest out there. I'll return with benchmarks once I get to that.
Comment From: neomantra
@Suor You can do an easy first pass of this by installing LuaJIT on your system using LD_PRELOAD since the LuaJIT .so should be drop-in compatible with the Lua 5.1 .so.
Beyond that, my (abandoned) branch tried to make the Redis implementation spend more time in Lua and leverage the FFI to access Redis structures; in particular I wanted long LuaJIT traces (i.e. no interpreter) and zero marshalling. I also did work to make it easier to have a Lua sandbox (e.g. like to specify the loading of the FFI library, which was my impetus for trying this at all).
If you are doing this for performance, also try LuaJIT's 2.1 branch. I think that some of its new features (e.g. trace stitching) might improve the situation here. But it is alpha and people are still reporting bugs.
Comment From: mattsta
Also of note: FreeBSD provides an option to automatically patch in support for LuaJIT instead of regular Lua when installing from ports: http://svnweb.freebsd.org/ports/head/databases/redis/Makefile?view=markup&pathrev=357277
For use on your own systems, it's pretty easy to replace Lua with LuaJIT inside Redis since LuaJIT is based on Lua 5.1 and Redis uses regular Lua 5.1.
Comment From: coleifer
For what it's worth I've put together a very minimal fork of the 5.0.3 sources, embedded LuaJIT 2.0.5 (latest stable). All tests are passing. https://github.com/coleifer/redis-luajit
Comment From: zcaudate
@coleifer, have you been using the luajit fork in production? and if you have, do you notice any differences?
Comment From: coleifer
No, not in production, just for personal/hobby projects. It works fine, but I have not bothered to profile it, nor do I have any measurements to share.
Comment From: zcaudate
@coleifer: I did some quick and dirty tests.
This is about 30x quicker on luajit.
EVAL "return (function ()\n local a = 0\n while a < 1000000000 do\n a = (a + 1)\n end\n return a\nend)()" 0
Though calling into redis - ie GET and SET is roughly the same.
Would it be possible for you to do an update to the newest 5.x version with LuaJIT-2.1.0-beta3?
I did try to compile the newest luajit on your repo (just changing the versions on the Makefile) but am having issues compiling. I think I'm missing something.
/usr/bin/ld: scripting.o: in function `luaLoadLibraries':
/root/dev/buffer/redis-luajit/src/scripting.c:851: undefined reference to `luaopen_cjson'
/usr/bin/ld: /root/dev/buffer/redis-luajit/src/scripting.c:852: undefined reference to `luaopen_struct'
/usr/bin/ld: /root/dev/buffer/redis-luajit/src/scripting.c:853: undefined reference to `luaopen_cmsgpack'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:219: redis-server] Error 1
make[1]: Leaving directory '/root/dev/buffer/redis-luajit/src'
Comment From: zcaudate
I figured out why those errors were occurring.
I've reconfigured the 6.2 and 5.0 branches to use the newest luajit v2.1 branch:
https://github.com/zcaudate/redis-luajit/tree/6.2-luajit https://github.com/zcaudate/redis-luajit/tree/5.0-luajit
Comment From: zcaudate
GitHub workflow has also been setup.
All 5.0 Tests are passing on x86_64 except one (msgpack related): - https://github.com/zcaudate/redis-luajit/runs/4647189983?check_suite_focus=true
*** [err]: EVAL - cmsgpack can pack and unpack circular references? in tests/unit/scripting.tcl
1566
Expected '82a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a17882a17905a17881a178c0 1 1' to equal or match '82a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a17882a17881a178c0a17905a17905a17905a17905a17905a17905a17905a17905 1 1'
All 6.2 Tests are passing until [62/64] (Not sure but I think they are experimental)
https://github.com/zcaudate/redis-luajit/runs/4647186532?check_suite_focus=true