Redis Functions - Libraries

Redis Functions PR introduces a new scripting approach. One of the common requests that came from the community is the ability to share code between functions, the following is a proposal of how users can achieve code sharing between different functions.

Before describing the solution we will point out some issues that lead to the idea we will describe later. 1. Code sharing can raise issues like version incompatibility, Assuming function 1 uses code from function 2. Function 2 might update the code in a way that will break function 1 invocation. 2. Code sharing between engines is problematic and can only be done using the FCALL command that will call another function. This raises (again) the topic of nesting function calls which are currently not supported. We keep it out of the scope of this proposal and will be discussed on another issue. 3. Name-spacing, 2 functions might declare same function names internally and this requires name-spacing to identify between them.

those points led us to these conclusions: 1. Code sharing between functions on the same engine should be done only by calling the other function entry point. The assumption is that entry points are less likely to introduce a drastic behaviour change because it is also exposed to clients. This proposal does not cover this case and keep it for future improvements. 2. Exposing one function entry point to another function is not enough. What if we want 2 functions to have a shared code which is not a function by itself (and one that uses a richer interface, like passing native language objects as argument)? To solve it, we introduce a new ability that allows batch loading of functions.

The Idea

Instead of allowing a single function creation per FUNCTION CREATE command, we suggest the option to create multiple functions in a single command (for example look at Lua library API section below). Functions that were created together can safely share code between each other without worrying about compatibility issues and versioning. We call this group of functions that was uploaded together a library. Library code is loaded into Redis using the FUNCTION LOAD command. When an engine receives the code given on FUNCTION LOAD command it will be able to register one or more functions. Each engine can decide how to expose the registration capability to the user, some engines will need to run parts of the code in order to know which functions it creates while others will only need to introspect the code. This new concept also changes the way FUNCTIONS are deleted or updated. Libraries are immutable so it is not possible to delete or update a single function inside a library, the entire library needs to be updated/deleted together.

API Changes (compare to unstable)

The following API changes/additions will be made to support library:

New API

• FUNCTION LOAD <ENGINE NAME> <LIBRARY NAME> [DESCRIPTION <LIBRARY DESCRIPTION>] [REPLACE] <LIBRARY CODE> Load a new library into Redis. Arguments:
• ENGINE NAME - the name of the engine to load the library
• LIBRARY NAME - the name of the library
• DESCRIPTION - optional description of the library
• REPLACE - optional, indicate to replace the library with the same name if exists
• LIBRARY CODE - library code

Changed API

• FUNCTION DELETE <LIBRARY NAME> Function delete will get the library name to delete instead of the function name.

• FUNCTION INFO <Library name> [WITHCODE] FUNCTION INFO returns information about libraries and not a single function. If the optional WITHCODE argument is given, the library code is also returned. The information given:

• NAME
• ENGINE
• DESCRIPTION

• FUNCTIONS LIST

• FUNCTION LIST Lists all the libraries. For each library list also the functions it created.

Deleted API

• FUNCTION CREATE

Notice: For easier migration from eval to functions, we can decide to keep the FUNCTION CREATE and treat it as a library with one function. To avoid confusion we believe it's better to drop it but it's debatable.

Lua Library API

When the Lua engine will get the code given to FUNCTIONS LOAD command it will immediately run it. In this run context it will be possible to call redis.register_function. This API will register a new function (with the given arguments that we will describe shortly) and link it to the current library. The arguments given to redis.register_function are: * Function name - the name of the new function to register * Function object - a Lua function object to call when the new function is invoke * Function flags - optional flags as describe here * Function description - optional text describe the function

The redis.register_function will only be available on FUNCTIONS LOAD invocation and will raise an error if used anywhere else. Also, during this function's registration phase, it will not be impossible to call Redis commands (redis.call) or interact with the key space in any way.

Lua library Example

The following example register a library with 2 functions test1 and test2 that simply return 2 and 3 respectively

> FUNCTIONS LOAD LUA lib1 “
    local function add1(x)
        return x + 1
    end

    local function test1()
        return add1(1)
    end

    local function test2()
        return add1(2)
    end

    redis.register_function(‘test1’, test1)
    redis.register_function(‘test2’, test2)
”
  OK
> FCALL test1 0
  2
> FCALL test2 0
  3
  ```

## Library Functions Flags
We suggest support flags that can be given to Redis when the library creates a function. Those flags are used to tell Redis when it is OK to invoke the function. For example, one flag might indicate to disallow running the function if Redis reached its memory limit. Here is the list of flags we plan to support first (other flags might be added in the future):
* DENY_OOM - do not allow the function to be invoked if the memory limit is reached.
* BREAK_ATOMICITY - indicate that it's OK to kill the function even if it already performed a write command or disallow write * commands if OOM reached.
* NO_REPLICATE - indicate that all the data created by the function is temporary and should not be replicated to replicas/aof. Equivalent to redis.set_repl.

Note: DENY_OOM can solve this issue: https://github.com/redis/redis/issues/8478


**Comment From: madolson**

My gut feeling is that this is over complicating the feature, so in the vain of keeping Redis simple I would want to better understand the motivation and the benefits that are being provided. This is encroaching deeply into a level of complexity that should probably be a module.

> One of the common requests that came from the community is the ability to share code between functions, the following is a proposal of how users can achieve code sharing between different functions.

What is the motivation for this? I have never heard this specific request before, unlike the persistence ask which comes up fairly often. It seems like it would be a more annoying pain point for EVAL, where you have to resend all the scripts all the time, but less of an issue when scripts are persisted.

> Exposing one function entry point to another function is not enough. What if we want 2 functions to have a shared code which is not a function by itself (and one that uses a richer interface)? To solve it, we introduce a new ability that allows batch loading of functions.

Can you elaborate more about this? I assume this is about typing or something.

**Comment From: MeirShpilraien**

@madolson Thanks for the feedback.
> My gut feeling is that this is over complicating the feature, so in the vain of keeping Redis simple I would want to better understand the motivation and the benefits that are being provided.

Basically, today if you have a shared code that you want to be available on 2 scripts (eval or functions) you have to put this shared code on both scripts. If the shared code changes you have the change it on both scripts. The requirement was the ability to share code between scripts so it will be easier to maintain. It is solvable on the client side with pre-processing of the script but it make user life harder.

Regarding complicating the feature, I do not think its complicated, it is different than eval but maybe its a good thing to change it now (after releasing it will be to late). I think the other alternatives are much more complicated (like nested functions call or calling one function from another which cause versioning issue and other problems that was mentioned above). Also notice that code sharing that is done using direct FCALL from one function to another make it impossible to pass native arguments and objects to the internal functions. I also believe it make sense to have initialisation phase, I believe that in the future we will be able to use it for other things as well.

> I have never heard this specific request before

Its been asked on the functions issue: https://github.com/redis/redis/issues/8693#issuecomment-815571778. maybe @itamarhaber  can point out more issue where its been requested.

> It seems like it would be a more annoying pain point for EVAL, where you have to resend all the scripts all the time, but less of an issue when scripts are persisted.

IIUC, you are saying that not having a code sharing on functions is less of an issue than on eval? If so I have to disagree because of the points mentioned above.

> Can you elaborate more about this? I assume this is about typing or something.

Look at the example I posted on the issue:

FUNCTIONS LOAD LUA lib1 “ local function add1(x) return x + 1 end

local function test1()
    return add1(1)
end

local function test2()
    return add1(2)
end

redis.register_function(‘test1’, test1)
redis.register_function(‘test2’, test2)

” OK

FCALL test1 0 2 FCALL test2 0 3 `` You can see there are 2 functiontest1andtest2both usesadd1function which is an internal method not exposed as Redis Function. Without this library proposaladd1would have to be maintained on both functions source code,test1andtest2`.

Comment From: madolson

@MeirShpilraien Thanks for adding more context.

Regarding complicating the feature, I do not think its complicated, it is different than eval but maybe its a good thing to change it now (after releasing it will be to late).

I think it complicates the end user experience, EVAL is really straightforward to understand you just send a script with the arguments. Functions were just a natural extension of that, where you could set the function ahead of time. I think the proposal is counter intuitive, as it adds a layer of indirection. I think the logical answer here is being able to call functions from other functions, which is #8693 (comment) proposed.

Your counter point is we want to share native data types between functions, which isn't unreasonable, but I think it's solving a different problem (performance and code complexity of building the serialization). So I suppose my preference is we should solve the problem of being able to do FCALL to other functions from within an FCALL, but I'm in favor of implementing what is outlined here.

Some assorted implementation comments: 1. FCALL doesn't take in a library, so functions are global, so there is contention when loading multiple libraries that both contain the same function registration. An example is calling redis.register_function('X', x) when another library declares 'X', it's unclear what the behavior is. We need some guard rails here so we don't partially setup a library or make a library inconsistent. Alternatively you also need to provide the library or you could declare the functions you're going to register. 2. What happens on syntax errors generally? Imagine this scenario instead:

FUNCTIONS LOAD LUA lib1 “ local function add1(x) return x + 1 end

local function test1()
    return add1(1)
end

redis.register_function(‘test1’, test1)

local function test2()
    return add1(2) oh no syntax error!
end

redis.register_function(‘test2’, test2)

”

and it fails to register test2, is test1 still registered? It seems like a property of libs should be atomicity in loading. 3. BREAK_ATOMICITY, is this really useful? It seems like it's a really dangerous tool and users are going to shoot themselves with it. You could argue people should be allowed to shoot themselves, but I would prefer we add it when someone wants to do it. 4. Thoughts about calling these namespaces instead of libraries? This is purely subjective, so feel free to disregard if you prefer the library term, it just seems like we are organizing the code into logical areas.

Comment From: oranagra

@madolson i don't think collision of names in function registration is a problem. indeed the function are global (i don't wanna pass a lib name to FCALL), but i don't see a problem in failing registration when we see a conflict.

regarding syntax errors after some code was already registered, that's no problem to rollback and keep the library creation atomic.

when you think of it, function libraries are like modules, a module is loaded and it registers multiple commands (conflicts are not allowed), so functions are the same.

in theory we could have also allowed functions to register commands (i.e. executed as a native command not via FCALL), we can maybe do that some day (there's nothing we do today that blocks us from adding that later), it'll require the library to provide more metadata (like key-specs).

IMHO this could be nice, since the other obvious difference (other than how they're executed) between modules and functions is the sandboxing (i.e. the fact that modules use native code limits their adoption, see #9880), so we can one day even think about them as modules, and allow them to append things to the command table too.

all the text above just goes to show my state of mind. i think that functions won't usually come one by one, they're intended to allow users to add native operations to the server side to manage the application data (so it won't use redis commands anymore, see my post here), so as such, it should be common to load several function together to handle a certain data structure that's composed of several keys, and provide various operations on it).

Comment From: madolson

@oranagra So why wouldn't we just rollback the script if someone tries to register a function that already exists? Based on your explanation that code will be implemented anyways. That seems easy to do and provides natural safe guards.

I don't really like your argument comparing them to modules either, because modules don't add a lot of safety functionality today, and I think functions should be more user friendly than modules since we are making them so easy to access.

I think the idea of one day allowing functions to be mapped to top level of commands seems sane.

Comment From: oranagra

I was comparing them to modules to make the point that they're libraries and not single stand alone scripts. They are aimed at providing a set of application native operations (I.e. Instead of using primitive LPUSH, LPOP, HSET, HDEL), so I expect logic of several operations to share code and I want it to happen in script native ways, and not via nested FCALL which is inconvenient, complex and limited.

Regarding modules not providing safe guards, I agree functions should be much safer, and I agree that if a function already exist we should fail the whole library load and rollback what it did.

From your last post I don't see what we still don't agree on, but in your previous post it seemed you disliked the library concept saying it's too complex. I also thought so at first, but the idea of allowing both FUNCTION CREATE and FUNCTION LIBRARY creation seemed that it'll lead to too much confusion, so I rather have just one.

Comment From: madolson

@oranagra I'm fine with the current proposal if we are adding the safe guards we talked about.

I probably just write too much of what I'm thinking while responding to issues, and sometimes I'll talk about pros and cons and not make my opinion very clear.

Comment From: oranagra

I do that too (as you can see from my long posts)... One of the disadvantages of being able to type fast (it take more time to organize the thoughts). I suppose the only disadvantage is for whoever comes to read it retroactively, too much text to process before conclusions are made. 8-)

Comment From: zuiderkwast

@madolson and @oranagra I'm very happy that you do share your thoughts openly. It's a very including attitude which makes others (myself) feel involved. It's a great power of the Redis community IMO.

Comment From: forkfork

Late to the party here, but would you see this working with a clustered Redis @MeirShpilraien ? I have new nodes coming in regularly, and if the new nodes do not have the function replicated to them as part of core Redis, this makes the feature feel very awkward.

Comment From: oranagra

@forkfork how do you join new nodes to the cluster? this should have been handled by #9938 (see #9899)