The problem/use-case that the feature addresses

Redis supports a handful of block commands, some of them are very important to implement some patterns efficiently without polling, such as for example reliable queues (using BRPOPLPUSH).

However, when implementing more complex behaviours we need to rely on lua scripts. Good examples (if I may add being the main author) are the Bull/BullMQ libraries.

It has been extremely complicated to build a queue system on top of BRPOPLPUSH and still give certain guarantees, specially for more advanced use cases, because often what you want is being able to run a LUA script as soon as an element in the queue has been moved from "wait" to "active". However since you cannot perform a blocking command AND a LUA script atomically, I had to rely on complicated mechanisms in order to give certain guarantees. For more more advanced use cases it becomes really complex and in some cases just impossible to find a robust solution, for example priority queues based on ZSETs are impossible to implement robustly, even if we have a blocking BZPOPMIN command, we cannot both block and move the queue item to its next state in the queue.

I know the question regarding using blocking commands from inside LUA has been risen before. It is my understanding that this is quite difficult to implement, not to mention resource intensive since you may potentially need a lot of LUA contexts kept alive in parallel until the blocking calls are all resolved or timed-out.

Description of the feature

My proposal is to add two new commands, BEVAL and BEVALSHA:

BEVAL srcKey script numkeys [key [key ...]] [arg [arg ...]]

BEVALSHA srcKey sha1 numkeys [key [key ...]] [arg [arg ...]]

These commands would work exactly the same as their non blocking counterparts, the only difference is that they will block until the srcKey is not empty. srcKey could be any Redis supported key type.

It is my understanding that this would be vastly easier to implement than supporting blocking commands inside LUA scripts or in MULTI commands, and at the same time it gives a piece of functionality that enables a lot of new use cases that would be very difficult if not impossible to implement otherwise.

Alternatives you've considered

The only alternative I know of is using Redis modules.

Additional information

N/A

Comment From: manast

And of course if this is implemented, it would be important to add also a BFCALL command with similar blocking behaviour.

Comment From: itamarhaber

/cc @MeirShpilraien @yossigo

Comment From: madolson

This sounds like a good idea to me. Any thoughts about how streams (and modules) would work? Unlike sorted sets and list blocking, streams support blocking based off a position in the stream, and modules could also have different semantic meaning.

This issue also reminds me of https://github.com/redis/redis/issues/8372, which allows triggering scripts based off of events, which maybe is more robust than having all this blocking stuff.

Comment From: manast

@madolson I guess support for streams could be added with some extra parameters for specifying the ID. Maybe something like: 

BEVAL srcKey [ID] script numkeys [key [key ...]] [arg [arg ...]]

and throw an error if ID is used with keys not holding a stream, etc. In any case, this is just a matter of formalizing a good interface, I think this can be easily improved if we get more people involved in this feature.

Regarding modules support, I do think it is more than enough to support the built in types in Redis.

The nice thing about this proposal vs #8372 is that it is very simple to implement and still opens for a lot of new scenarios that are currently impossible to implement without relying on modules. So it is a lot of value for small cost in my opinion.

Comment From: soloestoy

it's very interesting, a block script can make all datatypes support block mode, and if we decide accept it we should design the new command format carefully, especially about the key spec, avoid introduce another ugly command like migrate...