Redis BLPOP with LIMIT option - Nineya|java/go/python

First of all, thanks very much for all your passion and this awesome piece of software! Can't wait to see how it evolves even further (Redis-Cluster etc.).. keep up the good work :-)

Why a LIMIT option for BLPOP?

For Redis-backed queuing components (such as Resque or Sidekiq), BLPOP is an essential command for pulling new jobs. However, right now every BLPOP only delivers at most one single entry, which is a problem when jobs are small and very frequent, because pulling the jobs creates lots of overhead. One workaround is to use several connections pulling jobs, again at the cost of connection and bandwidth overhead.

My proposal thus is to have a command that allows retrieving multiple list items in a blocking way, i.e. a LIMIT option for BLPOP. This would minimize overhead and increase performance for very frequent pulls. Other queueing backends support this functionality already (e.g. Amazon SQS, where the LIMIT can be 10 at max).

An example using the resulting command specification would be: BLPOP list1 list2 list3 0 LIMIT 10

Where up to 10 items are popped from list1, list2, or list3 respectively, waiting for the first item indefinitely in case there is none.

LIMIT option with delay parameter

One question that arises IMHO is how to deal with pulling items in batches that are added one by one. In this case, the first BLPOP would return one item, and only the next BLPOP would return subsequent items (that were added in the meantime).

This is probably not a big issue, but it could be even optimized by giving a second parameter for LIMIT that specifies a delay (in milliseconds) that is triggered when popping the first item after having waited for it. This would further maximize the throughput of very frequent pulls.

The same example including the second proposal: BLPOP list1 list2 list3 0 LIMIT 10 5

Pops up to 10 items from the lists. If there is none, waiting for the first item plus another 5 milliseconds, popping up to 9 more items, then returning them.

Other blocking commands

The same functionality could naturally be applied to other blocking commands, such as BRPOP and BRPOPLPUSH.

Thanks very much for considering.

Comment From: sylvinus

+1, this would help us speed up our task queue.

Comment From: Plasma

Comment From: coodoing

Comment From: kainwinterheart

Comment From: springside

+1, it is very useful for queuing job system. why it keep open for 2 years....

Comment From: vivekn

Comment From: antirez

I agree this is a good idea. I don't think the two params LIMIT is going to be needed since in that case it means we are in a low messages-per-second situation so there is not much to optimize. Since it's a major change, please contribute a Redis Change Proposal if you can. Here are the instructions: https://github.com/redis/redis-rcp

Comment From: skyrocknroll

Comment From: tzickel

I have written a proof of concept of this issue at https://github.com/tzickel/redis/tree/limit

I am not versed in redis's code base so I would be happy to know if I did any mistakes.

The added commands in that branch are: MLPOP key [key ...] limit MRPOP key [key ...] limit BMLPOP key [key ...] limit timeout BMRPOP key [key ...] limit timeout

Where limit is 0 for getting all the available data or a positive integer for getting up to the given entries. The return is a bulk reply with the list of key1 value1 key2 value2 etc....

Currently if the blocking commands are blocked and a LUA / multi command puts data in multiple keys that are being blocked on, only multiple values from the first key will be returned. If the commands aren't blocked then multiple values from all the keys (up to limit) would be returned.

New commands must be defined because "BLPOP list1 list2 list3 0 LIMIT 10" are all valid BLPOP key names. And because LUA scripts can't run BLPOP the functionality of BMLPOP can't be replicated in a script.

Comment From: tim-kos

Comment From: yeongsheng-tan

Comment From: alexnavis

+1. This is be really helpful to avoid lot of calls for time or performance sensitive processes.