Redis has: - RPOPLPUSH - BRPOPLPUSH
It would be nice to have their "opposites" for symmetry: - LPOPRPUSH - BLPOPRPUSH
This would allow an application to easily use these variants if they currently manage a list with RPUSH to push a message and BLPOP to fetch a message.
Comment From: foxx
+1
Comment From: drasill
+1
Comment From: Elbandi
LPOPRPUSH: https://github.com/Elbandi/redis/commit/d4f329029116dbd722ca3224e050265dcb2d4818
Comment From: foxx
@antirez Any idea if that commit mentioned before is good for merging?
Comment From: antirez
This commit exactly, very unlikely as it's against 2.4 branch and Redis implementation of blocking operations in Redis >= 2.6 is completely different :-) About the feature itself, I'm thinking about it for Redis 2.8, so I changed the milestone of this issue to 2.8 and check in a few weeks if I'll implement this. I can't reply right now as this is a design issue and I think to analyze the affair in depth before replying. Thanks!
Comment From: foxx
Thank you @antirez! <3
Comment From: jschwertfeger
:+1:
We use RPOPLPUSH to implement the reliable queue pattern; move items from a "pending" queue into a "processing" queue. Unfortunately RPOPLPUSH only gets you half-way there. In the case the processing client goes down while processing an item that is now already in the "processing" queue, it has to move that item back to the head of the pending queue on restart. That's what LPOPRPUSH would do.
Comment From: tpena
:+1:
Comment From: antirez
Sorry, moved again to 3.0... something we'll end having in the long run but probably needs some refactoring and not an absolute priority now that with Lua scripting you can at least implement LPOPRPUSH (without the blocking semantics of course).
Comment From: foxx
@antirez Thanks for the update - sadly I'm not a C guy so I can't even contribute.. however more than happy to lend an extra pair of hands for testing etc when ready.
Comment From: jhoblitt
+1 for this feature.
Comment From: gresrun
+1
Comment From: mmueller
I might be missing something, but I don't think commit c491db5 actually fixes this issue. (As in I don't see it adding the LPOPRPUSH command anywhere.)
Comment From: mattsta
I might be missing something,
Yeah, user JackieXie168 has polluted lots of Redis issues by doing something bad with GitHub (or GitHub did something bad with their repos). Somehow their issues are linked back to the main Redis issues even though they have nothing in common. It's pretty bad seeing fake "references" all the time, but we can't fix it from our side.
It's even quintuply bad because of the way GitHub makes the commit reference look like they are inside the Redis repo when they aren't actually there at all. Look at the tree for the bad commit referenced here: https://github.com/antirez/redis/tree/c491db5597aa0ec3cd2b8f20c6b444f7248f7b48 — It's an entirely different project!
Comment From: mmueller
Ugh, that sucks, but thanks for explaining what happened. Can this issue be reopened then?
Comment From: mattsta
Can this issue be reopened then?
This issue isn't closed! :)
The [CLOSED] above is referencing another irrelevant cross-repository issue leak: https://github.com/gresrun/jesque/pull/43
Comment From: mmueller
... I'll shut up now. Thanks guys.
Comment From: mgsloan
+1
Comment From: DavidBennettPIO
+1
Comment From: felipou
I did a very quick test just copying the rpoplpush command code and replacing REDIS_HEAD with REDIS_TAIL and vice-versa, and created a working lpoprpush command easily. I'll try to tidy up everything until tomorrow and create a pull-request (including a test case and documentation). Is there anything else I should be aware of? Some special condition or corner case?
Comment From: dopesong
Any news of this?
Comment From: felipou
I created a pull request with these commands, but it still needs some work, I'm hoping to get some help here :) https://github.com/antirez/redis/pull/2664
Comment From: dopesong
@antirez Meybe some opinion from you?
Comment From: mrkamel
+1
Comment From: QMonkey
+1
Comment From: davixx
+1
Comment From: tpetmanson
+1
Comment From: uninxc
+1
Comment From: foxx
Hey @antirez, any thoughts on whether this feature would still be accepted? Is there anything I can do to help get this done? (can't do much C, but happy to help out with testing)
Comment From: pkatloktra
+1
Comment From: itamarhaber
FYIs: LPOPRPUSH is already a part of the rxlists module - once the API will provide support for blocking commands, the B variant will be added too.
Comment From: viyatgandhi
+1 for this feature.
Comment From: endseeker
+1
Comment From: ghanbari
+1
Comment From: taf2
Is this right?
local src_list, dst_list = ARGV[1], ARGV[2];
local value = redis.call('lpop', src_list)
if value then -- avoid pushing nils
redis.call('rpush', dst_list, value)
end
return value
but why are there implementations like this: https://github.com/edgurgel/verk/blob/master/priv/lpop_rpush_src_dest.lua
LRANGE key 0 -1 # gets everything expensive? Wouldn't LLEN here be much more efficient?
Comment From: itamarhaber
Seems to do the trick.
As for the implementation you've referred to, I guess that the author didn't learn about the RENAME command. Definitely inefficient.
On Mar 16, 2017 10:22 PM, "Todd" notifications@github.com wrote:
Is this right?
local src_list, dst_list = ARGV[1], ARGV[2];local value = redis.call('lpop', src_list)if value then -- avoid pushing nils redis.call('rpush', dst_list, value)endreturn value
but why are there implementations like this: https://github.com/edgurgel/ verk/blob/master/priv/lpop_rpush_src_dest.lua
LRANGE key 0 -1 # gets everything expensive? Wouldn't LLEN here be much more efficient?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/658#issuecomment-287179572, or mute the thread https://github.com/notifications/unsubscribe-auth/AFx1_B-WZHYh72cxl0rbPisZ05qxrNCBks5rmZoMgaJpZM4AIo35 .
Comment From: r3h0
+1 PR #2664 looks like it addresses this
Comment From: skanagavelu
Definitely we need this support, because when we return a work back to source due to work process failure, it should be added on the right side of the source work list to get the high priority to pop next since it is older work event than any other work in the source queue. Else Need to write LUA that has to LPOP from WorkerQ, RPUSH to SrcQ. And these two commands are not atomic when redis server failure happen in between these two commands during the EVAL execution. So we need single atomic command to do this.
Comment From: workplaylifecycle
Nice to have
Comment From: kingjerod
Would still like to see an lpoprpush for a proper worker queue.
Comment From: taf2
Just to confirm @kingjerod
local src_list, dst_list = ARGV[1], ARGV[2];
local value = redis.call('lpop', src_list)
if value then -- avoid pushing nils
redis.call('rpush', dst_list, value)
end
return value
^^ seems to work
Comment From: gloventRehan
This does solve the problem, assumes we are talking about the "Reliable Queuing" as described on Redis docs. Only downside is that it is 2 extra commands and your system must be idempotent:
- RPOPLPUSH back from your "Processing Queue" to your "Main Queue"
- Take the message returned from the above command and then LPUSH it onto the "Main Queue" (At this stage you have the message at both the head and tail of the "Main Queue")
- Do LREM 1 for the message on "Main Queue" (to remove it from the tail, most left)
This method at least promises that if the system doing these commands goes down after the first operation, the message will be on the most left or if it goes down after the second operation there will be 2 copies on your queue. BUT no message will be lost.
Comment From: taf2
@gloventRehan i might be mis understanding this part:
A Redis script is transactional by definition, so everything you can do with a Redis transaction, you can also do with a script, and usually the script will be both simpler and faster.
This duplication is due to the fact that scripting was introduced in Redis 2.6 while transactions already existed long before. However we are unlikely to remove the support for transactions in the short time because it seems semantically opportune that even without resorting to Redis scripting it is still possible to avoid race conditions, especially since the implementation complexity of Redis transactions is minimal. see: https://redis.io/topics/transactions#redis-scripting-and-transactions
That said, maybe there is no rollback within the context of the running script?
Would it just make sense to wrap the commands in the script inside MULTI EXEC?
Comment From: gloventRehan
I am not exactly sure how Redis internals work, but I am sure those 3 steps can easily be written in a Lua script yes.
Currently I am just doing three separate client calls since the project I am working on does not require high transnational rates, just reliability of always processing a message and never losing them
Comment From: sify21
@gloventRehan What if the node is down before rpush when lpop has benn executed?
Comment From: felipou
I finally finished implementing these commands, after almost 5 years: https://github.com/antirez/redis/pull/6929
Comment From: oranagra
solved by https://github.com/redis/redis/pull/6929