The problem/use-case that the feature addresses Addresses the problem similar to LMOVE where the value that is being popped from the source keys(can take in multiple source keys) is saved atomically in the destination key before returning to the client.
A description of the problem that the feature will solve, or the use-case with which the feature will be used. Currently, In our team, we are using the BLPOP command to pop a value from multiple keys(similar value type with different priorities each having its own queue). When the element is popped, and if the process that is handling the element dies then there is no way to retry that.
Description of the feature * Address this situation by reusing the existing LMOVE command and extending it into multiple source keys where each popped element gets saved into the destination key as backup. * A new command called LMMOVE, where an element gets popped from one of the multiple source keys and gets added into the destination key before it is returned to the requesting client. Similar to LMOVE, the element can be popped from left|right and pushed to left|right.
A description of what you want to happen. Similar to RPOPLPUSH, LMOVE where the element that is getting popped from the source list is being saved into the destination list before it is getting returned to the client but instead of using a single-source list, want to use multiple source lists where the popped element gets saved into the backup list.
Alternatives you've considered Until the popped element is persisted atomically, once the value is returned to the client by Redis, the client process can die which leads to the same issue that we are facing right now.
Any alternative solutions or features you've considered, including references to existing open and closed feature requests in this repository. Was not able to find any issues in Redis GitHub similar to the problem that we are facing.
Additional information
New commands syntax. 1. LMMOVE key [key ...] destination LEFT|RIGHT LEFT|RIGHT 2. BLMMOVE key [key ...] destination LEFT|RIGHT LEFT|RIGHT timeout
Any additional information that is relevant to the feature request. I have got the basic functionality working. If this feature request looks good can open an initial PR with current changes and work through it.
Comment From: madolson
Makes sense to me as a feature.
Comment From: pvs-praneeth
will open a PR and work through this :)
Comment From: zuiderkwast
I like the idea. Just some design choices to consider.
LMMOVE key [key ...] destination LEFT|RIGHT LEFT|RIGHT BLMMOVE key [key ...] destination LEFT|RIGHT LEFT|RIGHT timeout
We can simply allow LMOVE and BLMOVE to take multiple source keys. That syntax would be backward compatible. Currently, they have these syntaxes:
LMOVE source destination LEFT|RIGHT LEFT|RIGHT
BLMOVE source destination LEFT|RIGHT LEFT|RIGHT timeout
If not, and if we instead compare with (B)LMPOP, note that they have a numkeys parameter. Maybe we want to consider that for (B)LMMOVE too?
LMPOP numkeys key [key ...] LEFT|RIGHT [COUNT count]
BLMPOP timeout numkeys key [key ...] LEFT|RIGHT [COUNT count]
Comment From: pvs-praneeth
- I can extend the LMOVE and BLMOVE functionality to fit in the current use case.
- IMO, I don't see a value of adding an additional parameter with
numkeys. We can always get the count of the keys from the arguments themselves. Let me know if there is anything that I am missing related to the second option. Would be more than happy to take a look at it.
Comment From: zuiderkwast
I like option 1 = extend LMOVE and BLMOVE. @madolson WDYT?
The benefit of the numkeys in some commands is that it's easier for tools (such as cluster clients libraries) to figure out which arguments are keys and which are other stuff in cases where there are multiple optional arguments in the end. In this case there are only mandatory arguments (LEFT|RIGHT) afterwards so don't think it's necessary.
Comment From: sundb
@zuiderkwast It looks like adding numkeys will result in no backward compatibility.
How do we distinguish between the following two cases?
LMOVE 1 destination LEFT|RIGHT LEFT|RIGHT
LMOVE 1 sourcekey destination LEFT|RIGHT LEFT|RIGHT
Comment From: zuiderkwast
@sundb i never suggested adding numkeys to an existing command.
Comment From: sundb
@zuiderkwast Ohh, sorry, tt seems I misunderstood what you said.
Comment From: madolson
@zuiderkwast I didn't quite follow what the proposal was. Something like this?
LMOVE source1 [source2 ...] destination LEFT|RIGHT LEFT|RIGHT
We identify the destination as the 3rd argument from the last? One downside of this approach is that it constrains the command, and we can no longer add further arguments.
I would prefer adding a new command, even if it's ugly:
LMMOVE destination numkeys [srckey1 ...] LEFT|RIGHT LEFT|RIGHT
It's more inline with the newer command paradigms, like LMPOP you called out.
Comment From: zuiderkwast
Thx @madolson, yes those are the two options I wanted to consider. @pvs-praneeth I guess the differences are small enough so you can make a PR and continue the discussion there. :-)
Comment From: pvs-praneeth
Thanks for the inputs @zuiderkwast and @madolson. Moving forward with this syntax!!
LMMOVE destination numkeys [srckey1 ...] LEFT|RIGHT LEFT|RIGHT
Comment From: zuiderkwast
Needs to be said: It's slightly surprising to have the destination before the source, but I can see why (so that ppl don't have to ask "does numkeys include the source and destination keys or only the source keys?"). I don't mind though. :-)
Comment From: pvs-praneeth
I have opened a PR for this feature request. I am new to C coding & tried my best to fit in the changes. Let me know if there are any concerns or issues, will address them.
Comment From: SoulPancake
Is this still planned to be taken up? @madolson
Comment From: madolson
@SoulPancake Not really. I do have a question though, is there a reason that multi-exec doesn't work for the non-blocking case? I understand that blocking will not work since those aren't allowed in multi-execs.