Building a tool similar to Sidekiq but that works with multiple processes, dedicated for web crawling.

We use the queue where we need: - unique values - ordered by the priority

Sorted sets sound like a great option and they have BZPOPMIN/MAX. While the work is picked up from Redis we then need to push it back to another sorted set that holds "work in progress".

Depending on a result move it either to throttled, suspended, failed, etc sets, keeping the original score.

The fact we need to get data from Redis and push it back creates a point of failure in the case of an unexpected error, segmentation-fault, or other disruptions the work fetched from Redis will be lost.

Instead, it would be great to have atomic operations such as BLMOVE for lists. This way we can move jobs back and forth without the risk of losing them and properly recover after a failure.

Comment From: heaven

Forgot to mention that having non-blocking versions of these would also be fantastic.

Comment From: itamarhaber

Hello @heaven - I like what you're saying, which sounds to me like: BZMOVE <source> <destination> MIN|MAX [BYSCORE | BYLEX] <timeout> (and a ZMOVE w/o the timeout).

I'm unsure about SMOVE's necessity, especially given SPOP's semi-randomness, so more details would probably help me to understand.

Comment From: heaven

Hello @itamarhaber, regarding sets we also have a use case where we could use them instead of lists to prevent duplicates in the queue and SPOP doesn't work due to race conditions when multiple workers pick up the same job. But our primary missing feature is BZMOVE which would make our application quite bulletproof.

Comment From: VeaaC

I am definitely looking forward to BZMOVE, as it enables us to implement reliable per-customer-FIFO priority queues much more efficiently.

Comment From: piskvorky

+1: we have an almost identical need for atomic ZMOVE / BZMOVE. Also to prevent race conditions when shuffling elements between work-in-progress ordered sets.

Comment From: gabzim

Is this still in scope?

Comment From: guybe7

[B]ZMOVE added in https://github.com/redis/redis/pull/10430