Redis [QUESTION] Why do we need distributed lock in Redis if we partition on Key?

Folks,

I read thru https://stackoverflow.com/questions/37569680/why-we-need-distributed-lock-in-the-redis but it is not answering my question.

I went thru the following 1. https://redis.io/topics/distlock to be specific https://redis.io/topics/distlock#the-redlock-algorithm and 2. https://redis.io/topics/partitioning

I understand that partitioning is used to distribute the load across the N nodes. So does this not mean that the interested data will always be in one node? If yes then why should I go for a distributed lock that locks across all N nodes?

Example: 1. Say if I persist a trade with the id 123, then the partition based on the hashing function will work out which node it has to go. Say for the sake of this example, it goes to 0th node. 2. Now my clients (multiple instances of my service) will want to access the trade 123. 3. Redis again based on the hashing is going to find in which Node the trade 123 lives. 4. This means the clients (in reality one instance among the multiple instances i.e only one client ) will lock the trade 123 on the 0th Node. 5. So why will it care to lock all the N nodes? 6. Unless partitioning is done on a particular record across all the Nodes i.e Say a single trade is 1MB in size and it is partitioned across 4 Nodes with 0.25MB in each node.

Kindly share your thoughts. I am sure I am missing something but any leads are much appreciated.

Comment From: sundb

Hey, @hariraogotit, I feel like you're confusing the purpose of the distributed lock, which is not to lock the data in redis, but to lock your business code. If you just want to guarantee id 123, you can just use lua to achieve atomization. Before redlock we used setnx + lua to implement distributed locking, the reason why redlock is needed is to avoid the locking problem caused by single point.

Comment From: hariraogotit

@sundb - Thanks for the reply. Below is what confused me.

I have 5 instances of my microservice which when boots up reads an entry from Redis, processes i.e carries out a workflow of activities and removes the entry from Redis. I want to make sure that my 5 microservices do not step on each other while reading/removing the entry from Redis. Possibly I will have a bunch of Redis instances.

Since I will have bunch of Redis instances but only one redis instance will have the trade 123. I believe I do not need a distributed lock?

Comment From: sundb

You are reading string type data? Your 5 instances will read the same key at the same time? If so, then you need distributed locks, because your 5 microservices may read the key at the same time, and execute the workflow 5 times, and only one instance succeeds in deleting the key.

If you have 5 instances that may read a key at the same time to processing may not make sense, you should use stream to store the ids you need to process, then 5 instances use xread to get a key from the stream, then after carries out a workflow of activities, then use xdel to remove the entry from the stream. This way you can avoid using distributed locks.

Comment From: sundb

@hariraogotit I'm not sure where you got confused about redlock. Do you not understand why 5? Or do you not understand why the lock keys of the 5 redis instances are locked sequentially?

Comment From: hariraogotit

@hariraogotit I'm not sure where you got confused about redlock. Do you not understand why 5? Or do you not understand why the lock keys of the 5 redis instances are locked sequentially?

https://redis.io/topics/distlock#safety-and-liveness-guarantees - tells the objective

https://redis.io/topics/distlock#why-failover-based-implementations-are-not-enough - explains safty violation as the replica does not have the lock as it was not transmitted from master

https://redis.io/topics/distlock#correct-implementation-with-a-single-instance - this tells how a lock is framed and set

https://redis.io/topics/distlock#the-redlock-algorithm - This I think tells

there are N lock managers and N Redis masters. I assume all the N redis masters have same data i.e they are like replicas. Hence Fault tolerant because it needs N/2 + 1 locks. So if more than N/2 nodes goes down the locks of rest are released and if 1 to N/2 nodes go down we are okay.

Comment From: hariraogotit

You are reading string type data? Your 5 instances will read the same key at the same time? If so, then you need distributed locks, because your 5 microservices may read the key at the same time, and execute the workflow 5 times, and only one instance succeeds in deleting the key.

If you have 5 instances that may read a key at the same time to processing may not make sense, you should use stream to store the ids you need to process, then 5 instances use xread to get a key from the stream, then after carries out a workflow of activities, then use xdel to remove the entry from the stream. This way you can avoid using distributed locks.

To be specific I am using Redis as DB i.e Redis will have lots of records in it When my 5 microservice instance boots up. Each instance will read the records one by one from DB, process and remove it from Redis. I do not want microservices instances to end up in race conditions. So I will need a distributed lock here but will also explore the streams you suggested. Since I am building it from scratch I can move from Redis DB. Kindly can you share the documentation link for the streams that you suggested which will let me avoid using the distributed lock?

Comment From: sundb

@hariraogotit 1. Do you really need redlock? From what you want to achieve, if you really need to lock, I think SETNX + lua is perfectly adequate, redlock is reliable but also more expensive (5 masters).

1. Use stream. 1) Insert key to string and stream. SET id 123 XADD <stream-name> * id 123

2) Your 5 microservice instances listening to the stream to get id XREADGROUP GROUP <group-name> <consumer-name> COUNT <count> STREAMS <stream-name>

3) Use the id get from the stream to execute the workflow.
4) Use `XACK` to mark the message as processed.
5) Use `DEL` to delete the id from redis string dataset.

Comment From: hariraogotit

@hariraogotit

1. Do you really need redlock? From what you want to achieve, if you really need to lock, I think SETNX + lua is perfectly adequate, redlock is reliable but also more expensive (5 masters).

2. Use stream.

3. Insert key to string and stream.

SET id 123 XADD <stream-name> * id 123

1. Your 5 microservice instances listening to the stream to get id

XREADGROUP GROUP <group-name> <consumer-name> COUNT <count> STREAMS <stream-name>

1. Use the id get from the stream to execute the workflow.
2. Use XACK to mark the message as processed.
3. Use DEL to delete the id from redis string dataset.

Thank you. I will also read about redis cluster, client libs ( as I am using spring boot). Sounds like reddison supports streams but will read about it.

Comment From: hariraogotit

@hariraogotit I'm not sure where you got confused about redlock. Do you not understand why 5? Or do you not understand why the lock keys of the 5 redis instances are locked sequentially?

https://redis.io/topics/distlock#safety-and-liveness-guarantees - tells the objective

https://redis.io/topics/distlock#why-failover-based-implementations-are-not-enough - explains safty violation as the replica does not have the lock as it was not transmitted from master

https://redis.io/topics/distlock#correct-implementation-with-a-single-instance - this tells how a lock is framed and set

https://redis.io/topics/distlock#the-redlock-algorithm - This I think tells

there are N lock managers and N Redis masters. I assume all the N redis masters have same data i.e they are like replicas. Hence Fault tolerant because it needs N/2 + 1 locks. So if more than N/2 nodes goes down the locks of rest are released and if 1 to N/2 nodes go down we are okay.

@sundb - Please can you confirm if my understanding of the last point above is correct i.e about N lock managers / N redis masters, especially - N redis masters have same data i.e they are like replicas?

Comment From: sundb

@hariraogotit They are not like replicas, the N redis masters are completely independent, it is possible that you only set locks to 3 redis instances that succeed, 2 fail, but the locks succeed. At this point, their data is inconsistent.

Comment From: hariraogotit

@hariraogotit They are not like replicas, the N redis masters are completely independent, it is possible that you only set locks to 3 redis instances that succeed, 2 fail, but the locks succeed. At this point, their data is inconsistent.

Right this is where it is confusing. Say my microservice is looking for a record in Redis that is in 1st Redis Master, this means it has to lock 3 Redis Master instances. Like you said before it is not locking on the record in Redis but rather locking the Redis instance itself. So does this mean say I have 5 microservice instances and one of them acquired a lock in 3 Redis Master instances, it is happy and processing but rest of the other 4 Microservices although is looking for a different data in Redis needs to wait because 3 Redis Master instances are already locked by 1st Microservice?

This is where I think the unique key and TTL comes in to picture (https://redis.io/topics/distlock#correct-implementation-with-a-single-instance)

Now assume 1st Microservice is looking for a record 123 which is in Redis master 1. So 1st Microservice puts a key lock assume 1st-microservice-lock-01 on 3 Redis masters. 2nd Microservice comes along and looks for same record 123 which is in Redis master 1. So can 2nd Microservice put a lock 2nd-Microservice-lock-02 on 3 Redis Masters since the lock key strings are different?

Comment From: sundb

@hariraogotit Both the first and second microservice should lock the lock-id-123(need unique) key.

Comment From: hariraogotit

@hariraogotit Both the first and second microservice should lock the lock-id-123(need unique) key.

Okay in that case say

1. Microservice 1 locks Redis Master 1 (with the lock key 1st-microservice-lock-01 ) as it is going to work on id 123
2. microservice 2 locks Redis Master 1 (with the lock key 2nd-microservice-lock-02) as it wants to work on id 123
3. Now Microservice 1 processes id 123 and deletes from Redis Master 1 after it completes the process
4. While Microservice 2 is still holding the lock but with its lock string is 2nd-microservice-lock-02
5. How can this scenario be avoided?

Here id 123 is a record in Redis just to make sure we are in the same page.

Comment From: sundb

@hariraogotit I dont't understand why you want both microservice 1 and 2 to lock key 123, but then you are using two different lock key? If you want to lock the same piece of code or the same piece of data, you must be locking the same lock key.

Comment From: hariraogotit

@hariraogotit I dont't understand why you want both microservice 1 and 2 to lock key 123, but then you are using two different lock key? If you want to lock the same piece of code or the same piece of data, you must be locking the same lock key.

Exactly the lock string (or lock key should be same) but

Check out the below from https://redis.io/topics/distlock#correct-implementation-with-a-single-instance

This is important in order to avoid removing a lock that was created by another client. For example a client may acquire the lock, get blocked in some operation for longer than the lock validity time (the time at which the key will expire), and later remove the lock, that was already acquired by some other client. Using just DEL is not safe as a client may remove the lock of another client. With the above script instead every lock is “signed” with a random string, so the lock will be removed only if it is still the one that was set by the client trying to remove it.

The above says the lock string (lock key) should be unique across clients. So my understanding is the clients (microservice instances say) lock with different strings and remove the ones that they put. This is why I asked the question -https://github.com/redis/redis/issues/9651#issuecomment-949608643

Comment From: sundb

@hariraogotit First, you need to understand why the lua code if redis.call("get",KEYS[1]) == ARGV[1] then, when the client finds out that the value of the lock is not a random string generated by itself, it can't delete this string. When all clients lock the same key, they all generate a unique string and set that unique string to the lock key.

You may have overlooked my_random_value, which is the random string that each of your microservices generates and holds, and which you need to use when you release the lock.

Comment From: hariraogotit

@hariraogotit First, you need to understand why the lua code if redis.call("get",KEYS[1]) == ARGV[1] then, when the client finds out that the value of the lock is not a random string generated by itself, it can't delete this string. When all clients lock the same key, they all generate a unique string and set that unique string to the lock key.

You may have overlooked my_random_value, which is the random string that each of your microservices generates and holds, and which you need to use when you release the lock.

Thank you very much for all your replies @sundb . I will explore the streams that you suggested earlier as it does not need locks. The more I read about locks in Redis the more I think is not suitable for my use case eg I read https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html

Nevertheless I will read about streams and see how I can use it via the spring boot. You have been so kind :) I believe I might come back and bother you again :)

Comment From: sundb

@hariraogotit Welcome back.

Comment From: hariraogotit

@hariraogotit

1. Do you really need redlock? From what you want to achieve, if you really need to lock, I think SETNX + lua is perfectly adequate, redlock is reliable but also more expensive (5 masters).

2. Use stream.

3. Insert key to string and stream.

SET id 123 XADD <stream-name> * id 123

1. Your 5 microservice instances listening to the stream to get id

XREADGROUP GROUP <group-name> <consumer-name> COUNT <count> STREAMS <stream-name>

1. Use the id get from the stream to execute the workflow.
2. Use XACK to mark the message as processed.
3. Use DEL to delete the id from redis string dataset.

@sundb I am back again :) I read about Redis Streams and I tried out. For my use case it is slightly convoluted (I can explain if required). So I went ahead with the option #-1 (i.e use SETNX + lua) but there is a small catch in that 1. Imagine Microservice instance 1 locks a record 123 with the lock-key 123abf 2. also Microservice instance 2 locks a record 456 with the lock-key 123abc 3. So #-1 and #-2 are holding different locks on different records, so all good. 4. Now imagine Microservice instance 1 crashes 5. A new Microservice instance 3 comes up. Microservice instance 3 now will not be aware of unprocessed record 123 with the lock-key 123abf 6. Eventually record 123 with the lock-key 123abf will never be processed. In redis streams there is XPENDING and XCLAIM but nothing in here.

Kindly let me know how can the unprocessed records can be picked up by the new instances?

Comment From: sundb

@hariraogotit SETNX have timeout parameter. If Microservice 1 crashes, the new instance can lock the key again after timeout.

Comment From: hariraogotit

@hariraogotit SETNX have timeout parameter. If Microservice 1 crashes, the new instance can lock the key again after timeout.

@sundb I think I now understand this.

In the image there I created a lock for microservice 1 which expired that means the value of the lock will also go away. I want the value to stay there for the new Microservice service to pick the value and continue processing from where Microservice 1 left. May be Redis is not suitable for my use case. I will have to think about the RDBMS t

To give you more context on my use case.

1. Microservice 1 reads from Solr, gets hold of page 1, inserts the paging details and no of docs processed in Redis, process the records in the page 1 then goes to Solr to get next page using the paging details from Redis then updates the Redis with latest paging details and the no of documents processed.
2. So in any of the steps if microservice 1 fails the microservice 2 should be able to pick it up from where microservice 1 left.
3. I think its better to stick to RDBMS for my use case.

Comment From: hariraogotit

@sundb For my use case I managed to use my ip address as a lock key that is part of Set collection. Then I have all my actual records in Hashes collection. I wrote Lua script to manage the locks and the records in Hashes as it gives atomicity. Overall I have met the use case with Redis and Lua. Thanks a lot for your responses that helped me a lot. Much appreciated! I am closing this ticket now.

Comment From: GO-DIE

@hariraogotit I'm not sure where you got confused about redlock. Do you not understand why 5? Or do you not understand why the lock keys of the 5 redis instances are locked sequentially?

@sundb Why does Redlock need to try to acquire the lock in all the N instances sequentially? I wonder if you could help me with this problem，thanks.

Comment From: sundb

@GO-DIE Because the client needs to get more than half of the node locks in order for the lock to succeed, for example, if some of the nodes are not available, but you get more than half of the locks, then you can also get the lock.

Comment From: GO-DIE

@sundb Sorry, my description may not be accurate. In fact, i don't understand why it needs to try to acquire the lock in all the N instances sequentially. What can go wrong if we try to acquire the lock in all the N instances in parallel?

Comment From: sundb

@GO-DIE It is theoretically possible to parallelize, it will be faster, but it also has another waste, if we fail to get half of the locks, we already failed to get the locks, so we don't need to get the other locks.

Comment From: GO-DIE

@GO-DIE It is theoretically possible to parallelize, it will be faster, but it also has another waste, if we fail to get half of the locks, we already failed to get the locks, so we don't need to get the other locks.

Thank you very much for your answer

Comment From: Atom1010

@sundb I am using redis stream to broadcast the updation as a message of Collection1 to other services in my microservice architecture. Problem Statement: Avoid duplicacy of message processing in multiple instances of multiple services. I don't want all consumer's of multiple instace to consume the same message, i want only one consumer of one instance to process the message.

My Solution Approch: I am exploring distributed lock in redis to resolve this, But I am not sure whether it will resolve my problem and will not lead to any performance issue.

Comment From: sundb

@Atom1010 Why not let these cosumers in a same group, which prevents the same message from being consumed by mutiple consumers.

Comment From: Atom1010

@sundb no I am talking about scale up in kubernetics which will create a new instance for a service and that instance will have a same consumer group that means it will also consume the same message, Why I am using redis stream ? To avoid using grpc call across services, when I update the one collection of one service, I need to update the some context of it in other collections of other service, I needed async mechanism.

Comment From: Atom1010

@sundb Am I wrong someware

Comment From: sundb

@Atom1010 sorry that i've been busy in other place. I'm not sure i understand the meaning of a same consumer group that means it will also consume the same message. When the message is sent to a consumer group, it's delivered to one of the consumers. If the message is successfuly processed and acked, no other consumer will receive the same message.