Having executed SCAN via redis-cli to a master node A of a redis cluster, a list of keys returned and I tried to execute GET for each one of these. There are quite a few GET commands that returned that the key has been moved: (error) MOVED to master node B and when I tried to execute GET command in the master node B, indeed the value was returned.

Why SCAN command return keys that do not belong to the master node A? Is it expected? In a master node that has about 80000 keys totally, but 40000 of them return "(error) MOVED" when we are trying to access them, it seems to me to be a problem for us. This issue has been observed only in one master node of the cluster. Is there a way to overcome that situation and clear keys that do not actually belong to master node A? We are using redis 3.2.8 , along with custom implementation of redis-cluster. The cluster is stable, while we are experiencing the above mentioned issue.

Comment From: nanmark

In the same master node A, we faced a probably related issue. When we tried to execute the --bigkeys command in master node A, it failed we the following error: 

Invalid reply type (6) for TYPE on key 'xxx:yyyy'

 # Scanning the entire keyspace to find biggest keys as well as
 # average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
 # per 100 SCAN commands (not usually needed).

When we tried to get type of key on master node A, indeed a moved error is reported: 

<ip_of_master_node_A>:6379>type xxx:yyyy
(error) MOVED 720 <ip_of_master_node_B>:6379

But when we tried get type of the key on master node B, type was returned:

<ip_of_master_node_A>:6379>type xxx:yyyy
string

According to the code for bigkeys in redis-cli.c, a scan command is performed and then the following are called: 

/* Retreive types and then sizes */
        getKeyTypes(keys, types);
        getKeySizes(keys, types, sizes);

when getKeyTypes is called for a set of keys that has been returned by scan on master node A, unfortunately scan has returned some keys that belong to master node B as I have reported in my previous comment.

Is it ok --bigkeys to fail due to this reason, since we don't get any report about the size of keys and we just get an error for a single key?

Comment From: zryfish

does anyone has solution to this issue ? redis-cli --bigkeys always fail due to moved response returned by server

TYPE returned an error: MOVED 0 172.168.4.114:6379

Comment From: lileibuaa

does anyone has solution to this issue ? redis-cli --bigkeys always fail due to moved response returned by server

TYPE returned an error: MOVED 0 172.168.4.114:6379

I have the same problem. The cause seems to be there are some illegal encoding format keys in target node.

Comment From: yongwangalex

I found the same problem in redis 5.0.5. I got a rdb file with save command in a node in a cluster of redis 3.0.3. Then I restore the rdb file to a cluster of redis 5.0.5 with redis-migrate-tool. I ran the command in redis-5.0.5 as follows

redis-cli -p 6479 --bigkeys -i 0.1

the output is :

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).
[00.00%] Biggest string found so far 'bls:cam:vEEkT0oZy8M' with 102 bytes
[00.00%] Biggest string found so far 'bls:cam:uS5Up02LgNK' with 117 bytes
[00.10%] Biggest string found so far 'bls:cam:xRpSv13IV2M' with 120 bytes
[00.23%] Biggest string found so far 'bls:cam:sAmmn0wYNAA$26e�t_`ata=djR1UXB5cU1xR2gydllrRUJCZVdySE00dkozdkRlX3EBcXYBeHdxeFp3bWZ4MGN2Y3dtY1d4SjBoMFdoRkVXeHhKRWMBATB4Lm13LmhjMC5xZXgBcHpzcgFqRXR0OXY1SUhFRQFqRWpud3YwZjZCdQFvbTkxaHF2SGZ2YgFxcXhxMHZjY3cBbWV2cnF2dgF3dgF2AWhoZgFxdnZ2ZXdxAQFlbXZ2cQEBbGRsMSBPbUIzeWogWgFFOXNwMWRzAWMucS5xAXEBaAFsZGwxAXFxbS54eHcwcS0wYy5jeGgwbWMBAQEBAW56OGlleDRxZjQ0c2h6Y2NxMmVmd3pld2N6MHp3aDJ2Y2hpc21pY21jaGIBAWpFdHR0dklSWTBWAQE' with 490 bytes
[00.46%] Biggest string found so far 'bls:cam:t7POq0�CuQwe26gxt_data=djR1dFUxcVpJNzk5MHhueUtNZXVwcXpqdjJlUGFvX3EBcXYBAUZtRmZ3dkZXWmh3TTB4MEpldmUwbTBodlcwcVpteGhXAWN3LnFtLmMwLnd2AXB6c3IBb2ZCVmF2UUp5VTUBb2ZCVnB2Vk5QencBb2ZWOGN2c0VHTmQBcXF3aHdoMGVjATBodnJjdgFxY3YBdgFodm0BcXZ2djB2YwEBZW12dmgBAWRCazE5MmYscQFkVkEBcXYuMC4wAXEBcQFFODgzMgF2LnYtdi52AQEBAQFuejhzeml3dm

lodzB2Y2h4cXZ2dmZ6bTRpd3diZmJzemUwMGIwY2lxYnplAUpGZm1jaGV4LWZoaEUtZWZxRi14V01xLUZtbUVlaHhGeDBXdwFvZkJWYXY0U1pUdwEB' with 514 bytes
>TYPE returned an error: MOVED 6801 127.0.0.1:6480

keyname is something frustrating with unusual symbol, however, I ignored it. I thought after migration, the key will be assigned to right slot. and the hashdict should not contain the key not assigned to it.

I used redis-migrate-tool with the conf:

[source] type: rdb file servers : - /home/XXX/dump-6382.rdb

[target] type: redis cluster servers: -127.0.0.1:6479

Comment From: czhang223

I've got the same problem. There're nodes A, B, C. I scan key pattern from B and it returns some keys. Then I get the key, it says moved to A. I get the key in A, it says null and I can not delete that key. I still can scan that key (which is belong to A) from B, but I can not scan that key from A. What the xxx

Comment From: sayap

We also experience the same issue. After deleting a 14GB hash key from master X, we found the same hash key on master Y, with the size of 21GB. We are kinda screwed now as there doesn't seem to be a good way to delete this 21GB hash key, since the slot is owned by master X.

We got into this inconsistency state, probably due to timeout from MIGRATE followed by buggy redis-trib.rb fix.

Comment From: antirez

@sayap you can just open the slot (check the documentation for more info), delete the key, and re-close the slot in that instance.

Comment From: sayap

@antirez Thanks for the tip. Do you mean something like:

  1. (open the slot) Run CLUSTER SETSLOT <slot> IMPORTING <source-node-id> on the node with keys that don't belong to the node anymore.
  2. Use ASKING plus the necessary commands to get rid of the keys.
  3. (re-close the slot) Run CLUSTER SETSLOT <slot> STABLE on the node.

Comment From: VeryBadRobot

we meet this problem too, the error is type command error。 e47102a105c15273c11df56a5b9ab743_ 29f017eee1332261a31cb9d72523830b_

the --bigkeys comand use scan then type command and caculate each key，the problem in our situation is scan has foud some keys not in this node. and the key doesn't exists, then type command report an error. our problem is some key in the node is wrong, because of data migrate.

Comment From: 553589912

https://github.com/redis/redis/issues/4810 我也遇到的相同的问题

Comment From: DriverX

We have a same problem. Redis 6.2.11. SCAN discover keys that no longer exists. When try to GET or DEL it, MOVED is returned.

Comment From: DriverX

I found intresting command in redis-cli cluster management - --cluster-search-multiple-owners. Introduced in https://github.com/redis/redis/pull/5681