After errors during cluster scaling one of cluster's nodes became inconsistent. Some keys of one particular slot have remained on the old shard.

We have restored all the data, but keys are still on the wrong shard. SCAN returns them, but any other command returns MOVED error:

redis-cli -h 10.222.122.7
10.222.122.7:6379> scan 0 match pwe_userid:9fd386a4-94f9-4732-b2a7-ad2442d4f252 count 100000
1) "3505504"
2) 1) "pwe_userid:9fd386a4-94f9-4732-b2a7-ad2442d4f252"
10.222.122.7:6379> cluster keyslot pwe_userid:9fd386a4-94f9-4732-b2a7-ad2442d4f252
(integer) 6456
10.222.122.7:6379> cluster slots
...
4) 1) (integer) 8192
   2) (integer) 10922
   3) 1) "10.222.122.7"
      2) (integer) 6379
      3) "c4a81e6707c4bd0f3534df81803f0307ca2a18d6"
      4) (empty array)
   4) 1) "10.222.145.28"
      2) (integer) 6379
      3) "d8089ddc9970fcc9d16d85fb419fab9e2715e0c3"
      4) (empty array)
...
10.222.122.7:6379> del pwe_userid:9fd386a4-94f9-4732-b2a7-ad2442d4f252
(error) MOVED 6456 10.222.60.143:6379

Is there any way to just delete these keys without heavy migration procedures and without removing the node from the cluster?

Comment From: sundb

@pavel-krush can you give the reproduce steps?

Comment From: pavel-krush

@sundb i don't think i could reproduce "errors during cluster scaling". I have pressed a lot of buttons and I am not sure at what moment cluster became inconsistent.

What I know is that we had a lot of small keys and they were migrated successfully. But also we had one very large(several gigs) list. Redis could not to migrate this key until we set repl-backlog-size to "1g".

Sorry, have no other info.

The cluster is fine now, but one node still contain wrong keys. I just wanted to know how can we delete these keys