when I migrate a list type key containing 30 million fields to another Redis, prompting an error:
127.0.0.1:6379> MIGRATE 127.0.0.1 6380 testlist 0 600000 -IOERR error or timeout is written to the target instance
Through Wireshark's capture, the actual error returned is:
-ERR Protocol error: invalid bulk length.
The reason looks like that there are too many fields in the testlist. After serialization, a single sds exceeds 512MB, which does not comply with the RESP protocol. Therefore, it is better to judge whether the serialized value is in compliance with the RESP protocol.
Comment From: trevor211
Store such a large value in redis is not a good idea.
Anyway, we can still migrate it by increasing the config proto-max-bulk-len.
Note that the destination master node would block, and a auto failover may happen,
so you should increase cluster-node-timeout at the same time.
Comment From: oranagra
I would like to add that redis is currently handling such large values in a bad way in several aspects. another aspect is that both during serialization and deserialization the serialized payload is kept in memory in parallel to the actual value (list with 30 million fields in this case). so not only that it blocks both the source and the target, but also blots their memory, risking an OOM kill.
This is a known limitation that we plan to address at some point in the future.
meanwhile i'd advise to avoid such big values, and if you have no choice, you'll need to increase proto-max-bulk-len, client-query-buffer-limit, and possibly cluster-node-timeout.
Closing this issue for now. Thanks.