Redis Redis Cluster: Automatic re-balancing

The cluster should be able to automatically re-balance itself, instead of being triggered manually.

This could be on a configured schedule (e.g. every 7 days) or perhaps on tolerances, such as a node is at 90% max-memory usage.

Comment From: antirez

This feature is planned. Configured schedule looks fragile at a first glance compared to tolerances. This feature is due for v2 of the cluster. The general idea was: 1. Nodes gossip about free memory, data, % of fullness. 2. When there is enough variation, with a configurable threshold, nodes start to migrate hash slots to compensate. 3. This option should be configurable, so that the admin can switch it on/off.

This is a lot more complex than it looks like, because of the big keys problem, so that hash slots don't actually represent an homogenous slice of the memory used. More over certain keys may be subject to most of the traffic of the entire cluster, for example counters to generate IDs and other similar special keys.

So the cluster in order to re-balance itself, needs to have more clue about what is inside the key space it has currently. So: wanted, hard to get right, v2 goal.

Comment From: cusspvz

Comment From: shaharmor

+100

Comment From: antirez

Note that now we have rebalancing in redis-trib. Still it's not automatic, the redis-trib command must be invoked, but does all the resharding needed automatically. It is possible to assign weights to different nodes. I'm documenting it right now. The feature is already available in the latest stable version of Redis, however improvements are in line for the next patchlevel.

Comment From: greenrd

"I'm documenting it right now."

Where is this documentation? The Redis cluster tutorial still says this feature is planned.

Comment From: estebancrw

Any update on this front?

Comment From: antirez

Hello, this is part of the Redis 4.2 roadmap. Note that meanwhile manual rebalancing was added to redis-trib.

Comment From: ping2balaji

Hi, can you pls clarify if the automatic rebalancing feature available in latest 5.0 release?