The problem/use-case that the feature addresses
I have a large amount of data (> 10 TB, might scale up further) stored in a non-Redis datastore that I need to migrate over to a Redis cluster. In my particular use case, I need to be able to accomplish this quickly because I'll be performing it reasonably often, but I believe that an established bulk-import strategy would benefit others as well.
Description of the feature
I haven't done any benchmarking on this but wondering if there's a bulk-loading strategy that is common (in which case the request would be to document it and have it be supported for later versions of Redis as well), or if there's been any discussion around the fastest way to do this.
Alternatives you've considered
- Using a Redis client and streaming it in (this might be the best option, but curious if there's faster)
- Use redis-cli --pipe (doesn't work with cluster mode)
- Leverage AOF by manually generating my own AOF file(s), sticking it on a volume and mounting it onto the cluster (would this even work?)
Comment From: zuiderkwast
Hello.
I believe streaming using a cluster-capable client is the best option. Have you checked out the tools listed on this page: https://redis.io/resources/tools/? For example RIOT seems like it should be capable of doing this.
Comment From: retech9691
I haven’t seen that before, thanks for sharing. Since streaming is the best option, Riot looks like the perfect tool for that.