Its more of a feature request than a issue.
My deployment model for Redis is in cloud where the Redis servers are running in different subnet lets say A and the redis clients are running in different subnet lets say B. Also there are some route rule restriction across the subnet and the only point of entry to the subnet A is via a preconfigured NAT Gateway(with a floating IP) where we provide the list of ports to be accepted. In this use case, we need a configuration where the NAT GW IP is configured and the Redis server will use this IP while providing response for Redis cluster commands like CLUSTER NODES, CLUSTER SLOTS. According to the current implementation, clients will receive the subnet A ip addresses in the cluster commands which aren't reachable from subnet B but if I configure the announce ip with NAT GW IP then the Redis cluster won't be able to communicate with each other.
How are we handling this currently? As a workaround, we are maintaining a mapping table on the client side and after fetching the response of cluster commands, we are replacing the subnet A ip with the NAT GW IP and then establishing connection to Redis server.
Since Redis is powerful and is used in many use cases, still there are some challenges while deploying it over cloud.
Comment From: WiFeng
You can refer to the default redis.conf , as follow:
- master/slave
# However when port forwarding or Network Address Translation (NAT) is
# used, the replica may be actually reachable via different IP and port
# pairs. The following two options can be used by a replica in order to
# report to its master a specific set of IP and port, so that both INFO
# and ROLE will report those values.
#
# There is no need to use both the options if you need to override just
# the port or the IP address.
#
# replica-announce-ip 5.5.5.5
# replica-announce-port 1234
- cluster
########################## CLUSTER DOCKER/NAT support ########################
# In certain deployments, Redis Cluster nodes address discovery fails, because
# addresses are NAT-ted or because ports are forwarded (the typical case is
# Docker and other containers).
#
# In order to make Redis Cluster working in such environments, a static
# configuration where each node knows its public address is needed. The
# following two options are used for this scope, and are:
#
# * cluster-announce-ip
# * cluster-announce-port
# * cluster-announce-bus-port
#
# Each instruct the node about its address, client port, and cluster message
# bus port. The information is then published in the header of the bus packets
# so that other nodes will be able to correctly map the address of the node
# publishing the information.
#
# If the above options are not used, the normal Redis Cluster auto-detection
# will be used instead.
#
# Note that when remapped, the bus port may not be at the fixed offset of
# clients port + 10000, so you can specify any port and bus-port depending
# on how they get remapped. If the bus-port is not set, a fixed offset of
# 10000 will be used as usually.
#
# Example:
#
# cluster-announce-ip 10.1.1.5
# cluster-announce-port 6379
# cluster-announce-bus-port 6380
Hope it works for you.
Comment From: hasan4791
@WiFeng Thanks, but my use case is different. The first config is for Redis sentinel Master/Slave and next one is for Cluster. I'm interested in Cluster config but cluster-announce configs won't suffice my need. We can use announce feature only if that IP is reachable from the Network where the Redis is running.
Comment From: pfreixes
I was having a similar problem When I was trying to have the Redis cluster working within Docker but still having the client protocol exposed in the host network.
My idea was to implement a new option called cluster-announce-bus-ip which would allow you to provide a different IP address for the bus protocol. So client communication would happen with the ip configured with the cluster-announce-ip option - I guess that this the information that is returned to the client for making redirects to another host, while for bus communication we would be using the cluster-announce-bus-ip value.
Of course this should be backward compatible, so this will work on this way if and only if the cluster-announce-bus-ip would be provided.
WDYT? is there any redflag for implementing this? I could try to provide a PR if there is some quorum that this could eventually be merged.
Comment From: madolson
@pfreixes @hasan4791 I think announce hostnames might solve this problem? There is some ongoing effort (that I'm supposed to be doing right now honestly) to have a cluster-announce-hostname. The NAT gateway IP address could be placed behind a DNS record, which could do the resolution to the floating IP. Thoughts?
Comment From: madolson
Maybe related: https://github.com/redis/redis/issues/2186
Comment From: pfreixes
In the use case of docker announcing IPs would be good enought, for keeping the bus comunnication witin the docker network but expecting to have client comunicatiion at host level (not sure if there is any red flag regarding this)
Having support for hostname would help on not having to known the IPs in advance, which we are forced to know right now.
Comment From: madolson
@pfreixes You could use the new hostname functionality to announce the IP address to external clients, keep the clusterbus within whatever network the docker containers are in, I don't see any red flags with the setup. Seems like a use case worth documenting on the other SIM and perhaps changing the name of the feature. (I suppose we might need to handle a different port as well)
Comment From: kevinbosak
My use case is the same as what @pfreixes describes. Using a hostname is still not ideal as it would require developers who are using the docker setup to edit their hosts file and point the hostname being used to their loopback IP or local IP. Adding a feature to Redis so that it can report one IP or host to the client and another to the cluster nodes would be a much better solution as no extra setup is needed outside of the docker files.
Comment From: thxrben
For me its even worse: Since I use IPSec to connect 2 sides, they have the same subnet (192.168.1.x). I have to use IPSec with BiNat to let both networks talk to each other. I cant access Redis either, since the MOVED "notification" sends an IP-Address of the 192.168.1.X network, and not one from the BiNatted Networks (192.168.100.X or 192.168.200.X). This makes things complicated. I cannot imagine how Large-Scale REDIS-Databases are put together for redundancy and more, especially with different locations.
Comment From: soloestoy
Seems we need decouple the announce ip:port from the real ip:port. Now if users set cluster-announce-ip and cluster-announce-port, they would replace the real ip:port getting from socket/peer name.
A solution is: 1. cluster nodes communicate with each other by using the real ip:port in server side 2. commands like CLUSTER NODES|SLOTS|SHARDS and -MOVED use announce ip:port in client side
@redis/core-team WDYT
The perfect way I think is, return different results based on different sources. For example, return the announce topology for requests coming from "announce ip:port", and return the real topology for requests coming from "real ip:port". But we cannot get enough info to determine that, since the NAT problem...
Comment From: madolson
Yeah, I think what is all stated here makes sense. We should add some new configs.
Comment From: brutalmt
Bumping this thread - do we have any updates re. this issue please?
Comment From: thxrben
There are some rewrite-classes so you can re-write the source IP when receiving an answer, but I would love to hear more from this issue aswell
Comment From: madolson
Let me add this to Redis 8 project, I think this is something we should look into picking up.