We're running into problems when trying to run redis cluster from docker on different hosts. Dockerized redis node advertises its internal docker address, which is unreachable from other nodes in the cluster. I'm aware of various hacks and workarounds, but I'm proposing that redis cluster supports announce-ip and announce-port configuration options, similarly to redis sentinel.
Comment From: kremso
I did try the recommended workaround of specifying an explicit bind, but it fails too. It seems that it won't work without setting IP_TRANSPARENT option on the socket, which redis currently does not.
Comment From: harley84
Related to #2335
Comment From: antirez
Agreed, will write here a design spec in the next days. p.s. in the meanwhile you may want to prevent Docker from mapping ports, there is a specific feature for 1:1 mapping.
Comment From: KekSfabrik
I'm having the same trouble - see #2621 copy/pasta from there: [...] Master instances know the slaves running on the same machine by local (172.*) IPs and vice versa. This may possibly lead to getting the MOVED response to an IP that is not reachable by the client:
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6379 cluster nodes
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352847839 5 connected
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352848039 6 connected
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352847638 2 connected 5461-10922
e1fc77808551cccf5efe6bcbb5816b624bff9633 192.168.1.245:6379 myself,master - 0 0 1 connected 0-5460
721e4986d1cc2dbf87ac11054242233ac3d67ede 172.17.42.1:6380 slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434352847139 4 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352847839 3 connected 10923-16383
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6380 cluster nodes
e1fc77808551cccf5efe6bcbb5816b624bff9633 172.17.42.1:6379 master - 0 1434352872207 1 connected 0-5460
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352872509 5 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352872308 3 connected 10923-16383
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352872108 2 connected 5461-10922
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352872509 6 connected
721e4986d1cc2dbf87ac11054242233ac3d67ede 192.168.1.245:6380 myself,slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434101988241 4 connected
however i've noticed they all act the same -- the ones on the 192.168.1.246 IP behave exactly the same (for their own IP). Relevant bits as seen by 192.168.1.246:6379:
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 myself,master - 0 1434101985463 2 connected 5461-10922
8bd0598380fc05aa374ef22a74da1a691963cfbb 172.17.42.1:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434375798687 5 connected
So they each have a (machine local) 172.17.* net that is not available from outside the machine
Comment From: drshade
+1 please
Comment From: Ducatel
If have the same problem. It isn't possible to create a redis cluster on different servers with docker. Redis send the container IP instead of the host IP. So for this Redis should add the support of announce-ip like the sentinel.
+1 please
Comment From: bjackson
+1 This would be amazing.
Comment From: antirez
Prioritized in order to be shipped with the next release (3.2). Thanks for the feedbacks.
Comment From: ychrysler
:+1: i there a proposed work-around available for this issue?
Comment From: ZhuoRoger
+1
Comment From: mistypig
+1
Comment From: devaos
+1
Comment From: turbospaces
+1
Comment From: sst1xx
+1
Comment From: fab-io
As a workaround, if possible in your env, run the containers with the option --net=host (see https://docs.docker.com/articles/networking/#container-networking)
Comment From: rpannell
+1
Comment From: antirez
Scheduled for Redis 3.2 that will go in RC in a matter of weeks (~4 hopefully).
Comment From: antirez
p.s. this is going to be backported into 3.0.x anyway...
Comment From: himulawang
+1
Comment From: fatduo
Hi @antirez @harley84 Do you have some work-around solutions? I think you can get the hostname instead of ip address. In docker container, I can pass hostname env to the container.
Comment From: MrMMorris
@fatduo the workaround is using --net=host as mentioned here: https://github.com/antirez/redis/issues/2527#issuecomment-144401685
Comment From: srinikalyanaraman
Hi @antirez @harley84 I have put in a fix in a fork and opened a PR (as per the CONTRIBUTING document). I have tested them in kubernetes on openstack. Please review. Thanks!!
Comment From: antirez
I'm going to release RC1 without this feature, to add it in the course of the next RC so that we'll have it. However there is some design effort to do about it. It's not a matter of writing an implementation, but to understand how to implement it. For instance, the implementation provided by @srinikalyanaraman breaks binary compatibility with the Redis Cluster bus since it modifies the header structure without upgrading the version, or handling the older version format. Also the final implementation will allow to remap both the IP and port. So thanks for providing a PR, but the issue was marked as design needed because there were a few problems to fix. This is why I cannot merge the PR.
So, if there is to do a design effort, let's do the design effort:
Design proposal
In order to make the new implementation binary compatible with the past, it is possible, in theory, to use the unused field in the message structure. It's 32 bytes so enough to put inside the address (including the IPv6 address) in binary form. It's not a great solution however... IPs travel in cleartext and normalized in the rest of the protocol, and we have already the conversion functions in place.
If there is to break binary compatibility, we'll have to upgrade the protocol version in the header, so that older cluster implementations will discard the packets. Instead in 3.2 we'll have to handle the older version of the protocol as well, which is not great, but is needed.
In short:
1. We add a new field in the header, containing the announced IP.
2. We don't add a new port field, we already announce it currently. We'll just overwrite this field if a different port is announced as well.
3. We upgrade the version of the protocol CLUSTER_PROTO_VER from 0 to 1.
4. We make sure that when IPs are not remapped, the new field containing the announced IP is all zeroed. It means: auto detect it, dear receiver. Like 3.0 is doing by default.
5. The new field will be called just myip. Without references to the fact is an announced one. Simply if it's zeroed, we ask for auto detection to the receiver.
Now in order to handle the older version of the protocol in 3.2, the simplest thing to do is the following:
1. We read the packet as usually.
2. In clusterProcessPacket() what we do is to check the version of the protocol. If it's 0, we know we read an older protocol with different offsets because of the missing field. But fortunately we can just memmove to fix it all, and memset the new "hole" left with zeroes. And this will make the older protocol compatible with the newer protocol.
3. The unused field we already have, of 32 bytes, will be left as it is. We'll need it soon or later, as this issue shows. Unfortunately this time it was short, but 32 bytes are a lot, normally.
Exposed configuration options:
cluster-announce-ip
cluster-announce-port
Both can be provided independently.
Implementation
Given that it's very little, but very sensible code, I'll implement it myself so PRs will not be accepted in this context since to proof-read them is more work than writing it from scratch with the above proposal in mind. Instead what would be extremely valuable is ideas and analysis of the proposal in order to identify if it's good enough or not.
Documentation
The features must be documented in: 1. The cluster tutorial. 2. The example redis.conf file.
Comment From: allanwax
@antirez any thought to having cluster-announce accept multiple addresses as failovers in case one of the 'channels' goes bad?
Comment From: allanwax
SEE also https://github.com/antirez/redis/issues/2109
Comment From: karthimohan
@antirez Could you please let us know when this will feature be available?
Comment From: antirez
Hello folks, I'm working at this stuff right now and I'm 90% done, and I've a few bad news.
1. I don't think there is a simple way to make this backward compatible at binary level. I tried but it adds a lot of complexity, so basically this feature will not back ported to 3.0, will be 3.2 only, and when upgrading a cluster there will be to restart all the nodes with 3.2, since 3.0 <-> 3.2 chat will be impossible.
2. I'm also implementing cluster-announce-port, since it's the most useful thing with Docker, for instance. However the problem is that we have a base port, and a cluster bus port, and I don't believe they are guaranteed to have the offset of 10000 after remapping. So this means that cluster-announce-port will have to take two arguments, not just one: the base port and the bus port. Not just that, the header will also have an additional field to announce the bus port.
Does the above makes sense? I'm not finalizing the implementation until I get feedbacks. Thanks.
Comment From: kremso
@antirez Makes sense to me.
Comment From: drshade
Sounds reasonable. You are correct the bus port will no longer be a guaranteed offset from the base port so making that specifiable is good. Looking forward to testing this when you are ready. On Thu, 21 Jan 2016 at 01:04 Tomáš Kramár notifications@github.com wrote:
@antirez https://github.com/antirez Makes sense to me.
— Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/2527#issuecomment-173392811.
Comment From: jpe42
Would it be possible to also add 'client-cluster-announce-port/ip' configurations? This would be useful in scenarios where clients are tunneling in to the redis cluster. That way 'cluster-announce-port/ip' would expose private network ip's and 'client-cluster-announce-port/ip' would expose something like '127.0.0.1:7443' or however the client should have the local tunnel mapping configured.
Comment From: ramonsnir
@antirez Makes sense.
Comment From: korvus81
@antirez That implementation sounds good to me. Is this still planned for 3.2?
Comment From: allanwax
@antirez Looking forward to using this in our environment
Comment From: ovanes
Searched through the source, but did not find any cluster-announce-ip in the 3.2.0-rc3. With 3.0.7 I just receive for the in cluster nodes smth like:
24bbbaeb7792d84d3591fe31ed6052aa6bffaa3e :30001 myself,master - 0 0 0 connected
Thus nodes can't connect to a cluster. How it comes, that there is no IP at all? Even if the startup script writes to the config to connect with $(hostname -i).
Any ideas?
Comment From: allanwax
Just a guess, perhaps null is interpreted as localhost. Again, just a guess
Comment From: antirez
Currently this feature is only in the unstable branch, but will be merged into 3.2 before the final release.
Comment From: antirez
Commits:
5b7b235 Fix to Cluster test to support @busport format. cbcffed Cluster: redis-trib: support @busport format in ClusterNode. b841f3a Cluster: store busport with different separator in CLUSTER NODES. 92b9de2 Cluster announce: WIP, allow building again. 5ac5e3e Cluster announce ip/port/bus-port documented in redis.conf. d0a8512 Cluster anounce-ip/port WIP. 4abf486 Cluster announce port: set port/bport for myself at startup. 1c03837 Cluster: persist bus port in nodes.conf. dc98907 Cluster announce ip: take myself->ip always in sync. 11436b1 Cluster announce ip / port initial implementation. b093930 Cluster announce ip / port configuration handling. a455e4b Cluster: add announce ip field in messages header.
Comment From: ovanes
@antirez: Many thanks for clarification. Do you have a possible timeframe when this is going to happen? I am currently about to plan our release and need to figure out if we can wait for it.
@allanwax: Not sure how this should be the reason.
In general, the VM/container where the cluster node runs has only 2 interfaces: lo and eth0. I also wonder, why the cluster implementation picks the localhost address at all? This is what I saw in vagrant based example when the cluster was deployed to a single VM. I wonder how do people use the cluster in production then if it picks the localhost ip and there is no other way to ask the node to pick the ip from the public interface.
Comment From: korvus81
@ovanes - in my experience, the empty IP only shows up for the node you are connected to. So, you don't need an IP because you are already talking to it. You only need IPs for other nodes, since you may still need to connect to them. I don't know why this is the case, but I would guess it is to make things easier on clients so they don't have to realize they need to ignore that IP.
Comment From: ovanes
@korvus81: Thanks for clarification. Unfortunately, in my use case that's not going to work. I use Kubernetes (I read about docker issues in Redis cluster docs) and it might reschedule a container at any time => containers do not have persistent IP. Thus in front of them a service proxy is put in place. To connect the nodes I need this IP, because service proxy might connect me to any endpoint matching some selection labels and so having no IP there is pretty useless :( I don't really understand, why that is the case, because the node the client is connected to is labelled with myself.
Comment From: chavee
In cluster-announce-ip option, is it possible to use a domain name instead of an ip address? I use the domain name that is accessible from this instance but redis-trib.rb throw an error after "sending cluster meet message to join the cluster" saying that the node address is invalid.
Comment From: ovanes
@chavee: This is my experience as well. I ended up expanding the hostname to ip using this shell command: getent hosts ${CLUSTER_SERVICE_HOST} | awk '{print $1}'. This one depends on implementation in glibc, but is also tested with alpinelinux 3.3 (musl) to work.
There are other solutions to expand the hostname to ip, either using nslookup or ping. As we run CoreOS Linux, ping (ICMP) is not supported by default and this command seemed to be most appropriate to me.
Comment From: jpe42
If anyone on this issue is using Java, I've added a feature to my client library, Jedipus, that supports a host:port mapper. This won't solve the issue for containers, but it will solve the issue for anyone that is just trying to connect to an internally networked cluster. You will need to be able to map from the internally announced ip or port to a public ip:port. Here's a gist on how to create the client with the mapper. This should also be useful for mapping to ssl tunneled connections.
Comment From: fatduo
Hi @antirez @ovanes
After trying redis via Kubernetes multiple times, it still cannot fail over by sentinel.
Sentinel log:
15:X 09 May 02:17:57.893 # +sdown slave 172.16.29.0:6379 172.16.29.0 6379 @ mymaster 172.16.14.3 6379 15:X 09 May 02:17:57.894 # +sdown slave 172.16.77.0:6379 172.16.77.0 6379 @ mymaster 172.16.14.3 6379
Redis master replication info:
role:master connected_slaves:2 slave0:ip=172.16.77.0,port=6379,state=online,offset=193495,lag=0 slave1:ip=172.16.29.0,port=6379,state=online,offset=193495,lag=0
But the slave pods ip (It's the real slave ip, using hostname -i in the slave container) are 172.16.77.3 and 172.16.29.6
It seems that the master get the wrong ip.
Does the slaves got the wrong ip address and send them to the master or the master get the slave ip by the request?
Comment From: ovanes
@fatduo: Redis cluster has nothing to do with sentinel. Kubernetes wrongly calls Redis HA as Redis Cluster which is very misleading. Redis Sentinel already has a support of announce-ip: http://download.redis.io/redis-stable/sentinel.conf and you can manage it from your redis start-up scripts.
Thus I assume, a configuration you deploy/start is wrong somehow. I'd suggest to ask either in Kubernetes Slack Users Group or open a ticket in Kubernetes for support with "Redis Cluster" (named wrongly as K8S does). In addition, you might not start the proper number of Sentinels, so that they can't agree who should be a new master than.
Comment From: fatduo
@ovanes : You missed the point. It's not about the sentinel and announce ip, just about redis master and slave. The master got the wrong ip address of slaves. I think that kubernetes has it's own network, and using hostname -i in the slave container can get the correct ip.
Comment From: ovanes
@fatduo Well, the point is, that your issue is offtopic here. This is what I try to explain and I also write that that sentinel supports announce ip feature => thus the startup scripts are wrong. There is an example in Kubernetes GitHub repo to start redis HA. It worked for me very well on AWS, but you question has nothing to do with this feature request.
Comment From: ovanes
@antirez: Seems like announce ip was not included into the 3.2 release? Can you maybe give some feedback, when the feature is going to be included into the release?
Comment From: jessee9
+1
Comment From: coryjamesfisher
@ovanes Looks like this is the pull request for the merge to 3.2 (has conflicts that need resolution) https://github.com/antirez/redis/pull/3230
Comment From: jpe42
With the new Swarm integration for Docker, it is possible to use an overlay network so that each container task will use that same network rather than a completely isolated internal network. Each Redis node will be reachable from the same IP that it will naturally announce. Here is a complete walkthrough for using it: jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/.
However, if cluster-announce-ip supported hostnames, this would be much easier because you wouldn't have to hack/parse out the IP address for each container service on the overlay network.
Comment From: ovanes
@jamespedwards42 It's not only about docker/swarm :( I use Kubernetes and I am sure there are other technologies in field which have proprietary networking layers and need this features.
Comment From: jpe42
@ovanes Yes, completely understand/agree. I also wanted to point out the pain that is caused by not supporting hostnames. Would that make life any easier in the Kubernetes world?
Comment From: ramonsnir
This also happens when doing cross-data-center work, where each data center has its own internal network. VPNs/IPSecs are expensive to maintain (and keep HA), and slow stuff down.
Comment From: andrewmichaelsmith
@jamespedwards42 Can confirm that a very common method for service discovery in Kubernetes land is hostnames.
Comment From: ovanes
Well, in Kubernetes a pod can't be trivially addressed via a name. Only a service can select using the selectors to which pod to direct the data. Thus having N nodes requires one to create N Kubernetes services, which is not maintainable. The approach here is to retrieve the pod IP (of internal Kubernetes network) and announce it. If pod is rescheduled the startup script will re-determine the IP.
Comment From: jpe42
@ovanes I like that approach as well, but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?
Ultimately what kept me from using that approach is that I want to use network attached storage. With Swarm, volumes are configured at the service level. Not sure about all of the cloud platforms, but GCE only allows a drive to be attached to a single server for writes.
Just general curiosity, do you plan to use Slave nodes as well, or just rely on Kubernetes re-scheduling in conjunction with an NFS? I keep going back and forth on this because failover will probably be faster with Slave nodes depending on the size of your database.
Apologies if this is spamming up this thread, but I'm hoping as a community we can come up with a best-practice for managing Redis Cluster with container orchestration.
Comment From: ovanes
@jamespedwards42
but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?
Yes. I did and it works pretty well. I scheduled 3 masters and each master had 3 slaves. Unfortunately, this script is proprietary work, that's why I can't disclose it.
But in general, Kubernetes has a nice feature, you can enforce a physical port to be allocated on the Host node. Using that, gives you a very simple way to ensure that each master and its slaves run on the same physical port. Forcing them to use that port on the underlying host node, will prevent Kubernetes to schedule the same master and it's slaves on the same node :) and give you a failover distribution.
Finally, from time to time I schedule to run a pod which mounts S3 bucket using s3fs (you can ask Redis to run always as slave). This node just gets the entire-db from one of the distributed master/slaves and shuts downs/sleeps again.
In Redis one can use a config to acknowledge the write request only if one or multiple slaves have also acknowledged it. IMO, you don't need NFS etc, but just analyse number of potential failures and introduce the right amount of slaves with ephemeral local storage, which you don't care about :) As you ensure that most of the infrastructure runs and just in case you do S3 backups.
This setup also works because Kubernetes features a replication controller, where you can setup that you want to have at least N instances of particular pod to be running. This can ensure that you have at least one master and two slaves running on your infrastructure. There are some edge cases, where the entire cluster can become so busy that a new/restarted POD is not going to be started and just hanging in the distributed scheduler's queue for startup, but again this issue can be solved, by labelling the physical node. These labels can than be applied to control which POD are allowed to be run on which nodes.
Comment From: laurie71
This proposal sounds like it would solve the problem for a native Docker configuration, but I don't think it would be sufficient when using docker-machine, as the ports exposed by the Docker image get remapped by the docker machine virtual host. Am I missing something? Is there some way the correct IP/port numbers could be discovered and configured to be the advertised host/port?
Comment From: msch3ung
Does anyone know if there are any updates on this? 3.2 went out and it looks like the cluster-announce-ip is not a valid configuration option.
Comment From: antirez
This proposal is implemented in Redis unstable branch. The first stable version of Redis to support that will be 4.0, released as RC1 15th of October 2016. Anyone testing this support with success in testing environments? Feedbacks appreciated.
Comment From: ovanes
I tested it in our Kubernetes cluster deployment and it worked. It was last time 2 months ago. Worked just fine, and I was able to run a cluster with docker packaged Redis instances.
Also intentionally crashing instances and letting Kubernetes reschedule the entire container worked fine with announcing completely new ip and re-electing a new master. I might be able to retest this starting in september, but can't promise it right now.
Comment From: antirez
@ovanes this is a very useful info, thanks!
Comment From: mostolog
Hi.
Probably this comment doesn't match exactly the topic, but I hope is enough close-related.
AFAIK, redis requires different node.conf cluster files and ports. For that reason, when using docker, entrypoint scripts, variable substitution...are required.
This seems quite incompatible with new docker swarm mode and I was wondering if a gossip protocol, multicast or something else instead of static configurations could be used to create/handle the cluster on an easier way. (ie: the way elasticsearch does). Is this what you have planned for Redis 4.0?
If I understood correctly, this is an example of such trickyness https://jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/
Regards
Comment From: ovanes
@mostolog Can you explain how elastic search does this and what is incompatible with swarm mode? Swarm Mode is smth. Kubernetes does for a long time :) As soon as ip announcement works everything is fine, cluster nodes can be found by contacting an infrastructure service which routes the request to the appropriate node.
Comment From: mostolog
Hi
Consider the following docker command:
docker service create --name my-redis --replicas 3 redis:3.2.1 redis-server /etc/redis/redis.conf
Instead of creating a 3 node redis cluster, it creates a 3 node redis array (3 standalone instances of redis).
It would be great to have an option to configure a cluster name, multicast group, SSDP...to have a "PnP" cluster, if they shared the same password.. In other words: https://en.wikipedia.org/wiki/Zero-configuration_networking
Elasticsearch had multicast support enabled by default on previous versions, but they just moved to unicast, and we are experiencing the same issues: "the need of address to be able to form a cluster" IIRC, docker is considering something like apple bonjour mDNS-SD
Regards
PS: Even more: it would be great allowing redis listen on a ip/mask (eg: 172.16.0.2/16) instead of fixed IP. this would enable dynamic IP configuration while preserving the redis-network isolation
Comment From: ovanes
I don't think multicast is an option :( It'd mean a lot of redesign in Redis, afaik Redis is TCP/IP only, on top this answer might be interesting:
https://groups.google.com/forum/#!topic/coreos-user/tbh1op__qNM
Multicast is very hard to implement with cloud providers and from all available networking layers for docker only weave supports multicast.
Comment From: mostolog
Ok about multicast...but what about the other options?
Have you considered an easier way of deploying a cluster not involving redis-trib/utilities? For example: a unicast list of hostname:port in the config file, sharing the same password and creating a cluster+slots...working just out of the box
If you consider other proposals (bonjour or listening on network/mask) are interesting I can create those issues
Regards
Comment From: mostolog
@vvanholl I didn't mention those, 'cause as a docker user, swarm mode is replacing external registry needs (IMHO: that's the way to take)
Comment From: yank1
Docker Swarm is Good to me
Comment From: bironran
An additional feedback based on my own try-and-fail attempt with Redis cluster and K8s: We need to be able to distinguish between the internal Redis network and the external one. Specifically, we need to be able to have each Redis server identify by two IPs - one for it's fellow cluster members (local network) and that would be reported to clients using the "cluster nodes" and other cluster commands, as "moved" response and when using "asking". Because some commands rely on IP rather than node IDs this is where it gets complicated. With the current implementation I see no easy way out for the general case. However, if we can relax the requirements and commit to no overlap in networks, we can simply use each additional IP as an alias to the original IP the cluster would use, then add alias resolution to commands and add all known IP aliases to each response. As this is breaking protocol, perhaps toggle this by flag.
Comment From: drnybble
Can someone summarize -- can redis be run under Docker Swarm in the master/slave/sentinel configuration? I have tried (redis 3.2) but the cluster seems to get befuddled as containers change IP addresses when they are restarted. It seems to get into a state where it will never elect a master even though there are two redis servers running + 3 sentinels.
Comment From: ovanes
@drnybble If I am not mistaken, this feature was first introduced in Redis 4.0.
Here is what I read from the release history log's section "Redis 4.0.0-RC1 Released Fri Dec 2 10:40:01 CEST 2016" located in (https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES):
- Redis Cluster support for NAT / Docker. There are new functionalities in order to force cluster instances to announce specific sets of IP address, client and bus ports, to the rest of the cluster, regardless of the auto detected IP. This required a bus protocol change that will force users to mass-restart all the nodes of a Redis 3.2 installation in order to upgrade to 4.0.
Comment From: drnybble
Thanks -- I am talking master/slave/sentinel configuration (replication) though, not cluster.
I seem to have some success specifying this for each master/slave:
slave-announce-ip [my-virtual-ip]
I specified the hostname of the service -- there still seems to be some confusion here since the sentinels seem to recognize both the hostname and the ip address as separate entities:
1:X 17 Aug 18:04:37.371 # -sdown slave redis1:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379 1:X 17 Aug 18:04:37.372 # -sdown slave 10.0.0.3:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379 1:X 17 Aug 18:06:05.395 * +reboot slave redis2:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379 1:X 17 Aug 18:06:06.255 * +reboot slave 10.0.0.5:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379
I can always resolve the hostname before writing it to the .conf file if it is a problem.
I was able to force update services for masters and slaves (docker stack update --force redis_redis1 for instance) and so far it seems to be working as expected.
Comment From: ovanes
@drnybble I am a bit mislead: I think this is wrong issue to discuss about announce-ip in HA deployment, as this issue explicitly deals with Redis Cluster.
Comment From: drnybble
@ovanes You are correct. However, I think this information is also relevant for running Redis Cluster in Docker Swarm. The announce-ip should announce the Virtual IP of the service so it is stable over container updates.
Comment From: ovanes
@drnybble Yes, you are right. But than it should be tested with the version where it was introduced than, i.e. 4.0 and not 3.2. And redis release docs explicitly state, that there were bus protocol updates which are incompatible with version 3.2, etc. IMO, all the findings presented for 3.2 are pointless :( As one can't make any kind of conclusion to the version 4.0 because of severe changes.
Comment From: DrGunjah
So is this supposed to work in redis 4.0.2? There are cluster-announce directives now though I still can't get it to work. I have a docker container which contains three redis cluster nodes. I've set up the container in bridge mode and exposed the redis server and bus ports. All ports can be reached from the host machine and from within the container, however if I try to set up the cluster with redis-trib.rb the cluster won't join. I get this message in the redis.log: Unable to connect to Cluster Node [1.2.3.4]:12345 -> Name or service not known (dummy ip and port) However, the address is reachable from host and container. I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.
Cheers
Comment From: wizzu
I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.
The square brackets are part of the error message formatting, so the address is fine. See https://github.com/antirez/redis/blob/d9565379da0adf97caec12e9b88c17da51900e40/src/cluster.c#L3275 (I think this is the spot, but could be wrong).
As for the actual problem, sorry but don't have any idea about that.
Comment From: DrGunjah
@wizzu thank you for pointing this out. So if the ip address is correct I have no idea what else could be wrong at this point.
Comment From: ianmobbs
+1
Comment From: wojnosystems
Hey all. I ran into this as well.
I wrote a proxy that appears to work for simple set/gets on small clusters. This is a work in progress and was created only for testing a RedisCluster running in a docker-swarm, but it should work for K8s, too.
This proxy runs on the cluster-side and alters the IP addresses from the Redis Cluster to lie to clients so that they can route to a single proxy end-point. The proxy end-point listens on a single port for each node in the cluster and passes the traffic onto the actual nodes.
https://github.com/wojnosystems/redis_cluster_proxy
Please feel free to contribute for anything that's broken. This code has some limitations, but this enables me to test my cluster-integration tests.
Hope it helps you as well.
Comment From: yossigo
This has been implemented long ago in 5ac5e3ebd7, closing. This issue has diverted to other topics, please feel free to re-open new issues if relevant.