Question / Bug report

Hi,

I have a service running on my local pc, connecting to a eureka server running on a remote machine in the office. When docker is running on the client, the client serves the docker bridge (docker0, 10.0.71.1) ip as discovery instead of the host ip it is running on (10.0.1.xxx). Client is running in intellij (debug mode).

When stopping the docker service it returns to its normal host ip.

Running on linux machines, ubuntu 16.04 Used versions

<spring-boot-dependencies.version>1.5.13.RELEASE</spring-boot-dependencies.version>
<spring-cloud-dependencies.version>Edgware.SR3</spring-cloud-dependencies.version>

client yml

eureka:
  client:
    serviceUrl:
      defaultZone: http://eureka:8761/eureka/
  instance:
    prefer-ip-address: true
    leaseRenewalIntervalInSeconds: 10
    leaseExpirationDurationInSeconds: 30

Any suggestions, workarounds? Or is the actually a bug?

Comment From: ryanjbaxter

You might want to debug in this method https://github.com/spring-cloud/spring-cloud-commons/blob/ca1a97a3a484910527b17e95ae2cc6e1c06722c6/spring-cloud-commons/src/main/java/org/springframework/cloud/commons/util/InetUtils.java#L76. Seems like something might be going on in your environment.

Comment From: spencergibb

It's likely picking the internal Network first. The docs have options to exclude our include certain networks

Comment From: ryanjbaxter

http://cloud.spring.io/spring-cloud-static/Finchley.RELEASE/single/spring-cloud.html#ignore-network-interfaces

Comment From: tobiasvdp

Thanks, for some reason the bridge was indeed targeted as interface instead of the external network. ignoring the interface solved the issue.

Comment From: RodrigoPetter

So let me see if I get it right.

In my docker swarm, I have a custom network test_nerwork (10.0.0.0/24). When I try to deploy a new services that exposes a port, docker will create a Ingress network that is used for load balance and routing-mesh(in my case 10.255.0.0/16). I'm not 100% sure but not always this ingress network is the eth0 interface inside the container (can't find any documentation).

The services containers can't use the ingress network for http communication.

With the default Eureka configs, the services will use the eth0 interface IP (docker Ingress network) for registration. All communications with this service will fail because of the wrong IP advertised by the container.

To solve this problem I have 3 options:

  1. Explicit say what IP should by advertised by the service (this only works because of the docker swarm routing-mesh). You may use any of yours swarm nodes IP.
eureka:
  instance:
    preferIpAddress: true
    ip-address: $MY_HOST_IP

By doing this you may overload the host that you chose you IP from. I don't think this is a good option.

  1. Ignore all networks interfaces and set a preferred one. 
spring:
  cloud:
    inetutils:
      ignoredInterfaces:
        - docker
        - eth.*
        - lo
      preferredNetworks:
        - 10.0.0

Only setting the preferred one will not work.

  1. Not use an IP address at all.
eureka:
  instance:
    preferIpAddress: true
    hostname: $SERVICE_NAME

Where $SERVICE_NAME must be the name of the service in your swarm. It will be registered at Eureka as http://myservice:6003 and will work fine if you don't have any service running from outside the docker swarm. I just don't know how the load balance will behave (Ribbon + Swarm DNS).

Comment From: simplyi

@spencergibb @ryanjbaxter

It looks like the spring.cloud.inetutils.ignoredInterfaces does not work with Greenwich.RELEASE.

I run docker container with docker run -e "spring.cloud.inetutils.ignoredInterfaces=bridge" and it still registers with Eureka (running on a different remote server) using the private IP address rather than a public IP address of host machine.

I have tried ignoring docker0 as it is shown in the documentation page but getting the same result. Tried adding spring.cloud.inetutils.ignoredInterfaces=bridge to a application.properties file and to bootstrap.properties file but still the same result. When Microservice registers with Eureka it uses the private IP address of the host machine even though this machine is assigned a static public IP address.

The only way I am able to make my microservice register with Eureka Discovery with a public IP address is by using these two properties:

docker run -e "eureka.instance.ip-address=xx.xx.xxx.xxx" -e "eureka.instance.prefer-ip-address=true"

This way my microservice does register with Eureka using the public IP address. But I do not like this approach because I have to provide public IP address manually for each Microservice I start.

Is there a way to make a Microservice running in a docker container to register with Eureka(running on a remote server) using a public IP address of its host machine?

Comment From: ryanjbaxter

Please open a separate issue

Comment From: saba83rish

Could you share the new issue link please?

Comment From: filpano

Commenting here as it does pertain to the issue:

@RodrigoPetter

I just don't know how the load balance will behave (Ribbon + Swarm DNS).

I can confirm that this combination (Ribbon with service name routing instead of IP + Swarm DNS round-robin) can cause issues with load balancing when all instances of the destination service are running on the same port, internally (by e.g. setting server.port to a non-zero value).

The reason for this behaviour is explained in the following Docker documentation page: https://docs.docker.com/network/overlay/#bypass-the-routing-mesh-for-a-swarm-service

Quoting:

To bypass the routing mesh, you can start a service using DNS Round Robin (DNSRR) mode, by setting the --endpoint-mode flag to dnsrr. You must run your own load balancer in front of the service. A DNS query for the service name on the Docker host returns a list of IP addresses for the nodes running the service. Configure your load balancer to consume this list and balance the traffic across the nodes.

Since Ribbon already performs the LB under the assumption that the server list contains unique entries (in this case, it will not), ultimately the Ribbon Client server lists would look as follows (example: port 8080):

RibbonClient_service-1=[service-1:8080, service-1:8080, ... ],
...
RibbonClient_service-2=[service-2:8080, service-2:8080, ... ]

Since they are both identical, trying to resolve e.g. service-1:8080 will actually let the Docker daemon choose the destination host.

The end result of this is a "sticky" load balancing wherein a call to e.g. service-1 would call instances in (for example) the following order:

1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1

with seemingly no real deterministic order, other than that it tries to be round-robin. It ultimately is not, and can cause a discrepancy (that I've observed) of up to 10-20% in incoming requests between instances leading to uneven load spikes.

The solution to this problem (with Docker DNSRR enabled) would be to either:

  1. Set unique ports per application instance (server.port=0)
  2. Register to Eureka via IP instead of hostname by setting eureka.instance.preferIpAddress=true) on client applications.

I've opted for the latter, since even with random ports the chance is non-zero for this situation to transparently occur again, causing no end to troubleshooting headaches.

Hopefully this helps someone, as it was quite tricky to debug.

Comment From: ffroliva

Have you tested if this situation happens with Consul as discovery server?