Spring Cloud Netflix Eureka client local cache doesn't refresh in a "discovery first bootstrap" scenario

Describe the bug I've set up 3 spring boot apps: 1. discovery (eureka server) listening on port 9000 2. config server on 9001 3. admin application (spring admin) on port 9002

On admin application I configured bootstrap.yml to retrieve settings from config server using discovery, like this:

spring:
  cloud:
    discovery:
      blocking.enabled: true
      reactive.enabled: false        
    config:
      enabled: true
      discovery:
        enabled: true
        service-id: config
      fail-fast: false
      retry:
        initial-interval: 2000
        multiplier: 1.5
        max-attempts: 10
        max-interval: 30000

When starting apps in order with some delay (discovery then config, then after a while admin) everythings seems OK.

But if I change topology (for example shutting down config), admin doesn't refresh its cache. Can be checked using /actuator/health/discoveryComposite endpoint that still lists config, while on discovery is disappeared. Moreover, eureka status remains Eureka discovery client has not yet successfully connected to a Eureka server

Another test: if I start discovery first and then config + admin together (or at least before config become STARTED on Eureka server), admin eureka cache remains empty and I get Could not locate configserver via discovery warn on all subsequent retries.

Doing some debugging, seems to be related to the presence of two differents DiscoveryClient (one in bootstrap and the other in the application context?).

Reproduced with Spring Boot 2.2.2 + Spring Cloud Hoxton.RELEASE. Similar issues were present also on 2.1.10 (not analyzed in depth)

Sample I committed a repo with demo apps to reproduce the issue. https://github.com/bberto/springcloud-bootstrap-config

Comment From: spencergibb

If there are very few instances eureka will enter self preservation mode and will not evict instances.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: bberto

I disabled it with eureka.server.enable-self-preservation=false but I can't notice any difference. I think the root cause is the one reported here https://github.com/spring-cloud/spring-cloud-netflix/issues/3675#issuecomment-565388117

Comment From: Manuel-Linkener

I have the same problem. I run a eureka-server and config-server. Now I have a service with a config client in discovery first mode.

spring:
    config:
      discovery:
        enabled: true

discovery first config client bootstrap.yml

On boot, the discovery client fetches the list of services (to find the config server) and then never updates the list of available services again.

When you hit the actuator health endpoint you see:

{
  "status": "UP",
  "components": {
    "discoveryComposite": {
      "status": "UP",
      "components": {
        "discoveryClient": {
          "status": "UP",
          "details": {
            "services": [
              "config-server"
            ]
          }
        },
        "eureka": {
          "description": "Eureka discovery client has not yet successfully connected to a Eureka server",
          "status": "UP",
          "details": {
            "applications": {
              "CONFIG-SERVER": 1
            }
          }
        }
      }
    }
  }
}

It shows only the services, that where registered at the time the service booted (in my case there was just the config-server) and never changes again.

Now I deploy another service, that does not use service discovery for the config client and the ....eureka.description changes after a while to Remote status from Eureka server and the list of services updates. But if you start the service with a config client in discovery first mode, the list of services never updates.

I am not sure if this is a spring cloud netflix issue, looks more like a config client (or service discovery client) issue, but since I found this issue, I post it here.

Comment From: OlgaMaciaszek

Putting it on hold - we'll be discussing it with Netflix to try to come up with a solution.

Comment From: aveiros

Is the best work around for now to exclude EurekaDiscoveryClientConfigServiceAutoConfiguration ?

Comment From: OlgaMaciaszek

Changes need to be made in order to avoid EurekaClient for registration in SC Config. For a temporary workaround, please see this one proposed by @k631583871.

Closing in favour of SC Config #514.

Comment From: Manuel-Linkener

I just tested it again with Spring Cloud Hoxton.SR5 and Spring Boot 2.3.0 the issue is fixed 🎉🎉🎉 Thank you very much @OlgaMaciaszek and the guys in https://github.com/spring-cloud/spring-cloud-netflix/issues/3795 for fixing this!