I'm having an issue where the CloudEurekaClient bean is created twice.
I've tried both combinations of Spring Boot 2.1.2.RELEASE and 2.1.5.RELEASE as well as Spring Cloud versions for 2.1.0.RELEASE and 2.1.2.RELEASE, but the problem persists.
I've also tried adding and removing the @EnableEurekaClient but it had no effect.
The service starts, the first CloudEurekaClient is created, starts, and registers with Eureka, then EurekaDiscoveryClientConfigServiceAutoConfiguration shuts it down. A second CloudEurekaClient is then created, but the first one still exists even though it is shut down.
The issue then becomes:
1) Registrations, heartbeats, refreshes, and renewals use the SECOND bean. 1) But the actuator/health and local copy of the registry use the FIRST bean.
This causes the following issue:
The service starts, goes to Eureka, gets the initial endpoints for ConfigServer and then the local copy of the service registry is never updated. The list of services in actuator/health then becomes stale since a the second, newer bean is being updated. This means the normal service-service calls or calls to the ConfigServer may fail since the registry is stale. The health endpoint shows the client as UP but Eureka discovery client has not yet successfully connected to a Eureka server.
I've created a sample project: https://github.com/abracadv8/test-eureka-client
It contains some screenshots as well as the logs.
Sample logs:
2019-10-15 10:51:24.135 [o.s.c.c.u.InetUtils] Cannot determine local hostname
2019-10-15 10:51:24.166 [o.s.c.n.e.InstanceInfoFactory] Setting initial instance status as: STARTING
2019-10-15 10:52:41.184 [c.n.d.DiscoveryClient] Initializing Eureka in region us-east-1
......
2019-10-15 10:52:43.676 [c.n.d.DiscoveryClient] Discovery Client initialized at timestamp 1571151163673 with initial instances count: 44
2019-10-15 10:52:43.741 [o.s.c.n.e.s.EurekaServiceRegistry] Registering application TEST-EUREKA-CLIENT with eureka with status UP
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.5.RELEASE)
......
2019-10-15 10:52:53.864 [o.s.c.c.u.InetUtils] Cannot determine local hostname
2019-10-15 10:52:55.873 [o.s.c.c.u.InetUtils] Cannot determine local hostname
2019-10-15 10:52:56.010 [c.n.d.DiscoveryClient] Shutting down DiscoveryClient ...
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< SHUTS DOWN
2019-10-15 10:52:56.026 [c.n.d.DiscoveryClient] Completed shut down of DiscoveryClient
2019-10-15 10:52:56.048 [o.s.b.a.e.w.EndpointLinksResolver] Exposing 19 endpoint(s) beneath base path '/actuator'
2019-10-15 10:52:56.328 [o.s.c.n.e.InstanceInfoFactory] Setting initial instance status as: STARTING
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< STARTS BACK UP
2019-10-15 10:53:19.028 [c.n.d.DiscoveryClient] Initializing Eureka in region us-east-1
......
2019-10-15 10:53:19.309 [c.n.d.DiscoveryClient] Not registering with Eureka server per configuration
2019-10-15 10:53:19.311 [c.n.d.DiscoveryClient] Discovery Client initialized at timestamp 1571151199310 with initial instances count: 44
Comment From: ryanjbaxter
What is causing the shut down?
Comment From: abracadv8
The EurekaDiscoveryClientConfigServiceAutoConfiguration class.
It is a conditional configuration:
@ConditionalOnBean({ EurekaDiscoveryClientConfiguration.class })
@ConditionalOnProperty(value = "spring.cloud.config.discovery.enabled",
matchIfMissing = false)
And it basically checks to see if a EurekaClient already exists and shuts it down:
// If the parent has a EurekaClient as well it should be shutdown, so the
// local one can register accurate instance info
this.context.getParent().getBean(EurekaClient.class).shutdown();
I am not sure where the first one is being created at.
Edit
Ill try to dig a bit more into the configurations.
Comment From: abracadv8
Okay, from what I can tell:
EurekaClientAutoConfiguration creates a CloudEurekaClient via its internal static EurekaClientConfiguration class. Part of that is to wire in the healthCheckHandler in EurekaClientAutoConfiguration as seen here. This is done through AnnotationConfigApplicationContext.
The client immediately starts the registration process. I believe the health check handler part of the health endpoints are then tied to this bean.
Shortly after the rest of the Spring context loads: EurekaDiscoveryClientConfigServiceAutoConfiguration starts up and immediately shuts it down.
After that, a lazy-loaded CloudEurekaClient is created via EurekaClientAutoConfiguration, but this one contains no healthCheckHandler. This is done through AnnotationConfigServletWebServerApplicationContext.
A new ApplicationInfoManager and InstanceInfo are created separately for each bean, which possibly explains why it appears to connect to stale entries that should normally be refreshed.
I've set the application up to register with Eureka first, then get the ConfigServer registration listing, then reach out to one of them for properties.
I think it has to do with the fact that the bootstrap.yml setsspring.cloud.config.discovery.enabled=true for EurekaDiscoveryClientConfigServiceAutoConfiguration. That causes the first client to immediately shut down when it first starts. However, i can't explain why it would re-create the bean, especially when it is conditional on @ConditionalOnMissingBean(value = EurekaClient.class, search = SearchStrategy.CURRENT) as seen in EurekaClientAutoConfiguration
Properties:
spring:
application:
name: ${SPRING_APPLICATION_NAME:test-eureka-client}
cloud:
.......
discovery:
service-id: cloudads-configserver
enabled: true <<< this causes the shutdown and re-creation
Comment From: abracadv8
Oh, I believe this has other minor issues when using the config server.
Since the registration starts BEFORE it can find the config server, the first registration temporarily registers with the default port of 8080, finds the config server, unregisters and re-loads whatever context so it can get the remote properties.
It then registers a second time with the correct port
Comment From: tdanylchuk
Eureka Client is refresh scoped bean, and in case you are using cloud, it will be bootstraped twice. Since spring boot starts its context and tries to load cloud (like configuration service) after loading cloud context (including properties from configuration service as additional high priority property source) it will refresh itself. And while eureka client is refresh scoped bean, it will be recreated, means re-registered with proper values from configuration service.
Comment From: bberto
@abracadv8 thanks for your analysis. I didn't notice that DiscoveryClient is shut down and then restarted.
This relates with #3708
Seems that ConfigServerInstanceProvider continues to use the shut down DiscoveryClient, leading to Could not locate configserver via discovery if config server is not already registered at boot time.
Also EurekaHealthIndicator uses the shut down bean, causing Eureka discovery client has not yet successfully connected to a Eureka server error and no updates shown on health indicator
Comment From: davgia
@abracadv8 @bberto
This is also related to #1327.
I've too detected this strange behaviour where only at the startup the eureka registry is fetched correctly. All subsequent calls to getService() method of the EurekaDiscoveryClient will result in the same collection of services.
I can confirm that disabling the discovery on spring.cloud will solve problem (tested with SBA).
spring:
cloud:
config:
discovery:
enabled: false
Moreover, I can confirm that the problem is related to Eureka Client because when deploying on a kubernetes cluster everthing works fine (on kubernetes I need to disable eureka entirely and enable spring.cloud.kubernetes components).
Comment From: grigorryev
Hi, are there any known solutions for this?
I have my own code injecting DiscoveryClient, and looks like it injects the first DiscoveryClient, the one to be rejected after config fetch. I obviously need to inject the second one, because the first one doesn't receive any changes from eureka.
Comment From: k631583871
spring-cloud/spring-cloud-config#514
2139
These are caused by one problem
Comment From: k631583871
@OlgaMaciaszek I tried to fix this bug https://github.com/spring-cloud/spring-cloud-netflix/pull/3766
Comment From: OlgaMaciaszek
@k631583871 thanks, have marked that PR for team discussion.
Comment From: OlgaMaciaszek
Changes need to be made in order to avoid EurekaClient for registration in SC Config. For a temporary workaround, please see this one proposed by @k631583871.
Closing in favour of SC Config #514.