Hi there,

this may be somewhat related to #275 or #514 but I thought I'd open it anyway, since a) I have a project in GitHub that can be used to reproduce some of the effects b) I have noticed a few other quirks I am not sure about

But before that, let me say that I am a big fan of Spring Cloud and the work you do!
Super cool stuff! Looking very much forward to the RSocket / Spring Cloud Gateway talk!

My project setup:

  • I am using the Discovery First setup
  • config-server is a Spring Cloud Config Server connecting to Eureka service-registry
  • reservation-service is a service written with Spring Boot 2.2.0.M6 and Spring Cloud Hoxton.M2 exposing a RestController that sends a greetings message that is configurable
  • I use Spring Cloud Bus (based on RabbitMQ) and Spring Cloud Config Monitor to receive WebHook calls from GitHub and auto-refresh the services attached to the Rabbit bus.

What I would like to achieve:

  • Ideally, I would like to start each of the involved applications, i.e. config-server, service-registry and reservation-service in an arbitrary order. This is to make deployment easier and less error prone. The services should find each other, once they are up and running.
  • If the reservation-service gets up faster than the config-server, I want it to keep checking for the config-server via Eureka, until it is there, and then fetch the configurations.

My observations:

Pre-requisites for all scenarios described: 1. start up Zipkin server using scripts/startZipkin.sh 1. start up Rabbit server using scripts/startRabbit.sh

Scenario 1:

  1. Start up service-registry
  2. Start up reservation-service

Observation: reservation-service fails, with an exception stating the config value com.equalities.greeting could not be resolved. This is referenced via @Value in the RestController. After that, no retry is attempted anymore, instead exceptions for RabbitMQ connections are trhown:

org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[springCloudBus.anonymous.6NlJCiF3TzOG_9HOl4BIeA]

Expected behavior: The reservation-service is in an illegal / unhealthy state, and keeps retrying for the config server, until it finds it via Eureka. It then fetches the configs and refreshes itself, turning healthy.

Scenario 2:

  1. Comment out Line 17 in RestController, essentially removing the configuration reference.
  2. Start up service-registry
  3. Start up reservation-service

Observation: reservation-service starts, but complains that the config-server could not be found. Then it will keep on retrying to find the config-server. When starting the config-server then, reservation-service keeps retrying and never finds it. No RabbitMQ-related errors are thrown.

Expected behavior: reservation-service should retry until config-server instances are up and running and registered in Eureka, then pick one and, get the configs and stop retrying. If the config-server instance goes down, reservation-service should select a new one, from Eureka's list or start retrying if no other is available.

Scenario 3:

  1. Comment out Line 17 in RestController, essentially removing the configuration reference.
  2. Add spring.cloud.config.fail-fast=true to reservation-service's bootstrap.yml as described here.
  3. Start up service-registry
  4. Start up reservation-service

Observation: reservation-service starts, but complains that the config-server could not be found. Then it stops without retrying. This is different than what is described in the Spring Cloud Config documentation (which I assume is simply wrong).

Expected behavior: In my opinion this behavior is even expected, since the flag is called failFast. Yet, since the documentation describes it differently, I am not sure what to make of this.

Questions

  1. Is what I want to achieve actually possible / intended?
  2. Are the described scenarios bugs or intended?

Thanks!

Comment From: spencergibb

We need a minimal project. So please remove everything that isn't necessary to reproduce scenario 3 (including zipkin and bus, etc...). Also please update to the latest boot release and hoxton milestone (m3).

Comment From: FWinkler79

You can find an updated version here. This now references Spring Boot 2.2.0.RELEASE and Hoxton.BUILD-SNAPSHOT. The issue described in scenario 3 is still there.

To reproduce: 1. start service-registry 1. start reservation-service

According to this documentation, reservation-service should start retrying 6 times with backoff before it fails. However, that is not the case.

This would be required, though, to make the deployment order of services vs. config-service reasonably decoupled.

Comment From: spencergibb

Your app fails with

Consider defining a bean of type 'org.springframework.cloud.client.discovery.DiscoveryClient' in your configuration.

2019-10-21 16:30:20.260 ERROR 13097 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration': Unsatisfied dependency expressed through field 'instanceProvider'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'configServerInstanceProvider' defined in class path resource [org/springframework/cloud/config/client/DiscoveryClientConfigServiceBootstrapConfiguration.class]: Unsatisfied dependency expressed through method 'configServerInstanceProvider' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.client.discovery.DiscoveryClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:639) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:116) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:397) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1429) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:206) ~[spring-cloud-context-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:117) ~[spring-cloud-context-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
    at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:74) ~[spring-cloud-context-2.2.0.BUILD-SNAPSHOT.jar:2.2.0.BUILD-SNAPSHOT]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    at com.equalities.cloud.reservation.service.ReservationService.main(ReservationService.java:24) ~[classes/:na]
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'configServerInstanceProvider' defined in class path resource [org/springframework/cloud/config/client/DiscoveryClientConfigServiceBootstrapConfiguration.class]: Unsatisfied dependency expressed through method 'configServerInstanceProvider' parameter 0; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.client.discovery.DiscoveryClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:787) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:528) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1287) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:636) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    ... 30 common frames omitted
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.springframework.cloud.client.discovery.DiscoveryClient' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1695) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1253) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:874) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:778) ~[spring-beans-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    ... 43 common frames omitted

Comment From: FWinkler79

This may have been an issue since the sample is using Hoxton.BUILD-SNAPSHOT and these keep changing. I have updated the project once again, you can it here.

To reproduce:

  1. start service-registry
  2. start reservation-service

According to this documentation, reservation-service should start retrying 6 times with backoff before it fails. However, that is not the case.

This would be required, though, to make the deployment order of services vs. config-service reasonably decoupled.

Note that this order works fine: 1. start service-registry 1. start config-service 1. start reservation-service

Comment From: spencergibb

Retry was fixed with f0bc1af54719eb19b58cbff54998801653e33ff7