Describe the bug
We are leveraging eureka for the service discovery function and we have two different deployments for one app as blue-green deployments.
At beginning, deployment A is start up as eureka.instance.initialStatus=UP
while deployment B as eureka.instance.initialStatus=OUT_OF_SERVICE
. After running sometime, we switch them by PUT /eureka/v2/apps/appID/instanceID/status?value={OUT_OF_SERVICE or UP}
. It works good usually.
However, sometimes, we find eureka server is blocked by network issue to this app, and the lease for the app expires. Then eureka server will ask client to re-register. In this case, the initialized status is not correct for the two deployments.
Is there a way that we can override “eureka.instance.initial-status” without restarting these instances? Such as config hot reload? Will it work since app is up and running for a long time.
Comment From: leonard520
I find that the initial status
can be refreshed when EurekaInstanceConfigBean
is reloaded.
So one possible solution is to use Config Server to store eureka.instance.initialStatus
and when want to swap the two deployments,
1. update the value in Config Server
2. curl -XPOST http://{eureka-client}/actuator/refresh
together with overriding the status to the eureka server-side so that they can both update to the new status.
This is a bit complicated, not sure if any other methods can achieve it.
Comment From: spencergibb
You can use the ServiceRegistry.setStatus()
method. https://github.com/spring-cloud/spring-cloud-netflix/blob/main/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/serviceregistry/EurekaServiceRegistry.java#L75
Comment From: leonard520
@spencergibb Thanks for the response. May I double confirm that the operation ServiceRegistry.setStatus()
happens in client app side? If that case, when should the client app set the value? Does it mean the client app should be aware of the time the swap of blue-green happens?
Comment From: spencergibb
ServiceRegistry.setStatus()
can happen anywhere. It's just calling the eureka server API. I'd say that something has to be aware of the blue/green swap, but not necessarily a client.
Comment From: leonard520
@spencergibb I think the problem we found is caused by the obsolete client status. Even the server-side is aware of the blue/green swap, when the client lost connection due to a network issue or so, the client will re-registry with a wrong status.
Comment From: spencergibb
Something has to change the status of that client, there's no other way around it.
Comment From: spring-cloud-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: leonard520
Do we have a plan to improve this behavior that the client can be notified about the change? I believe this is a common issue for Eureka users.
Comment From: gzmrst-sr
We are running into this problem as a user of a PaaS with a provided Eureka server. The PaaS does blue/green deployments by setting the status as described by @leonard520 above... The application being deployed has no knowledge of how it is deployed or of any change it needs to make in the status.
What should be the intended behaviour in situations like this? Do we have to customise the client side Eureka client code to change the initial status? Are there any plans to fix blue/green deployments with Eureka?
Comment From: spring-cloud-issues
Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.
Comment From: gzmrst-sr
We are still struggling with this issue. It happens whenever the app loses connection to Eureka for a time, regardless of the cause.
@spencergibb Could you please answer my question above? How is it supposed to work?
Or is it the case that the initial status was not intended for use like this? (Although I have trouble seeing where else it would be used)
In any case, the problem is that the Eureka client is liable to re-register itself with the initial status every time it loses connection. And there is no mechanism to change the initial status that I am aware of.
Comment From: gzmrst-sr
@spencergibb @OlgaMaciaszek Apologies for bumping an old issue, but it is still a problem for us and we'd greatly appreciate some pointers on how it's supposed to work and/or how we should work around the problem. Right now we have to just accept occasional outages caused by an incorrect Eureka status and restart everything, which seems sub optimal to say the least.
Comment From: spring-cloud-issues
Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.
Comment From: gzmrst-sr
@spring-cloud-issues @OlgaMaciaszek Yes please, we would still like to have our questions around this issue answered...
Comment From: spring-cloud-issues
Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.
Comment From: spring-cloud-issues
Closing due to age of the question. If you would like us to look at this issue, please comment and we will look at re-opening the issue.