Spring Cloud Netflix Using Eureka with Blue/Green Deploy inside of PCF

We are running into an issue where our apps are being registered with a route that is being changed by our blue green deploy script but it isn't clear on a way to cleanly have it re-register or correct the info because of permissions issues around PCF.

Let me elaborate: We have PCF onsite and are using it for hosting of applications. As part of the CI/CD pipeline, we have a script to perform a blue/green deploy. This script creates a new app and registers it with a route of "app-new.cfapps.com". It then performs a test against that application and if it passes, it changes the routes of the new app to "app.cfapps.com" and the old application to "app-old.cfapps.com" and spins it down to save resources.

The issue as I understand it is that when our application is standing up the Spring applicationContext, it has the @EnableDiscoveryClient annotation and so it is using the information from the provided PCF service registry "service" to connect to the Eureka server instance. When it does this, it still has the route of "app-new.cfapps.com" So it sends the heartbeat, gets the 404 and then registers with that route.

At some point after this, it has passed its testing and the route has now been changed to "app.cfapps.com". However, the Eureka server still thinks the app has the route of "app-new.cfapps.com" from the initial set of heartbeats. That means that when any other service attempts to use the service registry to send requests to this application, they get 404's because they are going to "app-new.cfapps.com" which doesn't exist anymore.

I have seen suggestions around doing a PUT to update the metadata on the Eureka server directly but we are running into issues where we don't have permission because of the nature of the PCF "services" that are available not being accessible readily to units outside of PCF.

Do you have any suggestions or things we've overlooked to deal with this issue? Please let me know if you have any questions about the scenario.

Thanks, Charles

Comment From: ryanjbaxter

I have seen suggestions around doing a PUT to update the metadata on the Eureka server directly but we are running into issues where we don't have permission because of the nature of the PCF "services" that are available not being accessible readily to units outside of PCF.

Are using Spring Cloud Services in PCF to deploy Eureka or are you using the open source Spring Cloud Eureka server?

Comment From: azizabah

@ryanjbaxter Spring Cloud Services in PCF to deploy Eureka and binding it to our apps using their manifests.

Comment From: ryanjbaxter

So it sounds like this is a restriction on Eureka from Spring Cloud Services then, no?

Comment From: azizabah

@ryanjbaxter Yea a bit. I was hoping I could configure something in my application to prevent the initial registry and was looking at this setting eureka.client.initial-instance-info-replication-interval-seconds but I don't think that behavior is exactly as I expected. I set it to 10 minutes to test and it definitely registered with Eureka well before ten minutes had elapsed.

I'm mainly just trying to see if there's something we've missed that solves this use case inside of our Spring Boot Java applications.

It feels like this is a use case that other people should have seen but we're not running into any clean solutions for it. Our current workaround is to immediately restart the app after it passes testing and has the final route to get it to register with Eureka correctly. While that "works", it definitely feels less than ideal.

Comment From: william-tran

The root of the problem is in how to update a Java process with changes to VCAP_APPLICATION which contains the application_uris. I'm pretty sure this is impossible. We'd need another way for the application to monitor changes to it's "environment", once we have that, it could re-register.

So what you're doing by restarting the app is the only thing that can work at the moment.

Comment From: william-tran

I have seen suggestions around doing a PUT to update the metadata on the Eureka server directly but we are running into issues where we don't have permission because of the nature of the PCF "services" that are available not being accessible readily to units outside of PCF.

To be able to modify a registration record (eg via PUT), you must authenticate as the creator of that record, meaning getting the binding credentials of the app that registered, getting an oauth token via client credentials grant, and then doing the PUT with the token in the header.

Comment From: azizabah

@william-tran I was hoping I could do something along those lines by possibly exposing an endpoint in my new app that our deploy script could call once the routes were finalized to have it make that request back to Eureka but I don't remember seeing anything along those lines when I was looking.

My app should have the necessary credentials and information about Eureka to make a PUT to it but I'm a little unfamiliar with it since the Spring annotation handles all the real work related to Eureka.

Do you think that's a possible approach or it will have issues?

Comment From: ryanjbaxter

I was just revisiting this and I had another idea. Could you use the service status to bring the service in use after the tests pass? In other words register the new version with the correct route, but initially set its status to OUT_OF_SERVICE. Once the test pass change the status to UP so traffic will be routed to it.

Comment From: azizabah

@ryanjbaxter we actually had a plan to do something along those lines but we were waiting for a promised upgrade to the PCF CLI from one of the Pivotal folks we had out saying that you'd be able to do that through the CLI.

While we were waiting for that, we had some conversations with other folks in a sister organization and they had an alternative idea of having the initial route never get dropped so that route registered with Eureka would always be legitimate and we're looking at going down that path.

So for example, a new app would get registered as "app-green" route initially. Once it passes it's smoke tests, it would additionally get the "app" route so it would have both "app" and "app-green". Next app coming in would get "app-blue" and then you get more normal blue green deployment style behavior. This also fixes the issue of the initially registered with Eureka route continuing to work.

I'll go ahead and close this. Thanks for the follow up on it though.

Comment From: ryanjbaxter

OK. I have been playing around with this a little bit, you can take a look at what I have written up here https://github.com/ryanjbaxter/bluegreen#running-the-sample-on-cloud-foundry

Comment From: azizabah

That's an interesting read. I'll go over it with our devops folks and we might incorporate some of that into part of our B/G script. I like the idea of relying on the Eureka status over routes to handle getting to the "UP" service.

I was also unaware of the /service-registry endpoint that you're referencing. That can definitely help simplify things.

Thanks for the info @ryanjbaxter !

Comment From: ryanjbaxter

Sure no problem. If you have any feedback be sure to let me know.

Comment From: Sarvesh-D

Hi @ryanjbaxter , i wanted to implement a simple workaround which will help blocking the instances having some pattern from registering to eureka. This pattern is typically what our blue-green deployment adds to start a green instance on PCF. By registering custom InstanceRegistry in my eureka server i was able to block any instance which contains green in their hostname. Do you see any issue with this approach?

public class CustomInstanceRegistry extends InstanceRegistry {

    public CustomInstanceRegistry(EurekaServerConfig serverConfig, EurekaClientConfig clientConfig, ServerCodecs serverCodecs, EurekaClient eurekaClient, int expectedNumberOfClientsSendingRenews, int defaultOpenForTrafficCount) {
        super(serverConfig, clientConfig, serverCodecs, eurekaClient, expectedNumberOfClientsSendingRenews, defaultOpenForTrafficCount);
    }

    @Override
    public void register(InstanceInfo info, boolean isReplication) {
        if (containsIgnoreCase(info.getHostName(), "-green"))
            return;
        super.register(info, isReplication);
    }

    @Override
    public void register(InstanceInfo info, int leaseDuration, boolean isReplication) {
        if (containsIgnoreCase(info.getHostName(), "-green"))
            return;
        super.register(info, leaseDuration, isReplication);
    }
}