Not sure whether to put this report here, cloudfoundry, netty or with spring cloud gateway but as I can reproduce this issue consistently updating to spring-boot 2.7.10 while leaving everything else untouched I decided to put it here. Any help is appreciated.
We're running spring-boot-starter with spring-cloud-gateway 3.1.16 on cloud foundry platform. When updating from spring-boot-starter 2.7.9 to 2.7.10 the /cloudfoundryapplication-Endpoints (which are accessed by cloudfoundry-Apps-Manager for example by using Google Chrome) stop working.
This means that using actuator endpoints like configuring loggers on the fly or viewing build info via spring-boot-maven-plugin) no longer work with cloudfoundry-Apps-Manager.
This behaviour seems to result from a bad ssl-handshake. By observing the network protocol in Chrome Browser the following seems to happen in version 2.7.10: 1) OPTIONS-Request to https://[host]/cloudfoundryapplication (port 443) 2) GET-Request to https://[host]/cloudfoundryapplication happens (port 443). The response body contains json with urls of actuator-endpoints. However, these endpoints contain port 80, example:
{"_links":{"caches":{"href":"https://[host]/cloudfoundryapplication:80/loggers","templated":false},
3) On subsequent requests of cloudfoundry-apps-manager to https://[host]:80/cloudfoundryapplication/info (or /loggers etc) no https-connection can be established (connection refused) 4) However, manual requests to https://[host]:443/cloudfoundryapplication/info return status code 200 and the expected result indicating that the acutator-endpoints are still configured properly
Until version 2.7.9 the request in 2) returned endpoints with no port (indicating using default https-port 443).
{"_links":{"caches":{"href":"https://[host]/cloudfoundryapplication/loggers","templated":false},
Please note that we're also using spring-boot-mvc 2.7.10 with cloudfoundry where this issue does not occur so it maybe a netty issue.
Furthermore, server.management.cloudfoundry options are not set in application.yml (thereby defaulting).
Our actuator configuration revealsonly health; however this is unchanged since 2.7.9 and /cloudfoundryapplication requests seem not to be affected in any way by this configuration as it used to work as described.
management:
endpoints:
web:
exposure:
include: [ "health" ]
base-path: /actuator
Comment From: scottfrederick
Looking over the Spring Boot changes between 2.7.9 and 2.7.10, I don't see anything that might cause the :80 port to be added to the JSON response from the actuator endpoints.
I can reproduce this issue consistently updating to spring-boot 2.7.10 while leaving everything else untouched
If you would like us to spend some time investigating, please provide a complete minimal sample that works with Spring Boot 2.7.9 and fails with 2.7.10 with no other changes. You can share it with us by pushing it to a separate repository on GitHub or by zipping it and attaching it to this issue.
Comment From: boborperry
@scottfrederick Thanks for getting in touch so quickly and checking spring boot changes.
I added a simple maven project which reproduces the described issue after mvn clean package, cf push and assigning a route in cloud foundry.
As I found out, replacing gateway-dependency with webflux will also result in port 80 appended. Furthermore not only requests to cloudfoundryapplication have port 80 included but also requests to actuator resource directly (which is probably not surprising because of same resource mapping).
Comment From: boborperry
A workmate suspected a problem concerning the x-forwarded-port header which led me to this issue: https://github.com/spring-projects/spring-boot/issues/24796. Cloudfoundry uses also a load balancer.
Setting forward-headers-strategy to framework or none will restore the original behaviour. forward-headers-strategy: native keeps port 80 and the problem persists.
I'm not yet done analyzing the impact of this setting for our gateway; however there must be have been a change in default behaviour of netty between 2.7.9 and 2.7.10.
Comment From: boborperry
@scottfrederick So maybe this commit by @violetmight be responsible for the changed behaviour I am experiencing: https://github.com/reactor/reactor-netty/commit/5990ca8c689e38098e034bd3fb300e48d92959dd
Before the port was identified by the x_forwarded_proto Header (which is present in /cloudfoundryapplication-Request)
Matcher protoMatcher = FORWARDED_PROTO_PATTERN.matcher(forwarded);
if (protoMatcher.find()) {
connectionInfo = connectionInfo.withScheme(protoMatcher.group(1).trim());
}
int port = connectionInfo.getScheme().equalsIgnoreCase("https") ? DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT;
After the commit the the x_forwarded_host Header (which is missing in /cloudfoundryapplication-Request) must be present in order for the port to be set to 443
Matcher hostMatcher = FORWARDED_HOST_PATTERN.matcher(forwarded);
if (hostMatcher.find()) {
String scheme = connectionInfo.getScheme();
int port = scheme.equalsIgnoreCase("https") || scheme.equalsIgnoreCase("wss") ?
DEFAULT_HTTPS_PORT : DEFAULT_HTTP_PORT;
connectionInfo = connectionInfo.withHostAddress(
AddressUtils.parseAddress(hostMatcher.group(1), port, DEFAULT_FORWARDED_HEADER_VALIDATION));
}
Comment From: wilkinsona
Thanks for tracking this down, @boborperry. Closing in favor of the Reactor Netty issue.