Spring boot 2.3.7 Spring Cloud Hoxton SR8 Java 11
We are trying out Istio service mesh but when we deploy the spring boot app actuator endpoint returns incorrect port number in href or no port number.
We have configured management.server.port: 8081in bootstrap.yaml so we can reach actuator at this port.
So naturally when you go to http://10.0.0.1:8081/actuator/ it returns all the end point but with one issue
it doesnt include the configured port number. And ofcourse this happens only when Istio side car container is present
{
"_links": {
"self": {
"href": "http://10.0.0.1/actuator",
"templated": false
},
"health-path": {
"href": "http://10.0.0.1/actuator/health/{*path}",
"templated": true
},
"health": {
"href": "http://10.0.0.1/actuator/health",
"templated": false
},
"info": {
"href": "http://10.0.0.1/actuator/info",
"templated": false
}
}
}
Comment From: wilkinsona
The hrefs in the links are derived from the URL of the incoming request. If Istio is proxying requests, it should set forwarded headers that Boot must be configured to consume using the server.forward-headers-strategy property. Can you please check that you have configured your application as required? If you believe you have done so, please verify that Istio is adding the necessary headers when forwarding requests to the application. If neither of those helps to resolve the problem and you would like us to spend some more time investigating, we will require a minimal sample the reproduces the problem to be able to do so.
Comment From: sabareeshkkanan
@wilkinsona Appreciate pointing me on this direction but a quick change from using web to webflux seems to resolve the problem. Which indicates there is something specific to servlet mvc is causing issue.
Comment From: sabareeshkkanan
But the problem is we cant migrate all the web service to webflux yet.
Comment From: wilkinsona
WebFlux uses ForwardedHeaderTransformer to apply X-Forwarded headers to the request URL, but it's only used when server.forward-headers-strategy is set to framework. There's a similar class on the Servlet side. There's also the option of using the embedded container's built-in X-Forwarded headers processing. This will be in effect if server.forward-headers-strategy is set to native or you're running in an environment that's detected as a cloud platform.
In short, swapping from the Servlet based MVC to WebFlux may have changed the framework code that honours X-Forwarded header or the container code that does so. As I said above, if you would like us to spend some more time investigating, we will require a minimal sample that reproduces the problem to be able to do so.
Comment From: sabareeshkkanan
@wilkinsona here is the sample code. https://github.com/sabareeshkkanan/istio-debug , just switching to webflux fixes the issue. Also webflux works even without server.forward-headers-strategy: native
Comment From: philwebb
@sabareeshkkanan Can update the sample with a readme that describes how we should run the application with Istio? The application itself is only one half of the equation.
Comment From: sabareeshkkanan
@philwebb @wilkinsona i have added documentation, please let me know if you have more questions
Comment From: wilkinsona
Also webflux works even without
server.forward-headers-strategy: native
That's to be expected when deployed to K8S as it'll be detected as a cloud platform where the use of forward headers is enabled by default.
It sounds like the difference in behaviour is due to a difference in X-Forwarded headers handling between Tomcat and Netty. Hopefully we can use your sample to confirm that's the case.
Comment From: wilkinsona
As suspected the problem is with the X-Forwarded- headers that Envoy adds and how Tomcat handles them. Here's an example of the headers in a request as seen by Tomcat:
host = 10.111.130.85:8081
user-agent = curl/7.64.1
accept = */*
x-forwarded-proto = http
x-request-id = e6373443-1668-4fcd-9bc1-7de20e6f4683
x-b3-traceid = df1a38ef71f0cec9b8ce2605691fb12f
x-b3-spanid = b8ce2605691fb12f
x-b3-sampled = 0
content-length = 0
As indicated by the host header, the request has been received on port 8081. Envoy has indicated, via the x-forwarded-proto header, that this is an http request. In the absence of an x-forwarded-port header, Tomcat's RemoteIPValve assumes that the proxy forwarding the request will use port 80 so this is the port that it configures on the request. There's an open Envoy issue to add the port header that would tell Tomcat that a non-standard HTTP port is being used.
By contrast, Reactor Netty's DefaultHttpForwardedHeaderHandler doesn't change the port when only X-Forwarded-Proto is set. This suits your purposes but would not work for a proxy that expects http to use port 80 by default and https to use port 443 by default.
In this particular case, I think the X-Forwarded-Proto header is causing more harm than good. Without an accompanying X-Forwarded-Port header, the information it provides is potentially misleading and, given that everything is being done over HTTP anyway, the information isn't adding any value. This means that you can avoid the problem by disabling handling of forward headers by setting server.forward-headers-strategy: none:
$ curl http://10.111.130.85:8081/actuator
{"_links":{"self":{"href":"http://10.111.130.85:8081/actuator","templated":false},"health":{"href":"http://10.111.130.85:8081/actuator/health","templated":false},"health-path":{"href":"http://10.111.130.85:8081/actuator/health/{*path}","templated":true},"info":{"href":"http://10.111.130.85:8081/actuator/info","templated":false}}}
I'm going to close this issue as there's nothing we can do in Spring Boot to improve the situation. I would recommend subscribing to the Envoy issue so you can keep track of its progress.
Comment From: sabareeshkkanan
If any one stumbles on this issue, it appears you can fix it simply by using undertow as webserver as temporary fix, until convoy proxy fixes it up.
Comment From: wilkinsona
Thanks for sharing your finding about Undertow, @sabareeshkkanan. Undertow's ProxyPeerAddressHandler is similar to Reactor Netty's DefaultHttpForwardedHeaderHandler in that the X-Forwarded-Proto header has no effect on the request's port.
you can fix it simply by using undertow as webserver as temporary fix
While swapping container may be simple in that it only requires a small change in your pom.xml or build.gradle, it can have quite far-reaching consequences. Anyone else hitting this issue may want to consider setting server.forward-headers-strategy: none as a more focussed workaround that shouldn't bring any unexpected side-effects with it.
Comment From: sabareeshkkanan
I agree @wilkinsona but there is a case we dont want disable server.forward-headers-strategy , for that only solution is to migrate to webflux or undertow.
Comment From: sherifkayad
@wilkinsona @sabareeshkkanan I am stuck with the same issue since some time .. Unfortunately disabling the Forward Headers completely using server.forward-headers-strategy: none won't help us as we have some other cases for the same service where the proxy host and port are needed e.g. running the service behind reverse proxy. Are there any other solutions e.g. forcing Istio to add the forwarded port or whatever?
Comment From: abhi93104
Facing same issue, I resolved this by adding envoy filter for actuator calls. Check my answer here: https://github.com/envoyproxy/envoy/issues/3732#issuecomment-1178745033
Comment From: marcpintoo
This is what I did. I use a dedicated port for spring / controllers. So across the board - I set x-forwarded-port header when http requests are sent to a specific service + port.
Hope this helps
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: app-actuator
namespace: istio-system
spec:
hosts:
- "app.namespace.svc.cluster.local"
http:
- match:
- port: 1234
route:
- destination:
host: app.namespace.svc.cluster.local
port:
number: 1234
headers:
request:
set:
x-forwarded-port: "1234"