Describe the bug Updating from Spring Boot 3.3.6 to Spring Boot 3.4, has lead to Feign returning 403/FORBIDDEN for every Feign call, for every FeignClient call.
Spring Boot 3.4.0 Spring Cloud Dependencies: 2024.0.0 Spring Cloud OpenFeign: 4.2.0 Apache Http Client5: 5.13.5
Rolling back to previous SB 3.3.6 version: Spring Boot 3.3.6 Spring Cloud Dependencies: 2023.0.4 Spring Cloud OpenFeign: 4.1.4 Apache Http Client5: 5.3.1
gives:
No change to the component being called by Feign - it's still deployed/processing/unchanged. There are no error messages/exceptions in either component ie either the caller or callee - there is nothing to suggest any issue.
application.yml:
spring:
cloud:
openfeign:
compression:
request:
enabled: true
response:
enabled: true
Has this been seen before? Any further troubleshooting steps I can do?
Thanks.
Comment From: OlgaMaciaszek
Hello, @stephenmontgomery, thanks for reporting the issue. Please provide a minimal, complete, verifiable example that reproduces the issue.
Comment From: stephenmontgomery
Hi @OlgaMaciaszek, Yeah can't do that, unfortunately - it's only reproducible in our deployed environments - k8s with istio + envoy. Works fine with the same components deployed in my local docker compose.
Just tried https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy without any luck.
Comment From: OlgaMaciaszek
Hi @stephenmontgomery, there's only a single issue that comes to mind at this point that could be related in terms of changing the way the request is created: https://github.com/spring-cloud/spring-cloud-openfeign/issues/1070 - you can see if those upstream changes are not affecting you and change the prop if they are. If this is not the case, you may want to enable full logging on the client side in SC OF (https://docs.spring.io/spring-cloud-openfeign/reference/spring-cloud-openfeign.html#spring-cloud-feign-overriding-defaults) on both versions and compare the requests to see what's changed.
Comment From: stephenmontgomery
Hi @OlgaMaciaszek,
We do have that configurer.setUseTrailingSlashMatch(true) SB 3 workaround littered everywhere so prob not it but I'll double-check. There def doesn't seem to be any trailing slash in the Feign logs above.
Bit further on here though, with some of our Operations team's input - they found something similar with https://github.com/istio/istio/issues/53239 - outlines the 403 behaviour even though using http and not https. As mentioned above, I did try https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy - which should have workedaround this issue but it had no effect so I'm wondering if the default config didn't take.
I'm also considering downgrading the HC5 version. As per https://github.com/spring-projects/spring-boot/issues/43139, this TLS stuff came in HC5 5.4. As mentioned above, the last SB 3.3.6 version used HC5 5.3.1
Comment From: OlgaMaciaszek
I don't have a better suggestion than for you to compare the full requests on both versions. Once you identify the issue (i.e. what has changed and is breaking you, we can take a look at how to resolve it).
Comment From: stephenmontgomery
Sorry @OlgaMaciaszek - I thought I have given the full Feign request logging. Is there additional debug logging I can add?
Comment From: OlgaMaciaszek
I don't see any information about headers, for example. Some other request details might be missing. Also, we need to have the full log from both versions (the working one and then not working one) to compare for differences. Please make sure to add a Logger.Level bean in your Feign client configuration to return Logger.Level.FULL (as described in the linked doc) and make sure to set the logging.level for the package where the Feign Client is to DEBUG. Then send provide full request logs from the working and not working requests. Please provide them in text/log form and not as screenshots.
Comment From: stephenmontgomery
Ok @OlgaMaciaszek worth checking but I confirm FULL feign logging was enabled and headers are seen (request + response) in the screenshots above. Can't give u actual text because accessible from DataDog and I'll have to search over again.
logging:
level:
root: INFO
com.zzzzzzz: DEBUG
com.zzzzzzz.zzzzz.api.client: DEBUG
spring:
cloud:
openfeign:
client:
config:
default:
logger-level: FULL
Anyho just making sure I'm not missing any extra diagnostic debug.
Comment From: OlgaMaciaszek
That should be fine. If it takes effect, you should be easily able to spot it in the logs, as it's quite verbose.
Comment From: jaccarte
Also seeing this behaviour. Having enabled logging in the HTTP client it looks like the suggested fix (to disable the protocol upgrade) isn't having the desired effect. The output is the same with or without the fix applied:
Strangely when we upgraded the HTTP client and applied a similar fix in the previous Spring Boot it worked fine.
Comment From: OlgaMaciaszek
@jaccarte Could you please do this: https://github.com/spring-cloud/spring-cloud-openfeign/issues/1141#issuecomment-2538720051 and provide request info for comparison?
Comment From: jaccarte
I did FULL request logging with and without the fix and it was exactly the same (no mention of the connection upgrade), hence why I enabled logging in the relevant hc5 class which deals with the actual connection upgrade. It showed the logs above even when I had supposedly disabled the automatic connection upgrade (those logs should only appear when the feature is enabled).
Anyhow I found a working solution by adding the following to my configuration. I guess the other fix isn't touching the Feign configuration for some reason (or maybe it's not intended to?).
@Bean
public HttpClient5FeignConfiguration.HttpClientBuilderCustomizer httpClientBuilder() {
return (httpClientBuilder) -> {
var rcBuilder = RequestConfig.custom();
rcBuilder.setProtocolUpgradeEnabled(false);
httpClientBuilder.setDefaultRequestConfig(rcBuilder.build());
};
}
Comment From: OlgaMaciaszek
@jaccarte thanks for the update and I'm happy there's a workaround.
when I had supposedly disabled the automatic connection upgrade - can you please let me know in which way you set it up in your app? Could you also please provide a minimal, complete, verifiable example that reproduces the issue?
Comment From: OlgaMaciaszek
I see now this is related to https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.4-Release-Notes#apache-http-components-and-envoy. Will look into creating a better way of handling this.
Comment From: jaccarte
FYI the issue is can be easily reproducible whenever a plain non-encrypted HTTP call is made using a Feign client that hits Istio. Istio, instead of ignoring or upgrading the request to upgrade the connection to be encrypted, simply refuses the request and returns 403.
Comment From: OlgaMaciaszek
Upon some more consideration, we'll be handling it similarly to how it's been done in Spring Boot: by adding an example of how to set this using the available Customizer mechanism, as in the workaround provided by @jaccarte.
Comment From: stephenmontgomery
Hi @OlgaMaciaszek, This doesn't seem an optimal solution IMO - as a dev, doing the SB 3.4 upgrade, i'd expect to read the SB release notes, apply the SB fix at https://contrast.atlassian.net/browse/SCAN-5700 and have openfeign automatically apply @jaccarte workaround.
Why do I have to apply 2 workarounds (SB and Openfeign) for Istio/Envoy? Is there any point of me even applying the SB 3.4 workaround? I don't really want to stick in these 2 temporary workarounds into our microservices as these type of things tend to hang around forever and will probably make the next SB upgrade problematic.... I suspect Istio/HC5/Envoy 3rd parties will also be updated to address these issues too.
Just my 2 cents...
Comment From: OlgaMaciaszek
Hello @stephenmontgomery, that's a valid point. However, since Spring Cloud OpenFeign is in maintenance mode only (as documented, we're encouraging users to move over to Spring Interface Clients) and this would require more changes, we're not planning to do it at this point.