BufferingClientHttpRequestFactory creates requests which are not repeatable

A RestTemplate obtained form the following snippet will lose the default request configuration applied to the HttpComponentsClientHttpRequestFactory, in this case the setRedirectsEnabled(true) is not honored.

  • Affected Version: spring-boot 3.2.0-M3
  • Works with stable: spring-boot 3.1.4
  • Affected http client library is Apache HttpComponents 5 (HttpComponentsClientHttpRequestFactory).

I prepared a reproducible example here. Just run com.example.httpclient5demo.Httpclient5DemoApplicationTests

        RestTemplate restTemplate = restTemplateBuilder
                .requestFactory(() -> {
                    RequestConfig.Builder requestConfig = RequestConfig.custom()
                            .setRedirectsEnabled(true);
                    HttpClientBuilder httpClientBuilder = HttpClientBuilder.create()
                            .setDefaultRequestConfig(requestConfig.build());
                    HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClientBuilder.build());
                    //return requestFactory;
                    return new BufferingClientHttpRequestFactory(requestFactory);
                })
                .rootUri("https://httpbin.org")
                .build();
        String response = restTemplate.getForObject("/redirect-to?url=https://httpbin.org/base64/SFRUUEJJTiBpcyBhd2Vzb21l", String.class);
        assertThat(response).isEqualTo("HTTPBIN is awesome");

Comment From: wilkinsona

Thanks for the report and sample project. This doesn't appear to be a Spring Boot problem. The following tests illustrate the behavior without using Spring Boot:

package com.example.httpclient5demo;

import static org.assertj.core.api.Assertions.assertThat;

import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.impl.classic.HttpClientBuilder;
import org.junit.jupiter.api.Test;
import org.springframework.http.client.BufferingClientHttpRequestFactory;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.web.client.RestTemplate;

class Httpclient5DemoApplicationTests {

    @Test
    void redirectsCanBeEnabled() {
        RestTemplate restTemplate = new RestTemplate();
        RequestConfig.Builder requestConfig = RequestConfig.custom()
                .setRedirectsEnabled(true);
        HttpClientBuilder httpClientBuilder = HttpClientBuilder.create()
                .setDefaultRequestConfig(requestConfig.build());
        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClientBuilder.build());
        restTemplate.setRequestFactory(requestFactory);
        String response = restTemplate.getForObject("https://httpbin.org/redirect-to?url=https://httpbin.org/base64/SFRUUEJJTiBpcyBhd2Vzb21l", String.class);
        assertThat(response).isEqualTo("HTTPBIN is awesome");
    }

    @Test
    void redirectsCanBeEnabledWhenBuffering() {
        RestTemplate restTemplate = new RestTemplate();
        RequestConfig.Builder requestConfig = RequestConfig.custom()
                .setRedirectsEnabled(true);
        HttpClientBuilder httpClientBuilder = HttpClientBuilder.create()
                .setDefaultRequestConfig(requestConfig.build());
        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClientBuilder.build());
        restTemplate.setRequestFactory(new BufferingClientHttpRequestFactory(requestFactory));
        String response = restTemplate.getForObject("https://httpbin.org/redirect-to?url=https://httpbin.org/base64/SFRUUEJJTiBpcyBhd2Vzb21l", String.class);
        assertThat(response).isEqualTo("HTTPBIN is awesome");
    }

}

redirectsCanBeEnabled passes but redirectsCanBeEnabledWhenBuffering fails. We'll transfer this to the Framework team so that they can take a look.

Comment From: cachescrubber

Thank you @wilkinsona for analyzing the issue. I updated the example with your test cases.

Comment From: cachescrubber

I spotted another issue. Basic Auth (401) and Proxy Auth (407) Challenges are not working when buffering is enabled.

    @Test
    void basicAuthCanBeEnabledWhenBuffering() {
        RestTemplate restTemplate = new RestTemplate();
        BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();

        credentialsProvider.setCredentials(new AuthScope("httpbin.org", 443),
                new UsernamePasswordCredentials("demo", "secret".toCharArray()));

        HttpClientBuilder httpClientBuilder = HttpClientBuilder.create()
                .setDefaultCredentialsProvider(credentialsProvider);

        HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClientBuilder.build());
        restTemplate.setRequestFactory(new BufferingClientHttpRequestFactory(requestFactory));
        ResponseEntity<String> forEntity = restTemplate.getForEntity("https://httpbin.org//basic-auth/demo/secret", String.class);
        assertThat(forEntity.getStatusCode().is2xxSuccessful()).isTrue();
    }

The example has been updated with a test case.

Comment From: cachescrubber

The issue is not a configuration issue, as I assumed initialy. I'll change the issue title accordingly.

It is caused by changes introduced in #30557 . BufferingClientHttpRequestWrapper is creating an empty body, which is passed to HttpComponents as org.springframework.http.client.HttpComponentsClientHttpRequest.BodyEntity which in turn implements org.apache.hc.core5.http.HttpEntity#isRepeatable to return false unconditionally.

HttpEntity#isRepeatable returning false disables all kind of functionality in the library where requests are repeated - redirects (3xx), authentication challenges (401,407) but also internal retries.

A quick fix would be to not pass in empty HttpComponentsClientHttpRequest.BodyEntity for requests where no body is expected (GET, HEAD, ...).

But I think the changes in #30557 are problematic in this regard. As it is implemented right now, even requests using the BufferingClientHttpRequestFactory are converted into a streaming body, even though they are in-memory anyway.

Comment From: cachescrubber

Thanks @poutsma , my HttpComponents based test are good again. From reading the code I think at least OkHttp3ClientHttpRequest.BodyRequestBody#isOneShot need to be updated as well. Not sure for the other implementations.

Comment From: poutsma

Thanks for spotting that, it should now be fixed.

Comment From: poutsma

@cachescrubber I made some further related changes in 6dd93d4, which should enable repeatable writes for many types of request bodies, meaning that you typically don't need to wrap your request factory in a BufferingClientHttpRequestFactory any more. Please try a 6.1.0 Spring Framework snapshot and see if it works for you.

Comment From: cachescrubber

Thanks for the heads-up. I use BufferingClientHttpRequestFactory to implement request/response logging in a ClientHttpRequestInterceptor. IIRC we needed the buffering in order to use response#getBody multiple times.

Comment From: karlovskiy

Hi, I found this issue today during my investigation of redirects which stopped working after spring update. I found that in this line we have streamingOutputMessage.setBody(outputStream -> StreamUtils.copy(body, outputStream)); but if body was set like this with lambda than repeatable will be false. Is it intended behavior ? I thought that if we have byte array in InterceptingClientHttpRequest maybe body there also should be repeatable ?

PS: Currently, I resolved my redirect issue with wrapping HttpComponentsClientHttpRequestFactory with BufferingClientHttpRequestFactory.

Comment From: poutsma

@karlovskiy well spotted, this should be fixed in 6.1.4.