Affects: Spring Boot 2.4.2
Hi, I've noticed that multipart/form-data does not include a content length when the request is created with multipart/form-data . This has caused an issue with trying to access a route that requires a content-length to be set or else I get a 411. I've created a small demo to recreate the problem which can found here https://github.com/JoeBalduz/spring-webclient-multipart-no-content-length.
I also saw that #26046 appears to be the same issue as mine but it is closed. Is this possibly something that can be fixed in a future version so that the content-length is set automatically for multipart/form-data? I've tried setting the content-length myself in multiple ways but the closest I got was being off by about 10 bytes.
Comment From: rstoyanchev
It is not easy to compute a content-length. It requires serializing and buffering the entire content in memory, which is hardly ideal for a multipart request in particular. Granted we may know the length of a Resource (e.g. a file), but we'd still need to serialize headers, multipart boundaries, and all other parts with their content to know the content-length of the entire request.
Can you elaborate on "trying to access a route that requires a content-length"? For a multipart request that doesn't seem like a reasonable requirement.
Keep in mind also that we do set the content-length on individual parts where the length is know. For example for a file Resource I would expect the part headers to have a content-length. That is a more feasible expectation I think.
Comment From: JoeBalduz
There's an API route I need to access where if the Content-Length header isn't set on the http request, I get a 411 error and I'm not able to upload my file, along with a string, to that route. I don't control that API, so I've had to switch to using RestTemplate in order to send that request however I'd prefer to not use RestTemplate since it is blocking.
Comment From: rstoyanchev
The FormHttpMessageConverter also doesn't set the content-length for a multipart request. Can you elaborate?
Comment From: JoeBalduz
I'm not exactly sure where the content-length header is being set when I use RestTemplate but I can see in wireshark that it has a content-length header and I'm no longer getting the 411 error when sending a request to the API I'm accessing and the request is working. Here is how I'm using RestTemplate:
val headers = HttpHeaders()
headers.contentType = MediaType.MULTIPART_FORM_DATA
val httpEntity = HttpEntity(multipartBodyBuilder.build(), headers)
val response = restTemplate.postForEntity("url/im/accessing", httpEntity, String::class.java)
Comment From: JoeBalduz
I can also update the demo I have to add RestTemplate as one of the choices if that would help.
Comment From: rstoyanchev
Yes that would be great, thanks.
Comment From: JoeBalduz
The demo has been updated with a route that uses RestTemplate
Comment From: rstoyanchev
Thanks. Indeed the RestTemplate applies BufferingClientHttpRequestWrapper which buffers the entire request body which might require quite a bit of memory depending on the overall size and the number of concurrent requests.
The same can be achieved with the WebClient also by buffering. We don't have anything out-of-the-box but you can write one. I tested with the below for example:
class BufferingFilter : ExchangeFilterFunction {
override fun filter(request: ClientRequest, next: ExchangeFunction): Mono<ClientResponse> {
return next.exchange(ClientRequest.from(request)
.body { message: ClientHttpRequest?, context: BodyInserter.Context? -> request.body().insert(BufferingDecorator(message), context!!) }
.build())
}
private class BufferingDecorator(delegate: ClientHttpRequest?) : ClientHttpRequestDecorator(delegate!!) {
override fun writeWith(body: Publisher<out DataBuffer>): Mono<Void> {
return DataBufferUtils.join(body)
.flatMap { dataBuffer: DataBuffer ->
val length = dataBuffer.readableByteCount()
headers.contentLength = length.toLong()
super.writeWith(Mono.just(dataBuffer))
}
}
}
}
Comment From: JoeBalduz
Awesome, that worked. Thanks for the help!
Comment From: EstebanDugueperoux2
Hello,
I have same issue and I don't know Kotlin. Is it possible to have same filter in Java?
Best Regards.
Comment From: rstoyanchev
I suppose we should just make such a buffering ExchangeFilterFunction available. @EstebanDugueperoux2 could you create a new issue?
Comment From: nidhi-nair
@rstoyanchev I was unable to find a relevant issue for this, but have implemented an equivalent filter function in my project. May I open an issue to contribute a buffering ExchangeFilterFunction addition to this repo?
Comment From: rstoyanchev
@nidhi-nair, sure we can consider that.
Comment From: kozel-stas
Java example, I will leave it here, maybe it will help somebody.
public class MultipartExchangeFilterFunction implements ExchangeFilterFunction {
@Override
@Nonnull
public Mono<ClientResponse> filter(@Nonnull ClientRequest request, @Nonnull ExchangeFunction next) {
if (MediaType.MULTIPART_FORM_DATA.includes(request.headers().getContentType())
&& (request.method() == HttpMethod.PUT || request.method() == HttpMethod.POST)) {
return next.exchange(
ClientRequest.from(request)
.body((outputMessage, context) -> request.body().insert(new BufferingDecorator(outputMessage), context))
.build()
);
} else {
return next.exchange(request);
}
}
private static final class BufferingDecorator extends ClientHttpRequestDecorator {
private BufferingDecorator(ClientHttpRequest delegate) {
super(delegate);
}
@Override
@Nonnull
public Mono<Void> writeWith(@Nonnull Publisher<? extends DataBuffer> body) {
return DataBufferUtils.join(body).flatMap(buffer -> {
getHeaders().setContentLength(buffer.readableByteCount());
return super.writeWith(Mono.just(buffer));
});
}
}
}
Comment From: Pro456458
Hi I come across to this issue 26489 "Spring WebClient multipart/form-data does not include a content length" . Provided solution working fine and inside ExchangeFilterFunction, content-length is well calculated in case of uploading file in a single chunk. But in case of uploading file in multiple chunk (for large size) it started failing and giving exception {The length of the file input stream does not match the provided length. Expected: X, Actual (approximate): 'Y' "} Could @rstoyanchev @bclozel @sbrannen @nidhi-nair @spring-projects-issues one help me to provide the solution how to calculate the content for each chunk
Below is code snippet for reference
public Mono<Object> publishFileUsingChunks(Mono<DataBuffer> dataBufferMono, String generatedFileName, String accessToken) {
this.webClientBuild(); // initialize webclient.builder
Mono<InputStream> inputStreamMono = dataBufferMono.map(dataBuffer -> dataBuffer.asInputStream(true));
log.debug("dataBufferMono to inputStreamMono ... and the given filename :: "+ generatedFileName);
return inputStreamMono.zipWhen(
inputStream -> {
try(BufferedInputStream in = new BufferedInputStream(inputStream)) {
int chunkSize= 40485760; //40MB (approx)
byte[] bytes = new byte[chunkSize];
int length = 0;
Mono<String> fileUploadResponseMono = null;
while((length = in.read(bytes)) != -1){
log.debug("upload in chunks has been started and length:: "+ length);
byte[] inpBuf = ArrayUtils.subarray(bytes, 0, length);
Mono<String> resp = importFile(accessToken, inpBuf, generatedFileName);
if(fileUploadResponseMono !=null){
fileUploadResponseMono = fileUploadResponseMono.zipWhen(
fileUploadResponse -> resp, (fileUploadResp, response) -> response);
}else {
fileUploadResponseMono = resp;
}
}
return fileUploadResponseMono;
} catch (IOException e) {
e.printStackTrace();
throw new CMSWriteServiceException(e.getMessage());
}
},
(inputStream, uploadResponse)-> uploadResponse
);
}
private Mono<String> importFile(String accessToken, byte[] inpBuf, String generatedFileName) {
String uri=RELATIVE_URL;
MultipartBodyBuilder builder = new MultipartBodyBuilder();
builder.part("file", inpBuf);
return webClient.post().uri(uriBuilder -> uriBuilder.path(uri)
.build())
.header(HttpHeaders.AUTHORIZATION, "Bearer " +accessToken)
.header(HttpHeaders.CONTENT_TYPE, MediaType.MULTIPART_FORM_DATA_VALUE)
.bodyValue(builder.build())
.retrieve()
.onStatus(HttpStatus::isError, response ->
response.bodyToMono(String.class).flatMap(error ->
Mono.error(new Exception(" Error: --->"+ error)))
)
.bodyToMono(String.class)
.doOnSuccess( response ->
log.debug("uploaded multipart successfully:: {}", response)
)
.doOnError(WebClientResponseException.class, error ->
log.error("Error Message :: {} Error Response :: {}",error.getMessage(), error.getResponseBodyAsString())
);
}
private void webClientBuild() {
this.webClient = WebClient.builder()
.codecs(clientCodecConfigurer -> clientCodecConfigurer.defaultCodecs()
.maxInMemorySize(500000000))
.baseUrl(BASE_URL)
.filter(new MultipartExchangeFilterFunction())
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT_ENCODING, "gzip, deflate, br")
.defaultHeader("KeepAlive", "true")
.build();
}