SpringBoot Multipart Upload Issue After Spring Boot 3.3.5 Upgrade – Connection Closed by Peer

Hello Experts,

We’ve encountered an issue with multipart/form-data uploads in our microservices architecture after upgrading from Spring Boot 3.3.2 (where uploads worked as expected) to Spring Boot 3.3.5 and Spring Cloud 2024.0.3. Below is a quick overview:

Architecture:

Service A: Receives the multipart uploads. Cloud Gateway: Acts as a proxy Service A, forwarding requests. Service B: Uses Reactive Web Client to upload files to another service.

Evidence:

Small files (e.g., 10 records) upload successfully using service B reactive web client and failed with large size files.
The same large files upload successfully via Postman to the Service A proxy endpoint

Problem:

Service A log: Since the upgrade, Service A throws EOFException when handling larger multipart file uploads

Service B log:

org.springframework.web.reactive.function.client.WebClientRequestException: Connection closed by peer
    at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.1.14.jar:6.1.14]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ Request to POST https://serviceUrl/import [DefaultWebClient]
Original Stack Trace:
        at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:136) ~[spring-webflux-6.1.14.jar:6.1.14]
        at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.Mono.subscribe(Mono.java:4576) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:280) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:205) ~[reactor-core-3.6.11.jar:3.6.11]
        at org.springframework.http.client.reactive.HttpComponentsClientHttpConnector$ResponseCallback.failed(HttpComponentsClientHttpConnector.java:161) ~[spring-web-6.1.14.jar:6.1.14]
        at org.apache.hc.core5.concurrent.BasicFuture.failed(BasicFuture.java:138) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactive.ReactiveResponseConsumer.failed(ReactiveResponseConsumer.java:141) ~[httpcore5-reactive-5.2.5.jar:5.2.5]
        at org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2.failed(InternalAbstractHttpAsyncClient.java:347) ~[httpclient5-5.3.1.jar:5.3.1]
        at org.apache.hc.client5.http.impl.async.AsyncRedirectExec$1.failed(AsyncRedirectExec.java:248) ~[httpclient5-5.3.1.jar:5.3.1]
        at org.apache.hc.client5.http.impl.async.AsyncProtocolExec$1.failed(AsyncProtocolExec.java:295) ~[httpclient5-5.3.1.jar:5.3.1]
        at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.failed(HttpAsyncMainClientExec.java:131) ~[httpclient5-5.3.1.jar:5.3.1]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.failed(ClientHttp1StreamHandler.java:285) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.terminate(ClientHttp1StreamDuplexer.java:193) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.shutdownSession(AbstractHttp1StreamDuplexer.java:163) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:349) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:633) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSession.java:74) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:202) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86) ~[httpcore5-5.2.5.jar:5.2.5]
        at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44) ~[httpcore5-5.2.5.jar:5.2.5]
        at java.base/java.lang.Thread.run(Thread.java:1570) [?:?]
    Suppressed: java.lang.Exception: #block terminated with an error
        at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:104) ~[reactor-core-3.6.11.jar:3.6.11]
        at reactor.core.publisher.Mono.block(Mono.java:1779) ~[reactor-core-3.6.11.jar:3.6.11]
        at se.cambio.eco.ts.terminology.TerminologyClient.importCodeSystem$lambda$16(TerminologyClient.kt:143) ~[main/:?]
        at se.cambio.eco.ts.terminology.TerminologyClient.wrap(TerminologyClient.kt:36) ~[main/:?]
        at se.cambio.eco.ts.terminology.TerminologyClient.importCodeSystem(TerminologyClient.kt:129) ~[main/:?]
        at se.cambio.eco.ts.workflow.CodeSystemImport.processCodeSystemImport(CodeSystemImport.kt:131) ~[main/:?]
        at se.cambio.eco.ts.workflow.CodeSystemImport.createCodeSystem(CodeSystemImport.kt:76) ~[main/:?]
        at se.cambio.eco.ts.queue.JobScheduler$handleJobs$1.invokeSuspend(JobScheduler.kt:53) ~[main/:?]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith$$$capture(ContinuationImpl.kt:33) ~[kotlin-stdlib-2.0.0.jar:2.0.0-release-341]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt) ~[kotlin-stdlib-2.0.0.jar:2.0.0-release-341]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.8.1.jar:?]
        at se.cambio.eco.ts.queue.JobScheduler.handleJobs(JobScheduler.kt:38) ~[main/:?]
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) ~[spring-context-6.1.14.jar:6.1.14]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.14.jar:6.1.14]
        at io.micrometer.observation.Observation.observe(Observation.java:499) ~[micrometer-observation-1.13.6.jar:1.13.6]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.14.jar:6.1.14]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.1.14.jar:6.1.14]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:358) ~[?:?]
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[?:?]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1570) [?:?]
Caused by: org.apache.hc.core5.http.ConnectionClosedException: Connection closed by peer
    at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:349) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:633) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSession.java:74) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:202) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86) ~[httpcore5-5.2.5.jar:5.2.5]
    at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44) ~[httpcore5-5.2.5.jar:5.2.5]
    at java.base/java.lang.Thread.run(Thread.java:1570) [?:?]

Could you provide any insights or guidance on resolving this?

Comment From: mhalbritter

Are you sure this is the fault of Service A or could the Spring Cloud Gateway be at fault, too? You could try the direct connection and see if the problem is still there.

Might be related to https://github.com/spring-projects/spring-boot/issues/42940.

If you're sure this is an issue in Spring Boot, then please provide a reproducer and we'll reopen this issue.