I am using spring boot version 2.6.6 in kubernetes cluster. Pods are randomly getting killed due liveness probe failure. Upon checking application logs, I could see following message:
{"timestamp":1669108392970,"thread":"reactor-http-epoll-7","logger":"org.springframework.web.server.adapter.HttpWebHandlerAdapter","message":"[de68ce89-95572] Error [java.lang.UnsupportedOperationException] for HTTP GET \"/requestType16/health/\", but ServerHttpResponse already committed (200 OK)","context":"default","log_level":"ERROR","is_logging_enabled":false,"service_name":"service_temp","logger_id":"23e79bf4-c86b-4f27-85a3-5f76233335cc"}
Java version: 17 Spring boot: 2.6.6 Other spring dependency:
implementation 'org.springframework.boot:spring-boot-starter-webflux'
implementation 'org.springframework.kafka:spring-kafka'
implementation 'org.springframework.boot:spring-boot-starter-actuator'
Comment From: wilkinsona
Thanks for the report. Unfortunately, you haven't provided enough information for us to help you. It looks like you may have a health indicator that's failing with an UnsupportedOperationException but we can't tell if that's the case or which indicator is failing. Please try the latest Spring Boot 2.6.x release. If the problem still occurs and you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.
Comment From: subham611
these are the endpoint exposed
management.endpoints.web.base-path=/
management.endpoints.web.exposure.include=health,prometheus,info
livenessProbePath is /health/liveness, which gets queries every 20sec
Comment From: wilkinsona
I'm afraid that doesn't really help as we still have no detailed information about the failure. Please provide the requested sample and we can take another look.
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: spring-projects-issues
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.
Comment From: orxan-hasanli
Getting the same exception in Spring Boot 3.1.6 version Error [java.lang.UnsupportedOperationException] for HTTP GET "/actuator/health", but ServerHttpResponse already committed (200 OK)
Comment From: wilkinsona
@orxan-hasanli please read the rest of the issue. If you can provide the information that we need to diagnose the problem, we can take another look.
Comment From: sergeypavlikhin
@wilkinsona Hello, I've faced the same issue:
[04016861-43] Error [reactor.netty.channel.AbortedException: io.netty.channel.StacklessClosedChannelException] for HTTP GET "/sys/health", but ServerHttpResponse already committed (200 OK)
[b27093ba-3] Error [reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation] for HTTP GET "/sys/health", but ServerHttpResponse already committed (200 OK)
it's difficult to reproduce this error via sample project, because it's a floating issue. I also checked other metrics in my application and I see that there was high load moment, so it could be connected with resources like memory, network, etc. Spring Boot 3.2.0.
implementation("org.springframework.boot:spring-boot-starter-webflux")
implementation("org.springframework.boot:spring-boot-starter-actuator")
which details do you need? I will try to provide it.
Comment From: wilkinsona
As I said above, we'll need a sample that reproduces the problem to be able to investigate. FWIW, at least the first of the errors that you're seeing is likely to be out of the server's control. It appears that the client has closed the connection before the server had a chance to respond.
Comment From: herodotus-ecosystem
@wilkinsona
I aslo getting the same exception in Spring Boot 3.3.1· version
my project use Spring Boot 3.3.1 + Spring Cloud 2023.0.2 + SAS 1.3.1.
java.lang.UnsupportedOperationException: null
at org.springframework.http.ReadOnlyHttpHeaders.set(ReadOnlyHttpHeaders.java:110) ~[spring-web-6.1.10.jar:6.1.10]
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.MonoFlatMap] :
reactor.core.publisher.Mono.flatMap(Mono.java:3171)
org.springframework.http.codec.EncoderHttpMessageWriter.write(EncoderHttpMessageWriter.java:133)
Error has been observed at the following site(s):
*______Mono.flatMap ⇢ at org.springframework.http.codec.EncoderHttpMessageWriter.write(EncoderHttpMessageWriter.java:133)
|_ Mono.doOnDiscard ⇢ at org.springframework.http.codec.EncoderHttpMessageWriter.write(EncoderHttpMessageWriter.java:139)
*______Mono.flatMap ⇢ at org.springframework.web.reactive.result.method.annotation.ResponseEntityResultHandler.handleResult(ResponseEntityResultHandler.java:137)
|_ checkpoint ⇢ Handler Actuator web endpoint 'health' [DispatcherHandler]
Original Stack Trace:
at org.springframework.http.ReadOnlyHttpHeaders.set(ReadOnlyHttpHeaders.java:110) ~[spring-web-6.1.10.jar:6.1.10]
at org.springframework.http.HttpHeaders.setContentLength(HttpHeaders.java:967) ~[spring-web-6.1.10.jar:6.1.10]
at org.springframework.http.codec.EncoderHttpMessageWriter.lambda$write$1(EncoderHttpMessageWriter.java:135) ~[spring-web-6.1.10.jar:6.1.10]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2864) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2573) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:164) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoSingle$SingleSubscriber.doOnRequest(MonoSingle.java:103) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$MonoInnerProducerBase.request(Operators.java:2931) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoSingle$SingleSubscriber.onSubscribe(MonoSingle.java:115) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:68) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:57) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:245) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:305) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:122) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2097) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:850) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:612) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.innerComplete(FluxFlatMap.java:898) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxFlatMap$FlatMapInner.onComplete(FluxFlatMap.java:1001) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoZip$ZipCoordinator.signal(MonoZip.java:298) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.MonoZip$ZipInner.onNext(MonoZip.java:478) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.publisher.FluxSubscribeOnCallable$CallableSubscribeOnSubscription.run(FluxSubscribeOnCallable.java:252) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.6.7.jar:3.6.7]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.6.7.jar:3.6.7]
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:317) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
I tried to build a simple project to reproduce this problem, but a simple project cannot reproduce it.
But I found that the problem lies in the following code：
// org.springframework.http.codec.EncoderHttpMessageWriter
@Override
public Mono<Void> write(Publisher<? extends T> inputStream, ResolvableType elementType,
@Nullable MediaType mediaType, ReactiveHttpOutputMessage message, Map<String, Object> hints) {
MediaType contentType = updateContentType(message, mediaType);
Flux<DataBuffer> body = this.encoder.encode(
inputStream, message.bufferFactory(), elementType, contentType, hints);
if (inputStream instanceof Mono) {
return body
.singleOrEmpty()
.switchIfEmpty(Mono.defer(() -> {
message.getHeaders().setContentLength(0);
return message.setComplete().then(Mono.empty());
}))
.flatMap(buffer -> {
Hints.touchDataBuffer(buffer, hints, logger);
message.getHeaders().setContentLength(buffer.readableByteCount());
return message.writeWith(Mono.just(buffer)
.doOnDiscard(DataBuffer.class, DataBufferUtils::release));
})
.doOnDiscard(DataBuffer.class, DataBufferUtils::release);
}
if (isStreamingMediaType(contentType)) {
return message.writeAndFlushWith(body.map(buffer -> {
Hints.touchDataBuffer(buffer, hints, logger);
return Mono.just(buffer).doOnDiscard(DataBuffer.class, DataBufferUtils::release);
}));
}
if (logger.isDebugEnabled()) {
body = body.doOnNext(buffer -> Hints.touchDataBuffer(buffer, hints, logger));
}
return message.writeWith(body);
}
The key code is：
message.getHeaders().setContentLength(buffer.readableByteCount());
Although I haven't found the specific reason yet, by comparing the simple project I built with my own project, I found that the main reason for the error is
In the simple project I built，request HTTP GET "/actuator/health"，The state of the message（ReactiveHttpOutputMessage）in the above code is NEW，So message. getHeaders () returns an HttpHeaders object, setContentLength(buffer.readableByteCount()) will not make any errors。
See the code below
// org.springframework.http.server.reactive.AbstractServerHttpResponse
@Override
public HttpHeaders getHeaders() {
if (this.readOnlyHeaders != null) {
return this.readOnlyHeaders;
}
else if (this.state.get() == State.COMMITTED) {
this.readOnlyHeaders = HttpHeaders.readOnlyHttpHeaders(this.headers);
return this.readOnlyHeaders;
}
else {
return this.headers;
}
}
In my own project（Spring Boot 3.3.1 + Spring Cloud 2023.0.2 + SAS 1.3.1），request HTTP GET "/actuator/health"，The state of the message（ReactiveHttpOutputMessage）in the above code is COMMITTED。So message. getHeaders () returns ReadOnlyHttpHeaders object, then setContentLength(buffer.readableByteCount()) will make UnsupportedOperationException errors
Comment From: bclozel
@herodotus-ecosystem this means the error happens while the response is being written to. Since headers are already committed there is no way to change some parts of the response.
This does not point to a Spring issue as it stands. Maybe a custom actuator endpoint is causing a problem? Without a minimal sample to analyze we can't help here.
Comment From: herodotus-ecosystem
@herodotus-ecosystem this means the error happens while the response is being written to. Since headers are already committed there is no way to change some parts of the response.
This does not point to a Spring issue as it stands. Maybe a custom actuator endpoint is causing a problem? Without a minimal sample to analyze we can't help here.
Thank you for your reply. Based on your ideas, I will take a look at the code again to see if I can find the cause of the problem.