Spring 204 No Content respoinse is broken on HTTP/2 Tomcat

Affects: 5.3.24, 6.0.3

Fail to parse response from spring application returning 204 No Content via HTTP/2. Seems only Tomcat is affected, cannot reproduce on Undertow.

Stacktrace

org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type 'application/octet-stream' not supported for bodyType=pkunk.reprospringhttp2.Dto
    at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$12(BodyExtractors.java:201) ~[spring-webflux-5.3.24.jar:5.3.24]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ Body from GET http://localhost:8080/dto [DefaultClientResponse]
Original Stack Trace:
        at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$12(BodyExtractors.java:201) ~[spring-webflux-5.3.24.jar:5.3.24]
        at java.base/java.util.Optional.orElseGet(Optional.java:364) ~[na:na]
        at org.springframework.web.reactive.function.BodyExtractors.readWithMessageReaders(BodyExtractors.java:197) ~[spring-webflux-5.3.24.jar:5.3.24]
        at org.springframework.web.reactive.function.BodyExtractors.lambda$toMono$2(BodyExtractors.java:85) ~[spring-webflux-5.3.24.jar:5.3.24]
        at org.springframework.web.reactive.function.client.DefaultClientResponse.body(DefaultClientResponse.java:131) ~[spring-webflux-5.3.24.jar:5.3.24]
        at org.springframework.web.reactive.function.client.DefaultClientResponse.bodyToMono(DefaultClientResponse.java:146) ~[spring-webflux-5.3.24.jar:5.3.24]
        at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultResponseSpec.lambda$bodyToMono$2(DefaultWebClient.java:543) ~[spring-webflux-5.3.24.jar:5.3.24]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:250) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:150) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onSubscribe(MonoFlatMapMany.java:245) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.Flux.subscribe(Flux.java:8642) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:195) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172) ~[reactor-core-3.4.26.jar:3.4.26]
        at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:431) ~[reactor-netty-http-1.0.26.jar:1.0.26]
        at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:702) ~[reactor-netty-core-1.0.26.jar:1.0.26]
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:637) ~[reactor-netty-http-1.0.26.jar:1.0.26]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) ~[reactor-netty-core-1.0.26.jar:1.0.26]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.AbstractHttp2StreamChannel$Http2ChannelUnsafe.doRead0(AbstractHttp2StreamChannel.java:901) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.AbstractHttp2StreamChannel.fireChildRead(AbstractHttp2StreamChannel.java:555) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2MultiplexHandler.channelRead(Http2MultiplexHandler.java:180) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2FrameCodec.onHttp2Frame(Http2FrameCodec.java:707) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:639) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2FrameCodec$FrameListener.onHeadersRead(Http2FrameCodec.java:633) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2FrameListenerDecorator.onHeadersRead(Http2FrameListenerDecorator.java:48) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onHeadersRead(Http2EmptyDataFrameListener.java:63) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) ~[netty-codec-http2-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) ~[netty-handler-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[netty-transport-classes-epoll-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

Reproducer: server.http2.enabled=true

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.netty.http.HttpProtocol;
import reactor.netty.http.client.HttpClient;

@RestController
@SpringBootApplication
public class Application {

    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

    @Autowired
    private WebClient.Builder webClientBuilder;

    @GetMapping(value = "/dto", produces = MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<Dto> getDto() {
        return ResponseEntity.noContent().build();
    }

//    @GetMapping(value = "/dto", produces = MediaType.APPLICATION_JSON_VALUE)
//    public ResponseEntity<Dto> getDto() {
//        return ResponseEntity.ok(new Dto());
//    }

    @GetMapping(value = "/test-ok", produces = MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<Dto> testOk() {
        return getResponse(HttpProtocol.HTTP11);
    }

    @GetMapping(value = "/test-fail", produces = MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<Dto> testFail() {
        return getResponse(HttpProtocol.H2C);
    }

    private ResponseEntity<Dto> getResponse(HttpProtocol protocol) {
        HttpClient httpClient = HttpClient.create()
                .protocol(protocol)
                .secure();

        var response = webClientBuilder
                .clientConnector(new ReactorClientHttpConnector(httpClient))
                .build().get()
                .uri("http://localhost:8080/dto")
                .accept(MediaType.APPLICATION_JSON)
                .retrieve()
                .bodyToMono(Dto.class)
                .checkpoint()
                .block();

        return response == null
                ? ResponseEntity.noContent().build()
                : ResponseEntity.ok(response);
    }
}

record Dto() {
}

Link to the full project: https://github.com/pkunk/repro-spring-http2

Comment From: bclozel

After a debugging session, I think I have a better understanding of this problem.

In the case of the Apache Tomcat + Reactor Netty client combination, the client reports a "transfer-encoding: chunked" response header with a zero byte length body. This confuses the Framework codec as it's trying to deserialize byte[0]. Apache Tomcat does not write this response header, so I've raised https://github.com/reactor/reactor-netty/issues/2664; I'm still not sure why this doesn't happen with Undertow (it does use different response headers).

This might be a bug in Reactor Netty, Netty itself, Apache Tomcat (or a combination of those); I don't think that the codecs behavior on the Spring Framework side is incorrect, so I'm closing this issue in favor of the Reactor Netty one.

Thanks for reporting this problem!