Hi,
I can't pinpoint since when this started, but SSL based tests are really flaky on my machine. Mostly when run in the whole suite and not when run in isolation. E.g. the following tests fail relatively frequent:
org.springframework.boot.rsocket.netty.NettyRSocketServerFactoryTests > tcpTransportBasicSslFromClassPath()
org.springframework.boot.rsocket.netty.NettyRSocketServerFactoryTests > tcpTransportBasicSslFromFileSystem()
org.springframework.boot.rsocket.netty.NettyRSocketServerFactoryTests > websocketTransportBasicSslFromClassPath()
org.springframework.boot.rsocket.netty.NettyRSocketServerFactoryTests > websocketTransportBasicSslFromFileSystem()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > basicSslFromClassPath()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > basicSslFromFileSystem()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > sslNeedsClientAuthenticationSucceedsWithClientCertificate()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithClientCertificate()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithoutClientCertificate()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > sslWithValidAlias()
org.springframework.boot.web.embedded.netty.NettyReactiveWebServerFactoryTests > whenSslIsConfiguredWithAValidAliasARequestSucceeds()
org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithoutClientCertificate()
I'm mostly running on JDK 11 btw, but I'm doing that for over a year or longer and never experienced that.
An example stacktrace:
java.lang.AssertionError: expectation "expectNext(test payload)" failed (expected: onNext(test payload); actual: onError(java.nio.channels.ClosedChannelException))
at reactor.test.MessageFormatter.assertionError(MessageFormatter.java:115)
at reactor.test.MessageFormatter.failPrefix(MessageFormatter.java:104)
at reactor.test.MessageFormatter.fail(MessageFormatter.java:73)
at reactor.test.MessageFormatter.failOptional(MessageFormatter.java:88)
at reactor.test.DefaultStepVerifierBuilder.lambda$addExpectedValue$10(DefaultStepVerifierBuilder.java:501)
at reactor.test.DefaultStepVerifierBuilder$SignalEvent.test(DefaultStepVerifierBuilder.java:2212)
at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onSignal(DefaultStepVerifierBuilder.java:1484)
at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onExpectation(DefaultStepVerifierBuilder.java:1432)
at reactor.test.DefaultStepVerifierBuilder$DefaultVerifySubscriber.onError(DefaultStepVerifierBuilder.java:1091)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at io.rsocket.core.ResolvingOperator$DeferredResolution.onError(ResolvingOperator.java:476)
at io.rsocket.core.DefaultRSocketClient$FlattingInner.accept(DefaultRSocketClient.java:410)
at io.rsocket.core.DefaultRSocketClient$FlattingInner.accept(DefaultRSocketClient.java:366)
at io.rsocket.core.ResolvingOperator.terminate(ResolvingOperator.java:206)
at io.rsocket.core.DefaultRSocketClient.onError(DefaultRSocketClient.java:164)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onError(FluxMapFuseable.java:140)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onError(FluxOnAssembly.java:392)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189)
at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onUncaughtException(DefaultPooledConnectionProvider.java:219)
at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onUncaughtException(DefaultPooledConnectionProvider.java:468)
at reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:130)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)
at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:832)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
at io.netty.handler.ssl.SslUtils.handleHandshakeFailure(SslUtils.java:441)
at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1798)
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1066)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:831)
Suppressed: java.nio.channels.ClosedChannelException
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1063)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Assembly trace from producer [reactor.core.publisher.MonoCreate] :
reactor.core.publisher.Mono.create
reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:112)
Error has been observed at the following site(s):
|_ Mono.create ⇢ at reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:112)
|_ Mono.create ⇢ at reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:112)
|_ ⇢ at reactor.netty.tcp.TcpResources.acquire(TcpResources.java:162)
|_ ⇢ at reactor.netty.tcp.TcpResources.acquire(TcpResources.java:162)
|_ Mono.map ⇢ at io.rsocket.transport.netty.client.TcpClientTransport.connect(TcpClientTransport.java:119)
|_ Mono.map ⇢ at io.rsocket.transport.netty.client.TcpClientTransport.connect(TcpClientTransport.java:119)
|_ ⇢ at io.rsocket.core.RSocketConnector.lambda$null$5(RSocketConnector.java:525)
|_ ⇢ at io.rsocket.core.RSocketConnector.lambda$null$5(RSocketConnector.java:525)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:525)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:525)
|_ Mono.map ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:526)
|_ Mono.map ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:526)
|_ Mono.map ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:530)
|_ Mono.map ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:530)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:533)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:533)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:540)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.lambda$connect$15(RSocketConnector.java:540)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.connect(RSocketConnector.java:515)
|_ Mono.flatMap ⇢ at io.rsocket.core.RSocketConnector.connect(RSocketConnector.java:515)
|_ ⇢ at io.rsocket.core.RSocketConnector.connect(RSocketConnector.java:501)
|_ ⇢ at io.rsocket.core.RSocketConnector.connect(RSocketConnector.java:501)
|_ ⇢ at io.rsocket.core.DefaultRSocketClient.unwrapReconnectMono(DefaultRSocketClient.java:78)
|_ ⇢ at io.rsocket.core.DefaultRSocketClient.unwrapReconnectMono(DefaultRSocketClient.java:78)
|_ ⇢ at io.rsocket.core.DefaultRSocketClient.requestResponse(DefaultRSocketClient.java:93)
|_ ⇢ at io.rsocket.core.DefaultRSocketClient.requestResponse(DefaultRSocketClient.java:93)
|_ Mono.map ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:289)
|_ Mono.map ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:289)
|_ Mono.map ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:290)
|_ Mono.map ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:290)
|_ ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:272)
|_ ⇢ at org.springframework.messaging.rsocket.DefaultRSocketRequester$DefaultRequestSpec.retrieveMono(DefaultRSocketRequester.java:272)
Stack trace:
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1063)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:831)
Unfortunately, I can't tell you more at this point. It might be just some timing problem.
Cheers, Christoph
Comment From: wilkinsona
I've just seen some failures that appear to be similar:
NettyReactiveWebServerFactoryTests > basicSslFromFileSystem() FAILED
org.springframework.web.reactive.function.client.WebClientRequestException at ExchangeFunctions.java:141
Caused by: java.nio.channels.ClosedChannelException at SslHandler.java:1063
NettyReactiveWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithClientCertificate() FAILED
org.springframework.web.reactive.function.client.WebClientRequestException at ExchangeFunctions.java:141
Caused by: java.nio.channels.ClosedChannelException at SslHandler.java:1063
NettyReactiveWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithoutClientCertificate() FAILED
org.springframework.web.reactive.function.client.WebClientRequestException at ExchangeFunctions.java:141
Caused by: java.nio.channels.ClosedChannelException at SslHandler.java:1063
NettyReactiveWebServerFactoryTests > sslNeedsClientAuthenticationSucceedsWithClientCertificate() FAILED
org.springframework.web.reactive.function.client.WebClientRequestException at ExchangeFunctions.java:141
Caused by: java.nio.channels.ClosedChannelException at SslHandler.java:1063
NettyReactiveWebServerFactoryTests > basicSslFromClassPath() FAILED
org.springframework.web.reactive.function.client.WebClientRequestException at ExchangeFunctions.java:141
Caused by: java.nio.channels.ClosedChannelException at SslHandler.java:1063
I don't think I've seen these before. I suspect they're due to a change in Netty or Reactor Netty but I'm not sure which just yet. @violetagg, do you know of any changes in this area in the recent (Reactor) Netty releases?
Comment From: violetagg
@wilkinsona Please provide the full stack for the exception or a link to the logs from the failing tests.
Also in the original report there is org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactoryTests > sslWantsClientAuthenticationSucceedsWithoutClientCertificate() is this also related to Reactor Netty/Netty?
Comment From: wilkinsona
More failures, with a build scan this time. In both cases, I was building with Java 11 (11.0.10+9). Looking at the scan, the following looks rather suspicious:
java.lang.IncompatibleClassChangeError: Class io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback does not implement the requested interface io.netty.internal.tcnative.CertificateCallback
at io.netty.internal.tcnative.CertificateCallbackTask.runTask(CertificateCallbackTask.java:40) ~[netty-tcnative-boringssl-static-2.0.39.Final.jar:na]
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:38) ~[netty-tcnative-boringssl-static-2.0.39.Final.jar:na]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$3.run(ReferenceCountedOpenSslEngine.java:1447) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1512) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1526) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1390) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1245) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282) ~[netty-handler-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Comment From: wilkinsona
Thanks for taking a look so quickly, Violeta. Sorry, I didn't notice TomcatServletWebServerFactoryTests at the end of the tests that failed for @dreis2211. They use Tomcat and Apache HTTP Client so this can't be (just) a (Reactor) Netty problem. There are some full stack traces in the build scan but I think we need to figure out the IncompatibleClassChangeError before digging any further.
Comment From: violetagg
@wilkinsona This indicates a mismatch of the Netty binaries. Do you have other components (except Reactor Netty) that may have transitive dependencies to Netty or shadowing the Netty binaries? Also do you keep up-to-date also Netty tcnative module?
Comment From: dreis2211
The Tomcat one rarely fails in comparison to the Netty ones - I would say for 10 failures for Netty there is one for Tomcat.
Comment From: snicoll
I got the IncompatibleClassChangeError and the fact that it is flaky is really surprising.
Comment From: wilkinsona
I don't understand it at all at the moment. OpenSslServerCertificateCallback has implemented io.netty.internal.tcnative.CertificateCallback since it was introduced 3 years ago and the callback interface's package hasn't changed in that time either. I'd also expect a failure when creating the CertificateCallbackTask rather than when running it.
Mostly when run in the whole suite and not when run in isolation
I wonder if it's some sort of class loader problem such that CertificateCallbackTask and OpenSslServerCertificateCallback each reference a different io.netty.internal.tcnative.CertificateCallback that was loaded by a different class loader. I'd still expect the call to the constuctor of CertificateCallbackTask to fail in that case though.
Comment From: wilkinsona
It looks like a class loading and test ordering problem. I've reproduced the problem in the debugger and, in this case, CertificateCallbackTask was loaded by org.springframework.boot.testsupport.classpath.ModifiedClassPathClassLoader@4647c5c6 while the callback was loaded by jdk.internal.loader.ClassLoaders$AppClassLoader@67424e82. Looking at the path of the modified class path class loader, it appears to be from some Jetty 10 tests but the test that's failing is in NettyReactiveWebServerFactoryTests.
Comment From: wilkinsona
JNI is in the mix too. Here's the stack when the CertificateCallbackTask is created:
CertificateCallbackTask.<init>(long, byte[], byte[][], CertificateCallback) line: 29
SSL.readFromSSL(long, long, int) line: not available [native method] [local variables unavailable]
OpenSslEngine(ReferenceCountedOpenSslEngine).readPlaintextData(ByteBuffer) line: 635
OpenSslEngine(ReferenceCountedOpenSslEngine).unwrap(ByteBuffer[], int, int, ByteBuffer[], int, int) line: 1245
OpenSslEngine(ReferenceCountedOpenSslEngine).unwrap(ByteBuffer[], ByteBuffer[]) line: 1371
OpenSslEngine(ReferenceCountedOpenSslEngine).unwrap(ByteBuffer, ByteBuffer) line: 1414
SslHandler$SslEngineType$1.unwrap(SslHandler, ByteBuf, int, ByteBuf) line: 224
SslHandler.unwrap(ChannelHandlerContext, ByteBuf, int) line: 1338
SslHandler.decodeNonJdkCompatible(ChannelHandlerContext, ByteBuf) line: 1245
SslHandler.decode(ChannelHandlerContext, ByteBuf, List<Object>) line: 1282
SslHandler(ByteToMessageDecoder).decodeRemovalReentryProtection(ChannelHandlerContext, ByteBuf, List<Object>) line: 507
SslHandler(ByteToMessageDecoder).callDecode(ChannelHandlerContext, ByteBuf, List<Object>) line: 446
SslHandler(ByteToMessageDecoder).channelRead(ChannelHandlerContext, Object) line: 276
DefaultChannelHandlerContext(AbstractChannelHandlerContext).invokeChannelRead(Object) line: 379
AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext, Object) line: 365
DefaultChannelPipeline$HeadContext(AbstractChannelHandlerContext).fireChannelRead(Object) line: 357
DefaultChannelPipeline$HeadContext.channelRead(ChannelHandlerContext, Object) line: 1410
DefaultChannelPipeline$HeadContext(AbstractChannelHandlerContext).invokeChannelRead(Object) line: 379
AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext, Object) line: 365
DefaultChannelPipeline.fireChannelRead(Object) line: 919
NioSocketChannel$NioSocketChannelUnsafe(AbstractNioByteChannel$NioByteUnsafe).read() line: 166
NioEventLoop.processSelectedKey(SelectionKey, AbstractNioChannel) line: 719
NioEventLoop.processSelectedKeysOptimized() line: 655
NioEventLoop.processSelectedKeys() line: 581
NioEventLoop.run() line: 493
SingleThreadEventExecutor$4.run() line: 989
ThreadExecutorMap$2.run() line: 74
FastThreadLocalRunnable.run() line: 30
DefaultLoopResources$EventLoop(Thread).run() line: 834
The TCCL is the app class loader but CertificateCallbackTask's class loader is a ModifiedClassPathClassLoader from a previous test. It would appear that the native code beneath SSL.readFromSSL is using the wrong class loader.
Comment From: dreis2211
Well, at least https://github.com/netty/netty-tcnative/commit/eb23bb10e08eb319c5452212fc9326572f5525f2 was done fairly recent
Comment From: wilkinsona
😀 I was just coming back here to comment with a link to https://github.com/netty/netty-tcnative/pull/642. If my analysis is right, I think netty/netty-tcnative@eb23bb1 will fix the Netty parts of this at least.
Until Netty tcNative 2.0.40 is released and we can upgrade, we may be able to work around the problem by changing the modified class path class loader to skip io.netty.internal.tcnative classes.