SpringBoot IllegalStateExecption causing stop of processing incoming messages through WebSocket

Target library: Most likely org.springframework.boot:spring-boot-starter-jetty

Version: 3.2.0

Jetty Version 12.0.7

Description

We are running Java service with Spring Boot exposing Web Socket based connection. And some connections suddenly stop receiving incoming message. Every incident like that coincides with Warning log from Jetty

WARN org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy [qtp1558825448-9112] Task run failed
java.lang.IllegalStateException: already released ReferenceCounter@4efeb79f[r=0]
    at org.eclipse.jetty.io.Retainable$ReferenceCounter.lambda$release$2(Retainable.java:192) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at java.base/java.util.concurrent.atomic.AtomicInteger.updateAndGet(AtomicInteger.java:281) ~[?:?]
    at org.eclipse.jetty.io.Retainable$ReferenceCounter.release(Retainable.java:189) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.io.AbstractRetainableByteBuffer.release(AbstractRetainableByteBuffer.java:58) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.io.ArrayByteBufferPool$Buffer.release(ArrayByteBufferPool.java:598) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.websocket.core.WebSocketConnection.releaseNetworkBuffer(WebSocketConnection.java:322) ~[jetty-websocket-core-common-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.websocket.core.WebSocketConnection.fillAndParse(WebSocketConnection.java:496) ~[jetty-websocket-core-common-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.websocket.core.WebSocketConnection.onFillable(WebSocketConnection.java:332) ~[jetty-websocket-core-common-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:410) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156) ~[jetty-util-12.0.7.jar!/:12.0.7]
    at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

It is possible that issue could be fixed in Jetty library under https://github.com/jetty/jetty.project/issues/12082 but I would appreciate if you can have a look into this problem on your side as well and maybe invest in handling this case and similar ones in a more robust way if that's possible Thanks in advance!

How to reproduce? In our case issue can be reproduced only under high load

Comment From: snicoll

Please report this to Jetty.