Affects: 5.2.10.RELEASE, 5.2.11.RELEASE
When switching from spring-messaging version 5.2.9.RELEASE or lower to 5.2.10.RELEASE or newer we encounter a ConcurrentModificationException in org.springframework.messaging.support.NativeMessageHeaderAccessor#copyHeaders as soon as a client tries to connect via websocket.
The log entry is o.s.w.s.m.StompSubProtocolHandler : Failed to send message to MessageChannel in session 7475384d-daf2-24fe-ada2-6a679e5af772:Failed to send message to ExecutorSubscribableChannel[clientInboundChannel]; nested exception is java.util.ConcurrentModificationException
The stacktrace of the ConcurrentModificationException looks like this:
java.util.ConcurrentModificationException: null
at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1493) ~[na:na]
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1526) ~[na:na]
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1524) ~[na:na]
at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1662) ~[na:na]
at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1655) ~[na:na]
at java.base/java.util.Map.forEach(Map.java:651) ~[na:na]
at org.springframework.messaging.support.MessageHeaderAccessor.copyHeaders(MessageHeaderAccessor.java:382) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.messaging.support.NativeMessageHeaderAccessor.copyHeaders(NativeMessageHeaderAccessor.java:135) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.cloud.sleuth.instrument.messaging.TracingChannelInterceptor.outputMessage(TracingChannelInterceptor.java:224) ~[spring-cloud-sleuth-core-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.cloud.sleuth.instrument.messaging.TracingChannelInterceptor.preSend(TracingChannelInterceptor.java:191) ~[spring-cloud-sleuth-core-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel$ChannelInterceptorChain.applyPreSend(AbstractMessageChannel.java:181) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:135) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
... 28 common frames omitted
The exception seems to be related to the headers inserted by spring-cloud-sleuth, but it is working without an exception if we use spring-messaging version 5.2.9.RELEASE or lower.
We have several Spring Boot WebMVC Applications running behind a Spring Cloud Gateway and the exception occurs in all of them as soon as we switch to spring-messaging 5.2.10.RELEASE or newer.
I believe that the problem could be caused by the following commits:
- https://github.com/spring-projects/spring-framework/commit/7ad60d385bf4af507fba530a6e77e4fc4e3fded6#diff-9c9bd4734096797978f2493f3c2ddef3493fe35083db245f77489593613c55a2
- https://github.com/spring-projects/spring-framework/commit/86f2ebec1b1f5438ba4d9908c71c710ef2f14219#diff-9c9bd4734096797978f2493f3c2ddef3493fe35083db245f77489593613c55a2
The full stacktrace for reference:
org.springframework.messaging.MessageDeliveryException: Failed to send message to ExecutorSubscribableChannel[clientInboundChannel]; nested exception is java.util.ConcurrentModificationException
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:149) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:125) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.web.socket.messaging.StompSubProtocolHandler.handleMessageFromClient(StompSubProtocolHandler.java:306) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:335) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleMessage(WebSocketHandlerDecorator.java:75) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleMessage(LoggingWebSocketHandlerDecorator.java:56) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleMessage(ExceptionWebSocketHandlerDecorator.java:58) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.handleTextMessage(StandardWebSocketHandlerAdapter.java:114) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.access$000(StandardWebSocketHandlerAdapter.java:43) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:85) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:82) ~[spring-websocket-5.2.11.RELEASE.jar:5.2.11.RELEASE]
at org.apache.tomcat.websocket.WsFrameBase.sendMessageText(WsFrameBase.java:402) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.server.WsFrameServer.sendMessageText(WsFrameServer.java:119) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.WsFrameBase.processDataText(WsFrameBase.java:502) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.WsFrameBase.processData(WsFrameBase.java:301) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.WsFrameBase.processInputBuffer(WsFrameBase.java:133) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:82) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.server.WsFrameServer.doOnDataAvailable(WsFrameServer.java:171) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.server.WsFrameServer.notifyDataAvailable(WsFrameServer.java:151) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:148) ~[tomcat-embed-websocket-9.0.39.jar:9.0.39]
at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.39.jar:9.0.39]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.util.ConcurrentModificationException: null
at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1493) ~[na:na]
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1526) ~[na:na]
at java.base/java.util.HashMap$EntryIterator.next(HashMap.java:1524) ~[na:na]
at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1662) ~[na:na]
at java.base/java.util.Collections$UnmodifiableMap$UnmodifiableEntrySet$1.next(Collections.java:1655) ~[na:na]
at java.base/java.util.Map.forEach(Map.java:651) ~[na:na]
at org.springframework.messaging.support.MessageHeaderAccessor.copyHeaders(MessageHeaderAccessor.java:382) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.messaging.support.NativeMessageHeaderAccessor.copyHeaders(NativeMessageHeaderAccessor.java:135) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.cloud.sleuth.instrument.messaging.TracingChannelInterceptor.outputMessage(TracingChannelInterceptor.java:224) ~[spring-cloud-sleuth-core-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.cloud.sleuth.instrument.messaging.TracingChannelInterceptor.preSend(TracingChannelInterceptor.java:191) ~[spring-cloud-sleuth-core-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel$ChannelInterceptorChain.applyPreSend(AbstractMessageChannel.java:181) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
at org.springframework.messaging.support.AbstractMessageChannel.send(AbstractMessageChannel.java:135) ~[spring-messaging-5.2.10.RELEASE.jar:5.2.10.RELEASE]
... 28 common frames omitted
Comment From: rstoyanchev
I was able to reproduce it. When TracingChannelInterceptor finds a mutable MessageHeaderAccessor, it is effecitvely modifying the headers of that same instance. Then it copies those headers to a newly obtained reference also of the same MessageHeaderAccessor instance. So source and target are the same Map instance and that doesn't work well with the change in #25821 where NativeMessageHeaderAccessor intercepts setHeader and forces removal of native headers.
Comment From: rstoyanchev
I have rolled back the setHeader interception in NativeMessageHeaderAccessor and updated copyHeaders to ensure it still gets a copy of native headers. There is also a check to avoid copying altogether if source and target header maps are the same instance.
Comment From: rstoyanchev
@Johannes-Rost this should be fixed. If you're able to confirm with 5.2.12 snapshots that would be great.
Comment From: Johannes-Rost
@rstoyanchev I have tried with 5.2.12.BUILD-SNAPSHOT and can confirm that the issue is fixed. Thank you very much for the fast response.