Spring ConcurrentModificationException with MockMvc, MockHttpServletResponse, and StreamingResponseBody

Affects

Bug definitely occurs on at least Spring Boot versions 3.2.0-SNAPSHOT and 3.0.7.

Overview

Hello, we had encountered an issue with flaky tests on rest endpoints with a StreamingResponseBody. Upon further investigation it appears to be unrelated to our own code but rather an inherent issue within MockHttpServletResponse, due to the nature of using a streaming response body the request is handled concurrently, but within the MockHttpServletResponse a LinkedCaseInsensitiveMap is used (which isn't thread safe) for the headers.

This issue may be related to the following issues. Although some are marked as fixed, the fix doesn't appear to be entirely effective.

https://github.com/spring-projects/spring-security/issues/11452
https://github.com/spring-projects/spring-security/issues/7224
23460
30427

I was unable to replicate this issue outside of tests. So the real HttpServletResponse may be unaffected.

While the test passes most of the time, about 5 in 5000 runs fail (sometimes fewer, sometimes more) on our large repo. The rate of the test failing is a lot higher in the sample provided below (likely due to differences in config). This isn't too problematic, but when running a CI pipeline with many of these tests these failures are a concern.

Typical stack trace

Below is a typical stack trace, occasionally there is minor variation, like the parent exception being ServletException, but the root cause appears to be the same ConcurrentModificationException on the map every time.

java.util.ConcurrentModificationException
    at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1221)
    at org.springframework.util.LinkedCaseInsensitiveMap.computeIfAbsent(LinkedCaseInsensitiveMap.java:239)
    at org.springframework.util.LinkedCaseInsensitiveMap.computeIfAbsent(LinkedCaseInsensitiveMap.java:50)
    at org.springframework.mock.web.MockHttpServletResponse.doAddHeaderValue(MockHttpServletResponse.java:748)
    at org.springframework.mock.web.MockHttpServletResponse.setHeaderValue(MockHttpServletResponse.java:695)
    at org.springframework.mock.web.MockHttpServletResponse.setHeader(MockHttpServletResponse.java:669)
    at jakarta.servlet.http.HttpServletResponseWrapper.setHeader(HttpServletResponseWrapper.java:132)
    at org.springframework.security.web.firewall.FirewalledResponse.setHeader(FirewalledResponse.java:54)
    at jakarta.servlet.http.HttpServletResponseWrapper.setHeader(HttpServletResponseWrapper.java:132)
    at org.springframework.security.web.header.writers.frameoptions.XFrameOptionsHeaderWriter.writeHeaders(XFrameOptionsHeaderWriter.java:103)
    at org.springframework.security.web.header.HeaderWriterFilter.writeHeaders(HeaderWriterFilter.java:99)
    at org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse.writeHeaders(HeaderWriterFilter.java:132)
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:93)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82)
    at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
    at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:374)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191)
    at org.springframework.security.test.web.servlet.setup.SecurityMockMvcConfigurer$DelegateFilter.doFilter(SecurityMockMvcConfigurer.java:132)
    at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132)
    at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:201)
    at com.example.springbugdemorepo.FileControllerTest.testDownloadFile(FileControllerTest.java:66)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

Our current work-around is to re-run the test if the exception caught matches the issue, up to 3 times (this can be achieved with a simple while loop, config or @RetryingTest from JUnit extensions).

To replicate it yourself you need something similar to the sample repo I had created -- https://github.com/bartmarkiewicz/springBugDemoRepo -- where you can replicate this bug by running the provided test a large number of times.

You must use springSecurity(), or have some sort of filter chain.

My suggested fix albeit I didn't really look at the MockHttpServletResponse in detail -> make LinkedCaseInsensitiveMap thread safe, wrap it around something, or use a thread safe map.

Comment From: sbrannen

I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

Comment From: Laess3r

Ou're product is also affected from this bug. It only occurs during CI build, where a lot of tests run in parallel.

Thanks for the "retry" idea, we will add that until we have a better solution.

Stacktrace for reference:

java.util.ConcurrentModificationException at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1221) at org.springframework.util.LinkedCaseInsensitiveMap.computeIfAbsent(LinkedCaseInsensitiveMap.java:239) at org.springframework.util.LinkedCaseInsensitiveMap.computeIfAbsent(LinkedCaseInsensitiveMap.java:50) at org.springframework.mock.web.MockHttpServletResponse.doAddHeaderValue(MockHttpServletResponse.java:720) at org.springframework.mock.web.MockHttpServletResponse.setHeaderValue(MockHttpServletResponse.java:667) at org.springframework.mock.web.MockHttpServletResponse.setHeader(MockHttpServletResponse.java:641) at jakarta.servlet.http.HttpServletResponseWrapper.setHeader(HttpServletResponseWrapper.java:132) at org.springframework.security.web.firewall.FirewalledResponse.setHeader(FirewalledResponse.java:54) at jakarta.servlet.http.HttpServletResponseWrapper.setHeader(HttpServletResponseWrapper.java:132) at org.springframework.security.web.header.writers.XXssProtectionHeaderWriter.writeHeaders(XXssProtectionHeaderWriter.java:51) at org.springframework.security.web.header.HeaderWriterFilter.writeHeaders(HeaderWriterFilter.java:99) at org.springframework.security.web.header.HeaderWriterFilter$HeaderWriterResponse.writeHeaders(HeaderWriterFilter.java:132) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:93) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240) at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:323) at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:224) at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268) at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132) at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) at org.springframework.mock.web.MockFilterChain.doFilter(MockFilterChain.java:132) at org.springframework.test.web.servlet.MockMvc.perform(MockMvc.java:201)

Comment From: christophers88

Same issue in our CI build. We have quite some tests on StreamingResponseBody rest controllers and often one of them runs in this exception, thus making the build result pretty much unreliable.

Comment From: sbrannen

At a glance, this appears to be a duplicate of https://github.com/spring-projects/spring-security/issues/9175 which has not yet been resolved in Spring Security.

@rstoyanchev or @rwinch, can either of you confirm that?

Comment From: bclozel

Closing in favor of https://github.com/spring-projects/spring-security/issues/9175, as making MockHttpServletResponse thread-safe would not solve the underlying problem.

Spring ConcurrentModificationException with MockMvc, MockHttpServletResponse, and StreamingResponseBody

Affects

Overview

23460

30427

Typical stack trace