Affects: 5.3.19

The following exception appeared in our central error logging. I don't know what the input URL was exactly; unfortunately the following stack trace is all the information I have on this issue.

Java Version: 17

Stack Trace:

java.lang.NullPointerException: Cannot invoke "String.indexOf(String)" because "path" is null
    at org.springframework.web.util.UrlPathHelper.getSanitizedPath(UrlPathHelper.java:408)
    at org.springframework.web.util.UrlPathHelper.decodeAndCleanUriString(UrlPathHelper.java:551)
    at org.springframework.web.util.UrlPathHelper.getOriginatingRequestUri(UrlPathHelper.java:496)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.addContentDispositionHeader(AbstractMessageConverterMethodProcessor.java:431)
    at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:288)
    at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:183)
    at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
    at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:327)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
    at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126)
    at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.txture.microservice.common.compression.GzipFilter.doFilter(GzipFilter.kt:22)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.txture.microservice.common.security.JwtAuthenticationFilter.doFilterInternal(JwtAuthenticationFilter.kt:68)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
    at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:336)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:211)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:183)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:833)

Comment From: rstoyanchev

Looking at the stacktrace, the path came through a call to request.getRequestURI() in UrlPathHelper#getOriginatingRequestUri on line 493 since it is neither WebSphere, nor a forwarded request. From what I can see Tomcat does not return null from that method, tracing it down to org.apache.coyote.Request#requestURI. So I can only assume this is somehow related to Spring Security wrapping the request? @jzheaux or @rwinch any thoughts?

Comment From: rwinch

I do not believe this is anything to do with Spring Security. Spring Security wraps the HttpServletRequest but only DummyRequest implements HttpServletRequest.getRequestURI()and it doesn't appear to be used in the call stack. Below are all the places Spring Security wraps the HttpServletRequest:

  • DebugRequestWrapper - Used if DebugFilter is in the call stack which occurs if debugging enabled. The call stack does not contain DebugFilter
  • DummyRequest - Used for applications testing if authorization is allowed for another HttpServletRequest. Based on the call stack this does not appear to be applicable for this case.
  • FirewalledRequest (RequestWrapper, StrictFirewalledRequest) - Used for rejecting requests that look to be potentially malicious. This is being used, but FirewalledRequest and subclasses do not implement HttpServeltRequest.getRequestURI()
  • HeaderWriterRequest - This is only used when HeaderWriterFilter is in the call stack. It does not implement HttpServeltRequest.getRequestURI()
  • SavedRequestAwareWrapper - This is used to restore a request that was made prior to authentication to the original request so that it can be replayed after authentication success. Given the call stack, I am guessing that this is not being used because it mentions JWT which typically implies the request cache is disabled. Even if it is being used, it does not implement HttpServeltRequest.getRequestURI()
  • SaveToSessionRequestWrapper - This is probably not being used given the call stack states JWT. However, even if it is being used it does not implement HttpServeltRequest.getRequestURI().

There are a few other Filters in the call stack that I wonder if they wrap the request.

  • org.txture.microservice.common.compression.GzipFilter.doFilter(GzipFilter.kt:22)
  • org.txture.microservice.common.security.JwtAuthenticationFilter.doFilterInternal(JwtAuthenticationFilter.kt:68)

What is interesting to me is that it appears that the uri is null when passed into getSanitizedPath(String)

  • We know that path is null at UrlPathHelper.java:408 to to the NullPointerException. This is the same value that was passed into getSanitizedPath(String).

However, uri is non-null when passed into removeSemicolonContent(String)

Given the String uri is immutable and is not null when passed into removeSemicolonContent(String) (using the logic above) it appears that either removeSemicolonContent(String) or decodeRequestString(HttpServletRequest,String) are returning null some how.

I'm curious what the vendor and exact JDK version is as that might help determine how this would be possible.

Comment From: MartinHaeusler

@rwinch The issue occurred on Eclipse Adoptium 17.0.3 (successor to AdoptOpenJDK). We're using the embedded tomcat that comes with Spring Boot 2.7.0.

Comment From: rwinch

Thanks for the reply @MartinHaeusler

Looking at the removeSemicolonContent(String) and removeJsessionId(String) along with Eclipse Adoptium 17.0.3 source I'm not understanding how it would be possible for them to return a null value. I must be missing something. I'll see if @rstoyanchev or someone else can spot what I am overlooking.

I'm not sure it matters, but I'm curious. In the original report you stated this happens with Spring 5.3.19, but then you stated you are using Spring Boot 2.7.0. Did you override the Spring Framework version to 5.3.19 for some reason? I ask because Spring Boot 2.7.0's default Spring Framework version is 5.3.20.

Comment From: MartinHaeusler

@rwinch my team employs a rather strict policy when it comes to dependency management (and their versions). Most of the time we're on the latest patch level; maybe this was an oversight.

Comment From: rstoyanchev

Indeed the requestURI coming into decodeAndCleanUriString couldn't have been null or else it would have caused NPE. From there it's a fairly short path to the stacktrace.

Looking at removeSemicolonContent I don't see a way for that to return null. For decodeRequestString. If the request encoding is invalid, however, the UnsupportedCharsetException catch clause falls back on URLDecoder.decode(String) and that in turn relies on the default platform encoding which is the static field dfltEncName in URLDecoder. It seems unlikely that be invalid but if it caused UnsupportedEncodingException it would lead to a null return value.

@MartinHaeusler could you provide details for the JDK version?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Comment From: creckord

I'm getting the exact same stacktrace with Eclipse Temurin 11. JDK version details:

# java -version
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment Temurin-11.0.15+10 (build 11.0.15+10)
OpenJDK 64-Bit Server VM Temurin-11.0.15+10 (build 11.0.15+10, mixed mode)

Comment From: rstoyanchev

@creckord I've reopened but we could use some help with debugging potentially. See https://github.com/spring-projects/spring-framework/issues/28671#issuecomment-1173851957 for some clues. Anything further you might be able to find out would be great.

Comment From: creckord

@rstoyanchev I poked around a bit more in our log server and noticed that whenever this exception occured, the application was in the process of shutting down, but still processing some requests (i.e. I got a Stopping ProtocolHandler ["http-nio-8080"] roughly 200ms before the exception each time). Maybe that could trigger a behavior as you outlined above.

Anyway, I saw that there's some debug logging in UrlPathHelper.decodeInternal when that happens, so I'll activate that and see if I get something out of it when the error happens again.

Comment From: rstoyanchev

Servlet containers reuse request and response object instances. If the shutdown, resets those, that could help to explain. Spring Boot has a graceful shutdown feature that could help with this.

Comment From: johannespostler

I'm on @MartinHaeusler 's team. I'm not sure whether this is relevant, but I'd like to add that we are NOT using the embedded Tomcat, but rather the latest Tomcat 9.0.x. Our application is deployed as a WAR file in Tomcat.

Comment From: rstoyanchev

From https://github.com/spring-projects/spring-framework/issues/28671#issuecomment-1163313975 and https://github.com/spring-projects/spring-framework/issues/28671#issuecomment-1173851957, it looks unlikely there is a possible null value that isn't handled.

The most likely explanation at this point is that Tomcat resets the request, possibly due to concurrent request handling and a shutdown, and I'm not sure it's worth protecting against such a null value since in theory it could occur anywhere else where it wouldn't otherwise be expected.

I'm wondering if you have tried the graceful shutdown?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Comment From: awstutorials

Hi, I am also facing the similar issue. Running the spring boot application in Java 17. For me also the application is in the process of shutdown. I am using embedded tomcat running in CloudFoundry environment. What type of information you are looking for and I can try to provide that information.

Comment From: awstutorials

Also as additional point we are running 20-30 applications in the Cloud Foundry environment and this issue is happening only in handful of applications. I noticed a spring framework version difference and we upgraded to 2.7.4 but still i see the same issue.

Comment From: rstoyanchev

There is not much we can do on our end. The server is shutting down, and there are in-flight requests being processed. You can try graceful shutdown. At best we could put something that ignores the null but that's hardly the ideal place for such a fix and a similar issue could appear elsewhere.