Teh Kok How opened SPR-17599 and commented

16725 is closed but it happens on Tomcat 9. The suggested code snippet does not work as it gives similar exception:

| org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'endpointExporterInitializer' defined in class path resource [com/graphql/book/AppConfig.class]: Initialization of bean failed; nested exception is java.lang.RuntimeException: java.lang.IllegalStateException: javax.websocket.server.ServerContainer not available| https://github.com/eh3rrera/graphql-java-spring-boot-example/issues/10 https://github.com/graphql-java-kickstart/graphql-spring-boot/issues/165 Response I get from users@tomcat.apache.org: "Spring is throwing the exception. Tomcat ships with the javax.websocket.server.ServerContainer class.

I don't believe there are any differences between Tomcat 8.5.x and 9.0.x when it comes to ClassLoader layout and class visibility.

I'd ask Spring how they are performing that sanity check to see why they are throwing that exception."

Affects: 5.1.3

Issue Links: - #16725 ServerEndpointExporter causes application context refresh to fail with an NPE when used in a Spring Boot app

Comment From: spring-projects-issues

Juergen Hoeller commented

There seems to be a misunderstanding in that conversation on the Tomcat mailing list: It's not that we can't find the class of the name javax.websocket.server.ServerContainer, it's the ServletContext attribute with name "javax.websocket.server.ServerContainer" that we can't find. In other words, we can't find a pre-initialized WebSocket container instance.

So the root of the problem you can easily reproduce without Spring: Simply implement a Servlet and check the return value of getServletContext().getAttribute("javax.websocket.server.ServerContainer")...

Comment From: spring-projects-issues

Teh Kok How commented

I add a simple webapp/test/index.jsp with the following content:

=== start JSP === <%= application.getAttribute("javax.websocket.server.ServerContainer") %> === end JSP ===

And the page shows:

org.apache.tomcat.websocket.server.WsServerContainer@7d9225a7

Comment From: wilkinsona

The same (I believe) problem was reported against Spring Boot (https://github.com/spring-projects/spring-boot/issues/15746). I've done some initial analysis and the change in behaviour in recent versions of Tomcat 9 appears to be an ordering change in Tomcat to fix https://bz.apache.org/bugzilla/show_bug.cgi?id=62868.

Comment From: khteh

- trunk for 9.0.13 onwards but I still see it happen iin tomcat 9.0.14

Comment From: wilkinsona

@khteh That is what I would expect. The problem is occurring in Spring Framework due to the change in Tomcat 9.0.13.

Comment From: wilkinsona

FWIW, I don't think this should have been closed, certainly not on the basis of my comment above anyway.

Currently, Spring Framework is relying upon some ordering in Tomcat that no longer holds true. I'm not sure that the Servlet spec requires the container to order things as Framework currently expects. If it does not, then I think a Framework change is in order to cope with Tomcat's new behaviour.

Comment From: rstoyanchev

Sorry, I misinterpreted the above then.

Comment From: adexerivera

Fortunately I had a version of java 9, in which I managed to make it work, to continue while the fix is resolved, this is version 9.0.8 of tomcat.

I hope they resolve it soon in the following versions to be up to date.

I hope it helps.

Comment From: khteh

What did you mean you managed to "make it work"? Did it just work with that particular tomcat version or you did something to make it work? If later, can you provide more details?

Comment From: batsauto

It worked for me without changes by switching to 9.0.8

Comment From: adexerivera

@khteh, I mean that in the version of java 9 that I had installed, the 9.0.8, it worked for me without doing anything.

Comment From: anushyakrishnankutty

Any update on this?

Comment From: davydotcom

any update on this as there are critical CVEs on versions before this tomcat version

Comment From: cgalpin

or any workarounds? I'm seeing this with 9.0.24 as well

Comment From: rgordeev

it's still actual on tomcat 9.0.29 ;(

Comment From: cgalpin

In case it helps, I got around it by add this to my web.xml

<absolute-ordering>
    <name>spring_web</name>
</absolute-ordering>

Comment From: helhelhel

How to fix this problem in springboot project?

Comment From: davydotcom

create a file in src/main/webapp/WEB-INF/web.xml

<web-app xmlns="http://java.sun.com/xml/ns/javaee"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
        http://java.sun.com/xml/ns/javaee
        http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
    version="3.0">

    <absolute-ordering>
        <name>spring_web</name>
    </absolute-ordering>

</web-app>

Comment From: helhelhel

My Project is springboot 2.1.6.RELEASE and direcotry src/main/web-app/WEB-INF doesn't exist. Is it ok to create directorysrc/main/web-app/WEB-INF? Will this affect other configuration?

Add: I've created directory src/main/web-app/WEB-INF and create web.xml. It doesn't work. How can I fix this problem?

Comment From: rstoyanchev

Yes it is okay to create that directory. However there is a typo above in web-app. That path should be src/main/webapp/WEB-INF/web.xml.

Comment From: rstoyanchev

I've had a look into this. First I'm going to link to @wilkinsona's https://github.com/spring-projects/spring-boot/issues/15746#issuecomment-456107588 and https://github.com/spring-projects/spring-boot/issues/15746#issuecomment-456124940 which explain why this happened in the first place.

Currently, Spring Framework is relying upon some ordering in Tomcat that no longer holds true.

Unfortunately I don't see much that we can do from the Spring Framework side. Essentially the entire initialization of Spring configuration happens within SpringBootServletInitializer and before Tomcat's WsSci so there is no Spring hook we can use, and I don't see any hook in the Servlet API either unless I'm missing something.

That leaves us with trying to influence ordering. Section 8.2.2 of the Servlet API describes a way for absolute and relative ordering of initializers, which means the spring_web fragment could order itself after others but I think doing this at this level is too opinionated and might easily run into regression issues.

The situation arises from the fact that Spring's ApplicationContext is loaded through a ServletContainerInializer which makes it susceptible to ordering issues. In a traditional webapp it would be loaded via ContextLoaderListener and that would not run into the same issue since Tomcat's WsSci would have done its job. Perhaps this could be considered a Tomcat issue, since one might reasonably expect its own initializers to run first, and it would be interesting to know if that is why it doesn't fail on other containers.

That said I can think of a couple of suggestions for what Boot might be able to do. The first is to add a web-fragment when running in WAR mode that fixes the order of spring_web relative to other fragments with a declaration like this:

<absolute-ordering>
    <others/>
    <name>spring_web</name>
</absolute-ordering>

The relevant code that handles this is around here. When there is absolute ordering declared, Tomcat loads the ones listed (applicationServicesFound) after its own (containerServicesFound). When there is no absolute ordering, containerServicesFound is empty, and instead all initializers are loaded in containerServicesFound.

According 8.2.3.5.c, if both the application web.xml and any web-fragment.xml declare an element that should appear once, then web.xml wins. This should work well because if the app did declare absolute-ordering then that brings the same benefits, or otherwise Boot's fragment will do that.

Another option would be for Boot to not use Spring's WebApplicationInitializer and to create its own ServletContainerInitializer specifically for use with Boot WAR deployment, and that can be declared to be ordered relatively after all others as described in 8.2.2.2. That way the specific use case for what Boot needs to do is separated out from the more general spring_web mechanism and that allows it to be more opinionated.

Let me know what you think @wilkinsona so we can decide how to proceed. Ultimately even an application declaring an explicit ordering as shown by @davydotcom above works, which we could choose to document but hopefully a more transparent solution can be found.

Comment From: helhelhel

@rstoyanchev I've created src/main/webapp/WEB-INF/web.xml but nothing changes.

Comment From: rstoyanchev

I've tested and debugged to understand why it works, so you you'll need to provide something to demonstrate it.

Comment From: helhelhel

@rstoyanchev I use Intellij IDEA Editor. In my springboot project, there are serveral modules. The project layout is as follows:

// layout of project
projectname
       --src/main/webapp/WEB-INF/web.xml // created manually
       --module1/src/main/java/Application.java // Class with annotiation @SpringBootApplication
       --module2/src/main/java/...

Is it right?

Add:

// org.springframework.web.socket.server.standard.AbstractStandardUpgradeStrategy
    protected ServerContainer getContainer(HttpServletRequest request) {
        ServletContext servletContext = request.getServletContext();
        String attrName = "javax.websocket.server.ServerContainer";
        ServerContainer container = (ServerContainer) servletContext.getAttribute(attrName);

                 // container is null
        Assert.notNull(container, "No 'javax.websocket.server.ServerContainer' ServletContext attribute. " +
                "Are you running in a Servlet container that supports JSR-356?");
        return container;
    }

Comment From: rstoyanchev

No it doesn't seem right but this type of discussion is better suited to Stack Overflow, see the guidelines for contributing.

Comment From: helhelhel

@rstoyanchev Yes. I've posted this question in Stack Overflow. If it's convenient for you, you can answer the question to help me solve the problem.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: davydotcom

I would love for you to look at a critical issue and fix it. thanks spring-issuemaster.

Comment From: wilkinsona

Thanks, @rstoyanchev.

which means the spring_web fragment could order itself after others but I think doing this at this level is too opinionated and might easily run into regression issues.

I have a similar fear about regressions if we were to do this in Spring Boot. A fix in Boot also means that anyone not using Boot will be no better off.

I can't recall how I identified that it was the changes made for https://bz.apache.org/bugzilla/show_bug.cgi?id=62868 that altered Tomcat's ordering. Assuming that is the right issue, the connection to SCI ordering isn't clear and leaves me wondering if it was changed accidentally.

@markt-asf We believe there was a change in Tomcat's ordering of SCIs in 9.0.13. Prior to the change, SCIs from Tomcat's lib directory would consistently be found (and therefore called) before those in an application's WEB-INF/lib directory. After the change, SCIs in Tomcat's lib directory are now consistently found and called after those in an application's WEB-INF/lib directory. This breaks Spring Framework's WebSocket integration that is reliant on it being initialised after Tomcat's WsSci has been called. Was this change in SCI ordering intentional? If not, could the previous ordering be restored? If it was intentional and needs to remain as it now is, what would you recommend as a general purpose solution to ensure that as many people users as possible get the ordering the Spring Framework needs, ideally without them having to do anything?

Comment From: davydotcom

I do want to clarify this affects more than Tomcat 9 it also affects latest Tomcat 8

Comment From: markt-asf

Some initial thoughts. Which fix introduced this change is somewhat of a red herring. The Servlet spec is explicit that containers must follow the class loader delegation model (i.e. web app first, then container) when discovering services via SCIs. Given the explicit requirement of the Servlet spec, it is extremely unlikely that the current behaviour will be changed. In terms of a general purpose solution, the one that immediately comes to mind is to perform the relevant Spring Framework init in a ServletContextListener which is guaranteed to run after any SCIs.

Comment From: wilkinsona

Thanks, Mark.

The Servlet spec is explicit that containers must follow the class loader delegation model

For my own education, I just went looking for where this is stated. The relevant section of the Servlet 4.0 specification is 8.2.4 where it says the following:

Implementations of the ServletContainerInitializer interface will be discovered by the runtime's service lookup mechanism or a container specific mechanism that is semantically equivalent to it. In either case, ServletContainerInitializer services from web fragment JAR files that are excluded from an absolute ordering MUST be ignored, and the order in which these services are discovered MUST follow the application’s class loading delegation model.

Knowing that, I'm now wondering why this problem has, as far as we know, only been seen with Tomcat. It seems unlikely (but possible) that every other container isn't spec compliant in this regard. Perhaps Tomcat is the only container that uses a ServletContainerInitializer to bootstrap part of itself?

Comment From: markt-asf

Tomcat's approach of using an SCI to bootstrap originates from when the WebSocket implementation was container neutral. The approach was retained partly because it allows WebSocket functionality to be disabled / enabled simply by the presence (or not) of the JAR - a requirement that stemmed primarily from users of Tomcat embedded - and partly because it is a very good fit for the annotation scanning requirements of the WebSocket spec.

It is certainly possible that other containers are using different bootstrap mechanisms.

It is also worth noting that it took quite some time for Tomcat to get all of the ordering aspects for fragments, SCIs etc correct. I wouldn't be surprised if other containers had similar issues; or if there were still a few edge cases to be fixed in Tomcat.

I've been reading through 8.2.4 again and I think there is some room for manoeuvre here. The delegation order matters when both the web application and the container specify the same SCI. It must be the one from the web application that is used. However, when we are looking at different SCIs I think there is scope to load the container provided SCIs first. I don't think the spec language precludes that. The more I think about it, the more that makes sense. If the web app depends on the container services (like Spring does) then the container services need to be loaded first. If the web app doesn't depend on them the order doesn't matter (so it is OK for container services to be first).

If you open a Tomcat bug on this I should be able to take a look - probably in January now.

Comment From: rstoyanchev

I have a similar fear about regressions if we were to do this in Spring Boot. A fix in Boot also means that anyone not using Boot will be no better off.

@wilkinsona the Spring Framework provides a WebApplicationInitializer hierarchy for loading Spring configuration in web applications, including Spring MVC DispatcherServlet setup. Those classes rely on ContextLoaderListener to load the Spring configuration similar to how it was always done for apps with web.xml. I verified that when this is used, the problem doesn't occur. The loading of the Spring config happens later.

By contrast SpringBootServletInitializer loads the Spring configuration directly under WebApplicationInitializer#onStartup so that makes it a little different. It's why I thought a solution in Boot such as a Boot ServletContainerInitializer ordered after all others might be appropriate.

Comment From: wilkinsona

Thanks, @markt-asf. I've opened https://bz.apache.org/bugzilla/show_bug.cgi?id=64021.

Comment From: wilkinsona

A fix in Tomcat has been committed. It will be available in 9.0.31, 8.5.51, and 7.0.100.

Spring Error running the application as WAR in tomcat 9 [SPR-17599]

16725 is closed but it happens on Tomcat 9. The suggested code snippet does not work as it gives similar exception: