When packaging an spring-boot application as a WAR, the context class loader of a Rest Controller is very slow to load a class that does not exist.
- The problematic class loader is :
org.springframework.boot.web.embedded.tomcat.TomcatEmbeddedWebappClassLoader
- It can be invoked by the function:
Thread.currentThread().getContextClassLoader().
The more dependencies the project have, the more the class loader is slow to respond.
You can test the problem with this simple project : it exposes a Rest Controller that tries to load a class with the context class loader, vs the normal class loader.
Here is the screenshot where I try to load an non existent class "toto":
It takes more than a second to respond.
With the classical class loader:
It is much more faster.
We encountered this problem while migrating a GWT application from a JBoss to a spring-boot with a tomcat embedded.
GWT tries to load non existent serialization classes, and with this class loader issue, the GUI is very slow : 10 seconds for a page to show up.
As a workaround, we are currently using Jetty embedded server instead, which uses only the classical class loader.
Comment From: philwebb
That's quite a dramatic performance penalty. We'll need to do some profiling to find out if the issue is in our TomcatEmbeddedWebappClassLoader or one of the higher Tomcat classloaders that we inherit from. @markt-asf have you seen any similar reports on Tomcat before?
Comment From: markt-asf
Nothing that comes to mind based on this description.
Comment From: wilkinsona
Thanks for the sample, @cdelgado83. I have reproduced the problem, but only when the application is packaged and run using java -jar. The slowness does not occur when run directly in the IDE.
With trace level logging enabled, I can see the following when running using java -jar:
2019-04-15 15:08:50.953 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase : findClass(com.example.1)
2019-04-15 15:08:50.953 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase : findClassInternal(com.example.1)
2019-04-15 15:08:51.423 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase : --> Returning ClassNotFoundException
2019-04-15 15:08:51.424 DEBUG 29022 --- [nio-8080-exec-3] o.a.c.loader.WebappClassLoaderBase : --> Passing on ClassNotFoundException
The processing performed for findClassInternal is taking almost 500ms.
If two requests that load the same non-existent class are made in quick succession the second request is much quicker:
2019-04-15 15:12:19.355 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase : findClass(com.example.1)
2019-04-15 15:12:19.356 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase : findClassInternal(com.example.1)
2019-04-15 15:12:19.830 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase : --> Returning ClassNotFoundException
2019-04-15 15:12:19.830 DEBUG 29022 --- [nio-8080-exec-9] o.a.c.loader.WebappClassLoaderBase : --> Passing on ClassNotFoundException
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase : findClass(com.example.1)
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase : findClassInternal(com.example.1)
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase : --> Returning ClassNotFoundException
2019-04-15 15:12:23.741 DEBUG 29022 --- [nio-8080-exec-1] o.a.c.loader.WebappClassLoaderBase : --> Passing on ClassNotFoundException
The second request is also much quicker if two requests for different classes are made in quick succession:
2019-04-15 15:13:42.315 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase : findClass(com.example.1)
2019-04-15 15:13:42.316 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase : findClassInternal(com.example.1)
2019-04-15 15:13:42.783 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase : --> Returning ClassNotFoundException
2019-04-15 15:13:42.784 DEBUG 29022 --- [nio-8080-exec-5] o.a.c.loader.WebappClassLoaderBase : --> Passing on ClassNotFoundException
2019-04-15 15:13:44.044 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase : findClass(com.example.2)
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase : findClassInternal(com.example.2)
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase : --> Returning ClassNotFoundException
2019-04-15 15:13:44.045 DEBUG 29022 --- [nio-8080-exec-7] o.a.c.loader.WebappClassLoaderBase : --> Passing on ClassNotFoundException
Comment From: wilkinsona
The slowness appears to be caused by JarWarResourceSet which is only used when jar files are nested inside a packed war file. This is the case when running a .war file with java -jar and also when deploying to Tomcat configured with unpackWARs=false.
There is a JarWarResourceSet for every jar in WEB-INF/lib. When an attempt is made to load a non-existent class, each resource set produces a map containing every entry in its jar. This map is cached, but only briefly as it's cleared by the background processing thread. There is a method for looking up a single entry, AbstractArchiveResourceSet.getArchiveEntry(String), but its javadoc states that "for performance reasons, getArchiveEntries(boolean) should always be called first" and the implementation in JarWarResourceSet always throws an IllegalStateException. This behaviour seems inefficient to me. Making the same class loading attempt using Boot's LaunchedURLClassLoader is much quicker, and it has the same nested jars on its classpath.
@markt-asf Does the above give you enough to have another look at this?
Comment From: markt-asf
Running from a packed WAR is always going to be slower because of the two layers of compression. Note that SpringBoot doesn't apply compression to the WAR. This enables access at a comparable speed to an unpacked WAR. Tomcat has to work with the more general case where the WAR may be compressed so Tomcat is slower.
Caching the entries speeds things up but that trades additional memory usage for speed. There is going to be a range of places where users want to draw that line.
I can see a couple if different solutions here:
- Use org.apache.catalina.webresources.ExtractingRoot. This will extract the JARs to the working directory on start allowing faster access.
- SpringBoot provides alternative Resource and ResourceSet implementations that are aware of the WAR not being compressed.
- Add an option to reduce rate of / disable the triggering of the gc() method to Tomcat. This should improve performance at the cost of a larger memory footprint. Note that if gc() is disabled then file locking will occur. This has the greatest impact on Windows.
Comment From: wilkinsona
Thanks, Mark. We're going to investigate implementing our own Resource and ResourceSet. In the meantime, and in general, we strongly recommend using jar rather than war packaging.
Comment From: cdelgado83
Thanks for the analysis !
When do you think this enhancement will be released officialy in Spring Boot ?
Comment From: wilkinsona
It's quite low priority as the problem can be avoided by using .jar packaging rather than .war packaging or, I believe, by switching to another embedded container. As things stand, we have no plans to tackle this in the 2.x timeframe.
Comment From: cdelgado83
Ok, thanks for the quick reply.
Unfortunatelly, I have not mentionned that we are stuck to the war packaging.
In fact, it seems that there is another issue while using the jar packaging: while handling an RPC call, GWT tries to load policy files (*.gwt.rpc) using:
dispatchServlet.getServletContext().getResourceAsStream(
serializationPolicyFilePath))
Which returns null, and the process it then stopped.
Comment From: wilkinsona
it seems that there is another issue while using the jar packaging
Please open a separate issue for that. If you provide a small sample that reproduces the problem, we'll take a look and see if it can be fixed.
Comment From: cdelgado83
Thanks for the update, I just open a new issue regarding the problem of serialization policy file loading : https://github.com/spring-projects/spring-boot/issues/17240
Comment From: koszta5
Just have to say that this bug is still very much present in Spring-boot v. 2.1.5. We ran into it with old and large app migrated from .war
.jsp kept us on war but I was able to work around this using .jar with the help of this --> https://github.com/hengyunabc/spring-boot-fat-jar-jsp-sample
Ps for us performance was a DRASTIC drop since we perform at-runtime classloading for extensive plugin system
Comment From: Frettman
How would one configure Tomcat to use the mentioned ExtractingRoot in Spring Boot?
Comment From: wilkinsona
@Frettman You should be able to use a TomcatServletWebServerFactory sub-class that overrides postProcessContext(Context) and calls setResources(WebResourceRoot) with an instance of ExtractingRoot:
@Bean
TomcatServletWebServerFactory tomcatFactory() {
return new TomcatServletWebServerFactory() {
@Override
protected void postProcessContext(Context context) {
context.setResources(new ExtractingRoot());
}
};
}
Comment From: Frettman
Thank you, @wilkinsona, this has worked perfectly.
Comment From: chenniufly
Thank you, @wilkinsona, this has worked perfectly.
Comment From: Frettman
There is one caveat to using the ExtractingRoot that one should be aware of: By default, every start of the application creates a new folder like tomcat.1150352738695762234.443 in the temp directory. This happens with or without the ExtractingRoot. But: With the ExtractingRoot the Jars will be extracted to a application-jars subdirectory of this temporary tomcat folder. And even though the Tomcat documentation states that this folder is removed when the application stops, this does not happen; at least not for me. So every start of your application will use up more space (unless your application is e.g. in a Docker container). To avoid that set the application property server.tomcat.basedir to any static folder. The Jars will be extracted on every start, so missing files will be recreated and others overwritten. However, obsolete files are not removed. So if Tomcat simply loads all Jars in that directory (not sure how to test that yet) that will most likely turn into a problem at some point. So it's probably a good idea to remove the application-jars folder e.g. in a shell script with every start.
Comment From: kobexzf
The slowness appears to be caused by
JarWarResourceSetwhich is only used when jar files are nested inside a packed war file. This is the case when running a.warfile withjava -jarand also when deploying to Tomcat configured withunpackWARs=false.There is a
JarWarResourceSetfor every jar inWEB-INF/lib. When an attempt is made to load a non-existent class, each resource set produces a map containing every entry in its jar. This map is cached, but only briefly as it's cleared by the background processing thread. There is a method for looking up a single entry,AbstractArchiveResourceSet.getArchiveEntry(String), but its javadoc states that "for performance reasons,getArchiveEntries(boolean)should always be called first" and the implementation inJarWarResourceSetalways throws anIllegalStateException. This behaviour seems inefficient to me. Making the same class loading attempt using Boot'sLaunchedURLClassLoaderis much quicker, and it has the same nested jars on its classpath.@markt-asf Does the above give you enough to have another look at this?
when running with JarLauncher the classical class loader loades all the classes in BOOT-INF, when running with WarLauncher the classical class loader loades all the classes in WEB-INF, so why is TomcatEmbeddedWebAppClassLoader needed?
Comment From: wilkinsona
I believe that this has been addressed in Spring Boot 3.2 by https://github.com/spring-projects/spring-boot/issues/37452.
Comment From: jplandrain
I have conducted some tests with Spring Boot 3.2.0 and also with 3.3.4 and I can pretty much assure it's not fixed : I still see slow performances when launching the War with "java -jar" outside the IDE. And this goes away when using this code (based on the github example from @cdelgado83):
package com.misc.tomcatslowness;
import org.apache.catalina.webresources.ExtractingRoot;
import org.apache.catalina.Context;
import org.springframework.boot.web.embedded.tomcat.TomcatContextCustomizer;
import org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.stereotype.Component;
@Component
public class MyTomcatWebServerCustomizer
implements WebServerFactoryCustomizer<TomcatServletWebServerFactory> {
@Override
public void customize(TomcatServletWebServerFactory factory) {
factory.addContextCustomizers((Context context) -> {
ExtractingRoot extractingRoot = new ExtractingRoot();
extractingRoot.setCachingAllowed(true); // Enable caching for better performance
context.setResources(extractingRoot);
});
}
}
Comment From: wilkinsona
@jplandrain if you'd like us to investigate, please provide a minimal sample that reproduces the problem, some data that shows the slow performance in absolute terms, and some details of the hardware, JVM, etc that you have tested on.