SpringBoot Fat jars cause classloading problems with JDK11 and Security Manager

We have faced some classloading problems after migrating to Spring Boot 2.1.6 and Spring Framework 5.1.8. The WAR we build is built under JDK8 but the runtime environment is JDK11. We are trying to deploy our application, but then we get ClassNotFoundException and the Spring context fails to initialize.

We have double checked WAR file. The "missing" class is there.

Now, the really interesting part: we are able to reproduce this one only with Security Manager. It seems like it's not a problem with our own custom security policy, as we have also tried granting access to everything. The problem persists even in this, rather extreme, case.

Therefore, to run our application we have only two options: 1. Run it under JDK8 2. Run it under JDK11, but without the Security Manager

Another description of (apparently) the same problem: https://stackoverflow.com/questions/54063602/springboot-on-open-jdk-11-classnotfound-errors-when-securitymanager-is-activ

Comment From: philwebb

Can you please provide a stacktace and if possible a sample application that we can run to debug the issue?

Comment From: piotrlitwin

I have generated a simple app with https://start.spring.io/. When running WAR file with Security Manager I get:

2019-08-08 14:00:26.573  WARN 13016 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during
context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start web
server; nested exception is java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
        at org.springframework.boot.loader.WarLauncher.main(WarLauncher.java:58)
Caused by: java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy
        at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119)
        at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
        at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
        at ch.qos.logback.classic.Logger.log(Logger.java:765)
        at org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog.error(LogAdapter.java:410)
        at org.springframework.boot.SpringApplication.reportFailure(SpringApplication.java:822)
        at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:797)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:322)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1214)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1203)
        at com.example.demo.DemoApplication.main(DemoApplication.java:10)
        ... 8 more

It is not actually ClassNotFoundException, which happens in conjunction with another framework we use in our project. This NoClassDefFoundError I get from empty Spring Boot application seems to be the root cause of our problems.

Comment From: snicoll

Thanks for sharing the stacktrace. I've edited your comment to improve the formatting. You might want to check out this Mastering Markdown guide for future reference.

There are several reference to ThrowableProxy and there is nothing at this point that indicates this is a Spring Boot bug. See #6151 that points ultimately to this thread. Can you share what makes you think this might be an issue in Spring Boot?

Comment From: piotrlitwin

The #6151 is not the best example, because it is the runtime problem. In our case Spring Boot app does not even start. Ok, so I have generated a demo app and built it with mvn clean install. Then I tried to run it with java -jar -Djava.security.manager demo-0.0.1-SNAPSHOT.war As previously stated, it happens only under JDK11 with -Djava.security.manager. Without Security Manager everything works fine. I expect, that an empty application which I have generated will run with Security Manager as well. Which unfortunately didn't happened. To be sure, I have put this statement in Security Policy:

grant {
permission java.security.AllPermission;
}

and the problem persists, which shows us clearly, that this is not a problem with Security Policy itself, but with Spring Boot or, potentially, JDK. Moreover Caused by: java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy makes no sense, as we are able to localize it in generated WAR file.

It is also noteworthy, that this clearly is a problem with compatibility with JDK11. I have tried: 1. Build it under JDK8, run it under JDK8 -> no problem 2. Build it under JDK8, run it under JDK11 -> fail 3. Build it under JDK11, run it under JDK11 -> fail

Comment From: piotrlitwin

I have also tried to do the same with Spring Boot 1.5.1.RELEASE and did not observed any problems with classloading.

Comment From: snicoll

I did a few tries, excluding logback and including log4j (got a log4j class) then excluding the logging system entirely and got the same error on a core Spring Boot class.

The fact classloading is different with or without a security manager feels like a bug in the JDK at a first glance but we'll keep investigating to narrow down the scope of the problem.

Comment From: hohwille

As the issue also comes if you are using a policy with AllPermissions granted it apprears to be a bug or side-effect that the security manager is causing in Java11 negatively influencing the classloading. If this would be a bug in spring boot itself e.g. in the bootstrap classloader implementation this would IMHO imply that some exceptions from security manager are caught consumed and not propagated in any way but instead leading to the class not found exception. However, neigther the stacktrace of the NoClassDefFoundError nor the fact that it works with JDK8 seems to make this likely. I also ran the app with -Djava.security.debug="access,failure" but got no indication that a rejection from the Java Security Manager is actually causing the classloader to fail. Any ideas how to debug or trace down this issue to the root? I tried with some breakpoints but this is extremely tricky. I always get my breakpoint when it is too late or I end up debugging endless times of generic classlaoder stuff even with conditional breakpoints. Some expert hint anybody?

Comment From: wilkinsona

The root cause appears to be a NullPointerException in jdk.internal.loader.URLClassPath.JarLoader.checkJar(JarFile). Its source looks like this:

static JarFile checkJar(JarFile jar) throws IOException {
    if (System.getSecurityManager() != null && !DISABLE_JAR_CHECKING
        && !zipAccess.startsWithLocHeader(jar)) {
        IOException x = new IOException("Invalid Jar file");
        try {
            jar.close();
        } catch (IOException ex) {
            x.addSuppressed(ex);
        }
        throw x;
    }
    return jar;
}

This first test in the if shows why the problem only occurs when the security manager is enabled. The caller, jdk.internal.loader.URLClassPath.Loader.getResource(String, boolean), catches Exception and returns null which explains why the NullPointerException is not apparent in the application's output and why the problem appears to be a classloading failure.

The attempt to dereference a null variable is occurring in jdk.internal.misc.JavaUtilZipFileAccess.startsWithLocHeader(ZipFile) when the anonymous implementation in java.util.zip.ZipFile attempts to access zip.res.zsrc.startsWithLoc. zip is a org.springframework.boot.loader.jar.JarFile, res is a java.util.zip.ZipFile.CleanableResource and zsrc is a java.util.zip.ZipFile.Source. It's zsrc that is null when it is expected not to be.

Comment From: wilkinsona

zsrc is null because the JarFile has been closed. It's closed by org.springframework.boot.web.servlet.server.StaticResourceJars.isResourcesJar(JarFile). I think this close() call is correct and the problem is that org.springframework.boot.loader.jar.Handler continues to use a closed jar file or perhaps that it allows the close() call to actually close a jar file that it may reuse.

Comment From: wilkinsona

@piotrlitwin You should be able to work around the problem by launching your app with -Dsun.misc.URLClassPath.disableJarChecking=true.

Comment From: piotrlitwin

@wilkinsona Thanks a lot, that did the trick!

Comment From: hohwille

You really rock the house, spring-boot team 👍 Compared to some of my other customer projects where some old-school people that still pay huge piles of money to get a useless enterprise JEE server in order to be vendor-locked and get stuck with technical dept and workarounds I can only say that sping-boot is the best experience that could ever happen to Java/JEE. Political correct version of the above statement can be found here (LOL): https://github.com/devonfw/devon4j/blob/develop/documentation/guide-jee.asciidoc#jee

Comment From: philwebb

I think this fix is breaking the Windows build

Comment From: philwebb

Windows only fails on 2.2.x, I've opened #21126 to try and get to the bottom of that.

Comment From: sascha-kaufmann

I still face the issue described above (SecurityManager enabled working fine with jdk8, but failing with jdk11). I get the following exception (where AbcApplication is my main class, annotated as SpringBootApplication) when starting the application:

Exception in thread "main" java.lang.ClassNotFoundException: abc.def.AbcApplication
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:151)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:398)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:46)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:107)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

Tried several versions of spring-boot (2.2.9.RELEASE, 2.3.9.RELEASE, 2.4.3) where the fix should already be in place. I have the following setup: * Gradle Build, using the spring-boot gradle plugin * RedHat OpenJDK 11.0.10.9 * Developing on a windows machine

Comment From: wilkinsona

@sascha-kaufmann It's hard to say if that's the same problem. Judging by your stack trace compared to those above the symptoms are not the same. If you would like us to spend some time investigating, please open a new issue and provide a minimal sample that reproduces the problem.