SpringBoot NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse during shutdown

Hello. I'm on 1.4.0.RELEASE. My app is pretty basic, but what can be important here, I'm using @EnableScheduling. Often, not always, when I ctrl-c or kill my app (run by ./<jar>.jar) I'm getting below error:

Exception in thread "Thread-2" java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:235)
    at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:366)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:225)
    at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:277)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306)
    at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1010)
    at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:923)

Comment From: wilkinsona

Unfortunately, I've been unable to reproduce this from the information provided. @pawelantczak Can you please provide a small sample that reproduces the problem?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Comment From: DenEwout

Hi,

I am also experiencing this issue. The problem appears to be triggered whenever we perform a new deploy.

Kind Regards,

Ewout.

Comment From: philwebb

@DenEwout What version of Boot are you using? Are you able to share a sample that reproduces the problem?

Comment From: bendem

Same problem with Running with Spring Boot v1.4.1.RELEASE, Spring v4.3.3.RELEASE

Exception in thread "Thread-3" java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:240)
        at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:366)
        at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:224)
        at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:276)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1010)
        at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:923)

It happens each time we deploy (using /etc/init.d/<app> stop).

Comment From: snicoll

@bendem can you please provide a sample that reproduces the issue? That's what we need to better understand what happens.

Comment From: bendem

Will try to isolate the problem, or at least provide reproduction steps later today.

Comment From: eduardocorral

Suffering the same issue, but I have a theory for mine: I'm deploying a new jar on top of the existing one before stopping. This will most likely cause problems when the JVM tries to locate the class, if not loaded before, given the classloader reference to the previous jar is now invalid.

Replacing live jar/s used to cause JVM 5/6 SISSEGVs in Solaris. At least this is just an exception.

Comment From: bendem

Tried to setup reproduction steps, I cannot reproduce this locally 😠 This is not about overriding the jar, we stop our server before updating the jar.

Comment From: diegov

Also experiencing the same issue using 4.2.1.RELEASE with embedded tomcat. We do shut down the application before replacing the jar, but I'm finding this in the logs just before the NoClassDefFoundError:

2016-12-23 11:06:06.122 WARN 3873 --- [ Thread-5] o.s.c.support.DefaultLifecycleProcessor : Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000: [schedulerFactoryBean]

Maybe the shutdown isn't complete by the time we replace the jar.

Comment From: Traubenfuchs

As for me, I get the following line before the NoClassDefFoundError: Invocation of destroy method 'close' failed on bean with name 'dataSource': java.lang.NoClassDefFoundError: com/mysql/jdbc/ProfilerEventHandlerFactory

I am on 1.4.2.RELEASE and I use the default MySQL version. The error does not occur reliably.

Comment From: diegov

I found out the application wasn't being shutdown when I thought it was (we're using chef and it queues up service stops / restarts to run at the end), so we were definitely overwriting the jar with the application still running.

I've updated our deployment recipes to actually stop the application before the jar is overwritten and after a cycle of about 15 test deployments I haven't been able to reproduce the issue.

Comment From: andersthorbeck

I just experienced the same error message consistently on deploy. When I changed the deployment execution order so that the application was shut down before the new jar file (with the same name) replaced the old one, the issue immediately went away.

Comment From: WennySoft

We also hit this error and I succeeded to reproduce it always in our setup.

Here is what we (want) do on redeployment (linux): 1. we replace the jar file 2. we call via jolokia a MBean which does some preparation steps inside the application before shutdown. this takes about 1 Minute 3. Shutdown the application 4. Start the application

In reality on step 2 we are getting no response (timeout) and in the log we are getting the exception Exception in thread "http-nio-8201-exec-9" java.lang.NoClassDefFoundError: ch/qos/logback/classic/spi/ThrowableProxy INFO | jvm 1 | 2017/08/25 15:19:32 | at ch.qos.logback.classic.spi.LoggingEvent.<init>(LoggingEvent.java:119) INFO | jvm 1 | 2017/08/25 15:19:32 | at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419) INFO | jvm 1 | 2017/08/25 15:19:32 | at ch.qos.logback.classic.Logger.filterAndLog_0_Or3P

If we call the MBean without replacing the jar file everything works well.

I assume that for the JMX call there is a need to load some more classes and this fails as the file is replaced. Normally this should work as the process has the old version still open and therefore should not be influenced by the new file. The failing load of classes leads to starvation of the request somewhere so we do not get any response and wait for a timeout.

We cannot (easily) change the ordering as this is effectively a restart made by an ansible handler (which is triggered e.g. by the replaced jar file).

I've never seen this problem in non-boot-spring-webapps - it seems that spring boot does something special regarding the usage of classloaders here.

Is this a bug or a not supported usecase? Is there any chance that there will be a fix which make this setup working (in the near future)?

Comment From: wilkinsona

Unfortunately, all bets are off after step 1. The JVM's behaviour is, as far as I know, completely undefined if you replace a jar file that it's using while it's still running.

Comment From: WennySoft

The JVM's behaviour is, as far as I know, completely undefined if you replace a jar file that it's using while it's still running.

From a JVM perspective it is undefined. But from a filesystem perspective (on linux) an open file remains valid until it is closed as it still holds the handle. So this should work - and does in at least many applications in the past.

If a file is replaced it depends on how it is replaced. If it is deleted/renamed and readded (what a normal copy do) it will get a new handle and the open file still holds the old handle until it is reopend - so the old process should not see the changes at all. If the content of the file is replaced without creating a new file then the running process will see the new content.

So I see two viable reasons for this behaviour:

1. The jar file is not kept open but is closed and reopened but still uses the old offsets. If this is the case there is a bug - either by closing the jar file or by not checking timestamp or something else on reopening and if needed updating the offsets
2. Ansible (in our case) does not replace the jar file but opens it and replaces the contents - then the behaviour is indeed undefined but can be avoided by changing the ansible module

Comment From: wilkinsona

If this is the case there is a bug

I don't agree with that conclusion. What you are trying to do is unsupported (for now at least). We can consider supporting it but it will depend on how much complexity that will add and how easy it is to test. If this is something that you'd like to pursue, please open a new issue requesting an enhancement and providing a simple means of reproducing the behaviour that you're currently seeing.

Comment From: WennySoft

It's completly acceptable if this is unsupported even it leads to complexity on deployment.

What I mean by "there is a bug" that there is data cached and reused (I read somewhere that the offsets inside the file to the jars are cached) but not checked if these are still valid on use.

I would suggest to do at least a simple modification-timestamp-check on the jar file before blindly accessing it.

Comment From: user20161119

Get the same error after i deploy a few times on my server,but did not happened on my local laptop. Exception in thread "Thread-6" java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:240) at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:377) at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:241) at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:295) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155) at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1012) at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)

Comment From: aburmeis

Same error still with Spring Boot 1.5.10:

Exception in thread "Thread-5" java.lang.NoClassDefFoundError:
org/apache/catalina/Lifecycle$SingleUse
         at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:240)
         at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:377)
         at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stopTomcat(TomcatEmbeddedServletContainer.java:241)
         at org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainer.stop(TomcatEmbeddedServletContainer.java:295)
         at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.stopAndReleaseEmbeddedServletContainer(EmbeddedWebApplicationContext.java:306)
         at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onClose(EmbeddedWebApplicationContext.java:155)
         at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1014)
         at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)

Does not occur in local IDE (IntelliJ) nor when running with java -jar. But same jar started using init.d on a Debian creates the exception.

Comment From: philwebb

@aburmeis I don't think anyone has been able to provide a sample to us yet. If you've found a way to reliably trigger the problem, are you able to create a small sample application that we can build, run and debug?

Comment From: jvasileff

I ran into this error, specifically:

Exception in thread "SpringContextShutdownHook" java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle$SingleUse
        at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:269)
        at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:469)
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stopTomcat(TomcatWebServer.java:269)
        at org.springframework.boot.web.embedded.tomcat.TomcatWebServer.stop(TomcatWebServer.java:324)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.stopAndReleaseWebServer(ServletWebServerApplicationContext.java:306)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.onClose(ServletWebServerApplicationContext.java:172)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1035)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:948)

The problem turned out to be a premature call to URLClassLoader.close() on the classloader that loaded the application and all dependencies (including spring, tomcat, etc).

Comment From: oharsta

Very late to the party, but we were encountering the same issue and this was the first Google hit. The problem can be consistently reproduced by replacing the jar (or change the symlink) and then trying to stop it. We are using ansible for deployments and we ensured the boot app was completely stopped before changing the jar:

- name: stop app
  service: name={{ springapp_service_name }} state=stopped
  when: maven_result.changed

- name: wait for the app to be fully stopped
  wait_for:
    port: "{{ springapp_tcpport }}"
    state: stopped
    delay: 5

- name: change symlink to current version
  file:
    src: "{{ maven_result.dest }}"
    dest: "{{ springapp_dir }}/{{ springapp_jar }}"
    state: link
    owner: "{{ springapp_user }}"
    group: "{{ springapp_user }}"
  notify: restart {{ springapp_service_name }}
  when: maven_result.changed