I noticed a change in behavior for this endpoint after we decided to upgrade Spring Cloud (Finchley.SR2 -> Hoxton.SR7).
The usual behavior that we expected to see was that whenever we made a POST to this it could shutdown some services and more specifically, it would change our health check status to DOWN so that Eureka would know that our app was down and remove it from the server. However, now whenever we POST to /actuator/pause I see that the application is shutdown completely.
At first I thought it was a behavior related to Tomcat since the logs seem similar to other events of pause, except that now it logs that it stopped running Tomcat and seems some threads were not expecting this, so it gives me an Exception.
Eventually, the /pause endpoint simply gives a Empty reply from server rather than the usual true we used to get. What's the current expected behavior now?
This is the logs that I'm referring to now:
2020-08-06 18:12:45.082 INFO [app-name,,,] 23271 --- [nio-8180-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-08-06 18:12:45.082 INFO [app-name,,,] 23271 --- [nio-8180-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2020-08-06 18:12:45.108 INFO [app-name,,,] 23271 --- [nio-8180-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 26 ms
2020-08-06 18:12:45.173 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.impl.engine.AbstractCamelContext : Apache Camel 3.4.2 (camel-1) is shutting down
2020-08-06 18:12:45.174 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.i.engine.DefaultShutdownStrategy : Starting to graceful shutdown 15 routes (timeout 45 seconds)
2020-08-06 18:12:45.463 WARN [app-name,,,] 23271 --- ] c.c.j.DefaultJmsMessageListenerContainer : Setup of JMS message listener invoker failed for destination '/*queue.name*/' - trying to recover. Cause: Thread was interrupted while waiting for a restart of the listener container, but container is still stopped
// -> Confirms shutdown of several Apache Camel Routes
// -> Confirms shutdown of ExecutorService of a JmsConsumer
2020-08-06 18:12:47.791 INFO [app-name,,,] 23271 --- [ - ShutdownTask] o.a.c.i.engine.DefaultShutdownStrategy : Route: MF xml consume shutdown complete, was consuming from: jms://queue:queue.name
2020-08-06 18:12:47.791 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.i.engine.DefaultShutdownStrategy : Graceful shutdown of 15 routes completed in 2 seconds
2020-08-06 18:12:47.809 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.impl.engine.AbstractCamelContext : Apache Camel 3.4.2 (camel-1) uptime 2 minutes
2020-08-06 18:12:47.809 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.impl.engine.AbstractCamelContext : Apache Camel 3.4.2 (camel-1) is shutdown in 2.636 seconds // -> Up to here this was normal
2020-08-06 18:12:47.820 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-08-06 18:12:47.824 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.apache.catalina.core.StandardWrapper : Waiting for [1] instance(s) to be deallocated for Servlet [dispatcherServlet]
2020-08-06 18:12:48.860 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.apache.catalina.core.StandardWrapper : Waiting for [1] instance(s) to be deallocated for Servlet [dispatcherServlet]
2020-08-06 18:12:49.881 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.apache.catalina.core.StandardWrapper : Waiting for [1] instance(s) to be deallocated for Servlet [dispatcherServlet]
2020-08-06 18:12:49.986 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Destroying Spring FrameworkServlet 'dispatcherServlet'
2020-08-06 18:12:49.990 WARN [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.a.c.loader.WebappClassLoaderBase : The web application [ROOT] appears to have started a thread named [spring.cloud.inetutils] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2020-08-06 18:12:49.996 WARN [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] org.apache.tomcat.util.net.NioEndpoint : The executor associated with thread pool [http-nio-8180] has not fully shutdown. Some application threads may still be running.
2020-08-06 18:12:49.998 INFO [app-name,88afb3363848ce3b,88afb3363848ce3b,false] 23271 --- [nio-8180-exec-1] o.s.c.n.e.s.EurekaServiceRegistry : Unregistering application APP-NAME with eureka with status DOWN
2020-08-06 18:12:49.998 INFO [app-name,,,] 23271 --- [extShutdownHook] o.s.c.n.e.s.EurekaServiceRegistry : Unregistering application app-name with eureka with status DOWN
2020-08-06 18:12:50.005 INFO [app-name,,,] 23271 --- [extShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'taskExecutor'
2020-08-06 18:12:50.017 INFO [app-name,,,] 23271 --- [extShutdownHook] com.netflix.discovery.DiscoveryClient : Shutting down DiscoveryClient ...
2020-08-06 18:12:50.031 INFO [app-name,,,] 23271 --- [extShutdownHook] com.netflix.discovery.DiscoveryClient : Completed shut down of DiscoveryClient
Process finished with exit code 0
This happens right after I POST to /actuator/pause.
springCloudVersion = 'Hoxton.SR7' springBootVersion = '2.3.2.RELEASE' camelVersion = '3.4.2'
Comment From: OlgaMaciaszek
Hi, @memod, thanks for reporting this. It, indeed, was happening but was handled via an issue in commons and the fix has also been merged.