I have a small application that runs a spring-cloud-config-server service for configuration management, and number of other services that read their config from this. My issue is, up until recently, these other services automatically restarted (thanks to Docker) once they couldn't connect to the config server while it was starting up, and now they are stuck in a kinda limbo with the following error:
[main ] ERROR o.s.b.SpringApplication : Application run failed
with a connection refused (HttpHostConnectException) stack below to the config server endpoint. It seems that this error, or rather the exit code generated for this exception, is no longer causing the JVM to exit, which would allow Docker to handle the container restart.
My services run in Docker which has the following set in the docker-compose file:
restart: always
which should "Always restart the container if it stops".
An example of a service is:
@SpringBootApplication
@EnableWebFlux
@EnableDiscoveryClient
@Import({ SecurityConfiguration.class, ExceptionMappingConfig.class, AdminClientConfiguration.class })
public class MyServerApplication {
private static ConfigurableApplicationContext configurableApplicationContext;
public static void main(final String[] args) {
try {
configurableApplicationContext = SpringApplication.run(MyServerApplication.class, args);
} catch (final Exception e) {
e.printStackTrace();
throw e;
}
}
@PreDestroy
public void clearContext() {
configurableApplicationContext.close();
}
}
Some dependencies:
api("org.springframework.boot:spring-boot-starter-webflux")
api("org.springframework.cloud:spring-cloud-config-client")
api("org.springframework.cloud:spring-cloud-starter-netflix-eureka-client")
api("org.springframework.boot:spring-boot-starter-security")
api("org.springframework.cloud:spring-cloud-starter-gateway")
api(enforcedPlatform("org.springframework.boot:spring-boot-dependencies:$springBootVersion"))
api(enforcedPlatform("org.springframework.cloud:spring-cloud-dependencies:$springCloudStarterParentBomVersion"))
runtimeOnly("org.springframework.cloud:spring-cloud-starter-bootstrap")
implementation("org.springframework.boot:spring-boot-starter-webflux")
runtimeOnly("org.springframework.boot:spring-boot-properties-migrator")
runtimeOnly("org.springframework.boot:spring-boot-starter-validation")
runtimeOnly("org.springframework.boot:spring-boot-starter-actuator")
runtimeOnly("org.springframework.boot:spring-boot-starter-aop")
runtimeOnly("org.springframework.retry:spring-retry")
runtimeOnly("org.springframework.cloud:spring-cloud-starter-sleuth")
runtimeOnly("org.springframework.retry:spring-retry")
Spring Boot v2.5.3
Spring Cloud v2020.0.3
A poor workaround would be to add System.exit(1) instead of rethowing the exception in the catch to kill the JVM myself.
I can see that HttpHostConnectException nor ResourceAccessException implements ExitCodeGenerator, so have tried to implement my own so that when Spring exits, it always returns exit code 1:
public static void main(final String[] args) {
configurableApplicationContext = SpringApplication.run(MyServerApplication.class, args);
System.exit(SpringApplication.exit(configurableApplicationContext, exitCodeGenerator()));
}
@Bean
public static ExitCodeGenerator exitCodeGenerator() {
return () -> 1;
}
Also tried to map both exceptions to an exit code using ExitCodeExceptionMapper (this if check is pointless as it will always return 1, but in case I wanted something different) but was equally as fruitless:
@Bean
public ExitCodeExceptionMapper exitCodeToExceptionMapper() {
return exception -> {
if (exception.getCause() instanceof ResourceAccessException) {
return 1;
}
return 1;
};
}
Neither of these made a difference.
https://stackoverflow.com/questions/68758806/error-o-s-b-springapplication-application-run-failed-jvm-not-exiting-causing
Comment From: wilkinsona
The JVM should exit automatically when there are no non-daemon threads running. Alternatively, it should always exit once System.exit has been called. If the former isn't happening, that could be because there are one or more non-daemon threads still active. If the latter's not happening, that could be because a shutdown hook is still running or has deadlocked. Unfortunately, it's impossible to tell which, if either, of these apply in your case from the information you've provided thus far.
If you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: pbrown-gh
Apologies, I have been trying to get this reproducible in a complete minimal sample project. Turns out it is an issue with shutting down log4j2 Async appenders. Without any Async appenders, I see the SpringApplicationShutdownHook logs stopping and shutting down the LoggerContext etc, and the service restarts as expected.
2021-09-21 08:03:41,206 SpringApplicationShutdownHook DEBUG Stopping LoggerContext[name=5ffd2b27, org.apache.logging.log4j.core.LoggerContext@13d4992d]...
2021-09-21 08:03:41,209 SpringApplicationShutdownHook DEBUG Shutting down RollingRandomAccessFileManager /var/log/sample/sample-server.log
2021-09-21 08:03:41,210 SpringApplicationShutdownHook DEBUG Shutting down RollingFileManager /var/log/sample/sample-server.log
2021-09-21 08:03:41,211 SpringApplicationShutdownHook DEBUG All asynchronous threads have terminated
2021-09-21 08:03:41,211 SpringApplicationShutdownHook DEBUG RollingFileManager shutdown completed with status true
2021-09-21 08:03:41,211 SpringApplicationShutdownHook DEBUG Shut down RollingRandomAccessFileManager /var/log/sample/sample-server.log, all resources released: true
2021-09-21 08:03:41,212 SpringApplicationShutdownHook DEBUG Appender RollingRandomAccessFile stopped with status true
2021-09-21 08:03:41,212 SpringApplicationShutdownHook DEBUG Shutting down OutputStreamManager SYSTEM_OUT.false.false
2021-09-21 08:03:41,213 SpringApplicationShutdownHook DEBUG OutputStream closed
2021-09-21 08:03:41,213 SpringApplicationShutdownHook DEBUG Shut down OutputStreamManager SYSTEM_OUT.false.false, all resources released: true
2021-09-21 08:03:41,213 SpringApplicationShutdownHook DEBUG Appender STDOUT stopped with status true
2021-09-21 08:03:41,214 SpringApplicationShutdownHook DEBUG Log4j2 ConfigurationScheduler shutting down threads in java.util.concurrent.ScheduledThreadPoolExecutor@6989f04e[Running, pool size = 1, active threads = 0, queued tasks = 1, completed tasks = 1]
2021-09-21 08:03:41,215 SpringApplicationShutdownHook DEBUG Stopped XmlConfiguration[location=/opt/sample/config/log4j2.xml] OK
2021-09-21 08:03:41,215 SpringApplicationShutdownHook DEBUG Stopped LoggerContext[name=5ffd2b27, org.apache.logging.log4j.core.LoggerContext@13d4992d] with status true
However, with the Async appender present in configuration (regardless of whether its being used) this process doesn't appear to happen, and it just gets stuck after the exception, and the Java process never gets terminated.
I have a small, containerized project here that reproduces the problem: https://github.com/pbrown-gh/log4j2-async-spring-boot-restart-issue
You can find the log4j2 config under sample-server/src/main/resources/log4j2.xml.
Comment From: wilkinsona
Thanks for the sample. I'm almost certain that this is a duplicate of #27325 which, in turn, is a duplicate of LOG4J2-3102. You may want to try downgrading to 2.14.0 to confirm that it fixes your problem.