SpringBoot java.util.concurrent.RejectedExecutionException is logged when shutting down Spring Boot app with Cassandra

I have an application that uses Spring Boot 2.4.3 with Spring Data Cassandra (using the default version declared in the BOM, which is 3.1.5).

When starting the app and terminating it with SIG_INT, an exception is thrown:

[SpringContextShutdownHook] WARN org.springframework.beans.factory.support.DisposableBeanAdapter - Destroy method 'close' on bean with name 'cassandraDriverConfigLoader' threw an exception
java.util.concurrent.RejectedExecutionException: event executor terminated
    at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
    at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
    at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
    at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
    at io.netty.util.concurrent.AbstractEventExecutor.submit(AbstractEventExecutor.java:115)
    at com.datastax.oss.driver.internal.core.util.concurrent.RunOrSchedule.on(RunOrSchedule.java:57)
    at com.datastax.oss.driver.internal.core.config.typesafe.DefaultDriverConfigLoader.close(DefaultDriverConfigLoader.java:175)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:281)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:215)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1152)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1145)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1105)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1074)
    at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:990)

I've tested this in an isolated application that just launches the app and the problem persists.

If it helps, I've also analysed the problem, and found out the following: * org.springframework.boot.autoconfigure.cassandra.CassandraAutoConfiguration declares two beans: cassandraSession and cassandraDriverConfigLoader. * During shutdown, cassandraDriverConfigLoader bean is terminated twice (close method is called twice) * The first time it is closed is when cassandraSession bean is terminated. This bean is currently implemented in com.datastax.oss.driver.internal.core.session.DefaultSession. When theclose method is called in this class, it will internally call closePolicies method, which includes the cassandraDriverConfigLoader bean, referenced through the supplier context::getConfigLoader * The second time, it happens as expected, when cassandraDriverConfigLoader bean is terminated.

Comment From: snicoll

Thanks for the report. Paging @adutra for some insights as this exception happens in the Cassandra driver.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: adutra

Hi, I admit that the way some close() methods are implemented in the driver are a bit racy.

For DefaultSession, the race condition is handled in an optimistic way, see here.

For DefaultDriverConfigLoader however, no such protection exists. I created JAVA-2932 to capture this. However, we don't have plans to release a new version of the driver any time soon.

I'd suggest, if possible, to close DefaultDriverConfigLoader first, then DefaultSession. Or alternatively, don't close DefaultDriverConfigLoader explicitly, since DefaultSession will do it anyways.

Comment From: wilkinsona

Thanks, @adutra. We can't close DefaultDriverConfigLoader before DefaultSession as the bean dependencies are the other way around. I think we could, however, prevent DefaultDriverConfigLoader from being closed, and rely on DefaultSession closing it instead.

Comment From: wilkinsona

The warning is logged when running CassandraDataAutoConfigurationIntegrationTests:

2021-04-01 17:16:12.528  WARN   --- [           main] o.s.b.f.support.DisposableBeanAdapter    : Destroy method 'close' on bean with name 'cassandraDriverConfigLoader' threw an exception: java.util.concurrent.RejectedExecutionException: event executor terminated

Comment From: adutra

FYI, the fix will be delivered in driver 4.11.2, due this week.