SpringBoot The data source has been destroyed, but the memory not released.

A case of DataSource RefreshScope, the first DataSource instance cannot be garbage collected after the data source destroy.

@Configuration
@ConfigurationProperties(prefix = "spring.datasource")
@Setter
public class DataSourceConfiguration {

    private Class<DataSource> type;

    @Bean
    @RefreshScope
    @ConfigurationProperties(prefix = "spring.datasource.hikari")
    DataSource dataSource() {
        return DataSourceBuilder.create().type(type).build();
    }
}

The data source can be refreshed successfully, but the first datasouce instance cannot be garbage collected.

jmap -histo:live [pid] |grep HikariDataSource
 929:             2            352  com.zaxxer.hikari.HikariDataSource

There are two instances.

I have an ugly solution，like this

@Component
public class RefreshScopeListener {

    @EventListener
    public void onApplicationEvent(RefreshScopeRefreshedEvent event) {
         try {

            Class<?>[] declaredClasses = DataSourcePoolMetrics.class.getDeclaredClasses();
            Field cache = declaredClasses[0].getDeclaredField("cache");

            cache.setAccessible(true);
            Map map = (Map) cache.get(declaredClasses[0]);

            map.clear();
        } catch (Exception e) {
            //todo ...
        }
    }
}

Related #26376

Comment From: wilkinsona

Looking at the code more closely, the cache is already using a ConcurrentReferenceHashMap. This map uses soft references by default which means that it won't cause the DataSources to be held in memory and that they're eligible for garbage collection when required.

AIUI, jmap -histo:live will trigger a full GC so, if there was no strong reference to the DataSource it should have been garbage collected. I suspect there is something other than the ConcurrentReferenceHashMap references the DataSource which is preventing it from being garbage collected. Unfortunately, I can't tell what that may be from the information provided thus far.

If you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

Comment From: juriad

Yesterday, we found a similar problem in org.springframework.boot.actuate.autoconfigure.endpoint.condition.AbstractEndpointCondition where the whole Environment is the key. Even though SoftReference does not prevent GC, it delays GC of the target until JVM is literally running out of memory.

Our use-case requires creating and destroying application contexts often (run-time reconfiguration and reevaluation of conditional beans). We were tracking a memory-leak and found that there were tens of contexts on the heap. Cutting the application down to bare minimum, we found that use of org.springframework.util.ConcurrentReferenceHashMap in the aforementioned class. Including/excluding a single actuator autoconfiguration (for example org.springframework.boot.actuate.autoconfigure.web.mappings.MappingsEndpointAutoConfiguration but it was true for many others) made the difference (we were focusing on counting the number of instances of DefaultListableBeanFactory on heap after calling System.gc()). This was not the cause of the main issue, but we spent way too much time on this side track.

Using SoftReference causes: * false lead when investigating memory-leaks (or investigating anything in heap dumps) * bad estimates for sizing memory requirements of an application

I assume that this has not been a problem yet because it is not a common use-case to create and destroy beans (OP) or whole contexts (us).

I would therefore propose to reevaluate the use of SoftReference in Spring Boot project and replace them with WeakReference where appropriate. There are only a few such places: https://github.com/spring-projects/spring-boot/search?q=ConcurrentReferenceHashMap. AFAIK the use of https://github.com/spring-projects/spring-boot/search?q=SoftReference are correct - it is extremely unlikely that someone is going to be replacing JARs while the application is running.

Comment From: wilkinsona

Thanks for sharing your experience, @juriad, but we would prefer to keep this issue focused on the originally reported problem.

If you would like us to consider your problem, please open a separate issue as I am not sure that the situations are the same.

Comment From: wuwen5

https://github.com/wuwen5/spring-boot-datasource-refresh/tree/master

DemoApplicationTests

Comment From: philwebb

@wuwen5 Can you tell us how we should use the sample? If I comment out the reflection code that clears the cache dump1 is still empty and the assert passes.

Comment From: wuwen5

jdk1.8.0_181.jdk ProductName: Mac OS X ProductVersion: 10.15.7

I just run it directly, If I comment out the reflection code that clears the cache

2021-05-14 18:13:12.367  INFO 13332 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-3 - Shutdown initiated...
2021-05-14 18:13:12.378  INFO 13332 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-3 - Shutdown completed.
[om.sun.proxy.$Proxy96
 600:             1            176  com.zaxxer.hikari.HikariDataSource
 601:            11            176 ]

org.opentest4j.AssertionFailedError: 
Expected :true
Actual   :false

Comment From: wilkinsona

I think there's a bug in the processing of the heap dump. With the code to clear the cache commented out the test fails for me occasionally. I changed the test a little to output dump and dump1. When the test passes, I see output like this:

dump: [ 608:             1            176  com.zaxxer.hikari.HikariDataSource
 609:            11            176  java.text.NumberForma]
dump1: []

When it fails, I see output like this:

dump: [] dump1: [sun.proxy.$Proxy96 600: 1 176 com.zaxxer.hikari.HikariDataSource 601: 11 176 ja]

Looking at the heap in YourKit, there are no strong references to the original DataSource and it is eligible for garbage collection. As such, I still haven't seen any sign of a memory leak.

Our cache is using soft references which is their most common application according to the SoftReference javadoc:

Soft reference objects, which are cleared at the discretion of the garbage collector in response to memory demand. Soft references are most often used to implement memory-sensitive caches.

It looks to me like this issue should be closed.