After upgrading our applications to Spring boot 2.7.1 version, we have started encountering memory issues.
Our Observations
Changes were made in 2.7.1 in the org.springframework.boot.loader.jar package around closing jar files. A list of nested jars were added and that is what is holding on to objects in our heap dump.
We can see 22,000 JarFile objects in the heap dump where we'd expect about 118 as that is how many nested jars there are in the application. Note that apart from the expected jar files, the other 22,000-odd are all url = jar:file:/deployments/application-web-1.0.0.jar!/BOOT-INF/lib/swagger-ui-4.11.1.jar!/
Please find below the heap dump and Dynatrace screenshot as reference
Note :- No changes were made in org.springframework.boot.loader.jar package in 2.7.2 so I suspect the problem will still be there.
Can someone please look into this.
Comment From: wilkinsona
Thanks for the report, @loveshjain. There are some similarities with #32042 and #31853. Are you using PropertiesLauncher to run your app? To allow us to diagnose the problem, please provide a minimal sample that reproduces the leak.
Comment From: philwebb
@loveshjain I've pushed a fix for #31853 that removes the nested jars list. Once build 713 could you please try the latest SNAPSHOT and see if the memory issues persist?
Comment From: beggsj42
Hi @philwebb - I work with @loveshjain and am looking at this. Our problem is that we haven't been able to replicate this in non-prod. I'm going to see what I can do in that regard in the next couple of days.
I'm speculating that this might have to do with the JarFile objects being removed by the GC (as they are soft referenced?) if there is memory pressure, and then being reloaded later, but never getting removed from the nested jars list? I've never really looked at custom class loader code though so I'm a bit in the dark. Does that sound reasonable though - to maybe try to get the JVM close to the heap limit and see if that causes it? I'm just looking for things to try really.
It was fairly odd behaviour in production - it went for several days without us noticing an issue, and when we did hit the issue it wasn't in a high-load period and it seemed to self-correct. In the tenured gen picture above it looks like it started to have a GC issue about 10am, but we didn't notice until it started affecting response time badly at about 9:45pm.
We have had similar issues on multiple services though and rolling back to 2.7.0 resolved it.
Comment From: beggsj42
I believe we're using the JarLauncher - we're just using the spring-boot-maven-plugin and we're not overriding the 'layout'.
Comment From: philwebb
See https://github.com/spring-projects/spring-boot/issues/31853#issuecomment-1218359941. We're going to revert #29356
Comment From: beggsj42
Right, thanks - I'm fairly sure that will solve our issue. I'll still try to put some effort in to replicating the problem though hopefully it'll mostly just be for the intellectual exercise now.
Comment From: snicoll
@beggsj42 upgrading to 2.7.3 contains the change that Phil mentioned if you want to double check.
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: beggsj42
Sorry for the late reply. I still haven't been able to replicate the problem in 2.7.1 in non-prod, but 2.7.3 definitely has solved the issue for us in production.
We're happy for this issue to be closed, thanks.
Comment From: snicoll
Thanks for following up.
Comment From: sm0217
Apologies for commenting on a closed ticket but we are seeing similar issues upgrading from 2.4 to 2.7.16. May I know if there was any fix provided for this and it is present in the latest 2.7.x versions
Comment From: wilkinsona
Please see above. A change was made in Spring Boot 2.7.3 that fixed the problem that was reported in this issue. That fix has not be removed and should be present in the latest 2.7.x versions.
Comment From: sushmitha-malae
Hi @sm0217 Were you able to fix the issues? we are also facing similar issues when upgraded to 2.7.16.