Hi,
After upgrading from Spring Boot 3.0.5 to 3.0.6 I've been running out of heap in production in a matter of seconds. Narrowing this down I can see this is due to a build up of io.micrometer.observation.SimpleObservation objects.
I've distilled the problem down and have a couple of examples
1) https://github.com/davidmelia/spring-cloud-function-zipkin/tree/memory_leak_webclient shows that with the combination of WebFlux 3.0.6 + Spring Security + enableAutomaticContextPropagation if you run the app and ping http://localhost:8080/memoryLeak you get the leak
2) https://github.com/davidmelia/spring-cloud-function-zipkin/tree/memory_leak_webclient2 is slightly different in that you get the leak without spring security if you try and hit http://localhost:8080/memoryLeak async i.e.
curl http://localhost:8080/memoryLeak & curl http://localhost:8080/memoryLeak & curl http://localhost:8080/memoryLeak
3) https://github.com/davidmelia/spring-cloud-function-zipkin/tree/spring_security_memory_leak shows that with spring security and simply spamming the actuator health endpoint you get a memory leak. Simply run spamHealthCheck.sh
In all cases you will notice io.micrometer.observation.SimpleObservation is never totally reclaimed by garbage collection and builds up over time:
N.B I have previously raised https://github.com/spring-projects/spring-boot/issues/34695 which is similar in symptoms but different objects are not garbage collected.
Potential Fix: For anyone having similar problems downgrading micrometer to the spring boot 3.0.5 versions seems to fix it (obviously this can be dangerous downgrading certain spring dependencies)
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-commons</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-jetty11</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-observation</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-observation-test</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-appoptics</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-atlas</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-azure-monitor</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-cloudwatch</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-cloudwatch2</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-datadog</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-dynatrace</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-elastic</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-ganglia</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-graphite</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-health</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-humio</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-influx</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-jmx</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-kairos</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-new-relic</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-opentsdb</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-otlp</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-signalfx</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-statsd</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-wavefront</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-test</artifactId>
<version>1.10.5</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-integration-test</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-reporter-wavefront</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-test</artifactId>
<version>1.0.3</version>
</dependency>
</dependencies>
</dependencyManagement>
Comment From: wilkinsona
Thanks for the report and analysis, @davidmelia. I've asked the Observability team to take a look.
Comment From: marcingrzejszczak
@davidmelia after merging this https://github.com/micrometer-metrics/micrometer/pull/3788 I checked the 3 branches from your sample and the memory leak seems to be gone. Can you please wait for the latest snapshots of micrometer and double check things on your end?
Comment From: davidmelia
@marcingrzejszczak I have pulled down 1.10.7-SNAPSHOT (1.10.7-20230425.040713-7) but cannot see those changes in the source - has the micrometer build run?
Comment From: marcingrzejszczak
Ehhhhhhhhhh it's in 1.11.0-SNAPSHOT, I merged the PR to main instead of 1.10.x :facepalm:. I've just pushed the change to 1.10.x. I did check the change locally yesterday with proper versions. Sorry for the confusion.
Comment From: davidmelia
@marcingrzejszczak I've tested on 1.11.0-SNAPSHOT (1.10.x simply didn't download anything new) and looks like it's fixed the problem.
Thanks for this.
Comment From: marcingrzejszczak
Thanks for being so thorough with testing this. Hopefully we got rid of all the memory leaks for now. @wilkinsona this can closed in favour of https://github.com/micrometer-metrics/micrometer/pull/3788