StackOverflowError observed when running a Spring Boot 3.0.2 app with micrometer tracing/metrics and logback metrics enabled - it appears to be linked to lazy loading of the Tracer within LazyTracingSpanContextSupplier which results in a recursive loop of logging and counter incrementing.
Disabling logback metrics in application.yml resolves the issue
Spring boot version 3.0.2
pom.xml snippet
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>3.0.2</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
...
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</dependency>
stack trace snippet
at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan(PrometheusExemplarsAutoConfiguration.java:91)
at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.isSampled(PrometheusExemplarsAutoConfiguration.java:82)
at io.prometheus.client.exemplars.DefaultExemplarSampler.doSample(DefaultExemplarSampler.java:46)
at io.prometheus.client.exemplars.DefaultExemplarSampler.sample(DefaultExemplarSampler.java:34)
at io.micrometer.prometheus.PrometheusCounter.updateExemplar(PrometheusCounter.java:79)
at io.micrometer.prometheus.PrometheusCounter.increment(PrometheusCounter.java:58)
at io.micrometer.core.instrument.Counter.increment(Counter.java:38)
at io.micrometer.core.instrument.binder.logging.MetricsTurboFilter.decide(LogbackMetrics.java:205)
at ch.qos.logback.classic.spi.TurboFilterList.getTurboFilterChainDecision(TurboFilterList.java:49)
at ch.qos.logback.classic.LoggerContext.getTurboFilterChainDecision_0_3OrMore(LoggerContext.java:251)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:375)
at ch.qos.logback.classic.Logger.log(Logger.java:780)
at org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog.trace(LogAdapter.java:473)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:256)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:254)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.addCandidateEntry(DefaultListableBeanFactory.java:1628)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1585)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1368)
at org.springframework.beans.factory.support.DefaultListableBeanFactory$DependencyObjectProvider.getObject(DefaultListableBeanFactory.java:2002)
at org.springframework.util.function.SingletonSupplier.get(SingletonSupplier.java:97)
at org.springframework.util.function.SingletonSupplier.obtain(SingletonSupplier.java:115)
at org.springframework.boot.actuate.autoconfigure.tracing.prometheus.PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan(PrometheusExemplarsAutoConfiguration.java:91)
Comment From: wilkinsona
Thanks for the report. I've asked the observability team to take a look.
Comment From: jonatan-ivanov
It's not in the repro steps but out of the box this works to me (in many sample/example/test apps). In order to reproduce the issue, I needed to make the app log something out triggered by the PrometheusExemplarsAutoConfiguration$LazyTracingSpanContextSupplier.currentSpan call.
For example, if I set logging.level.org.springframework.beans.factory.support=TRACE, I am able to repro the issue, without it, it works.
We are looking into how to fix it, in the meantime another workaround could be modifying some of your log levels.
Comment From: wilkinsona
This is going to be addressed in Micrometer. Please subscribe to https://github.com/micrometer-metrics/micrometer/issues/3623 for updates.