We were using Spring Boot Actuator Metrics for some time now. They almost always worked flawlessly. Recently, however, we started to experience a very strange behaviour.
When we query /actuator/metrics endpoint all of the metrics are still available in the names list, however when we query the metric itself, for example /actuator/metrics/application.started.time it shows just an empty value.
What's more interesting is that the behaviour is not consistent. Restaring application sometimes helps. Even in case of different instances of the exactly the same app version; one instance can return metrics successfully, but another just empty value.
The only big change to the application itself recently was hibernate cache adjustments which increased our memory usage, but that should not be related to the failure of metrics, does it? Does anyone know if working condition of actuator metrics somehow depend on the memory limits set to the application? How can we debug this further?
We are using Spring Boot 2.7.4.
Comment From: mhalbritter
@marcingrzejszczak Does Micrometer drop metric values when under memory pressure?
Comment From: mhalbritter
@ViliusS Could you upgrade to Spring Boot 2.7.6 and see if this problem persists?
Comment From: ViliusS
@mhalbritter it would be difficult to upgrade at this very moment because we run a lot of microservices, however given the small amount of fixes between 2.7.4 and 2.7.6 I doubt it will change anything.
Anyway, today we found a workaround (maybe?). We are using opentelemetry-spring-boot-starter to auto instrument our Spring Boot applications. We found that disabling OpenTelemetry Micrometer shim (otel.springboot.micrometer.enabled=false) brings back standard metric values again.
I've read somewhere that if any 3rd party metering mechanism is enabled Spring Boot Actuator Metrics are automatically disabled. Though, this still doesn't explain why with previous configuration the metrics appeared and disappeared randomly on app restart. Also, I would have expected that if the above behaviour regarding 3rd party metric extension is true /actuator/metrics just won't list any metrics at all, not to be listed but empty.
Comment From: bclozel
@ViliusS we don't support opentelemetry-spring-boot-starter (if I'm not mistaken, this one). This is based on a Java agent and should not be applied on top of the Micrometer instrumentation in Spring Boot.
I've read somewhere that if any 3rd party metering mechanism is enabled Spring Boot Actuator Metrics are automatically disabled.
I'm not aware of any such arrangement. Can you find a reference for that? I see you've opened an issue against the OTel project. Since Spring Boot is not aware of this instrumentation, I think the behavior you're describing here is very much linked to that project. I'm closing this issue as a result. Feel free to reopen this issue if the problem turns out to be in Spring Boot.
Thanks!
Comment From: ViliusS
@bclozel I found a reference mentioned here https://docs.spring.io/spring-boot/docs/2.7.4/reference/html/actuator.html#actuator.metrics.export.simple
I went ahead and tested my theory by running an app with just management.metrics.export.simple.enabled=false. This indeed produces just empty names array in /actuator/metrics. So, unless automatic disablement takes completely different path in the code it should not be connected to running OpenTelemetry instrumentation.
My theory is that otel.springboot.micrometer.enabled=false works as a workaround because it lowers memory usage and gives head space for those Micrometer Simple backend metrics. The real issue must still be somewhere in how Micrometer Simple in-memory backend works.
Comment From: bclozel
As, I see. I think that statement is about the metrics support in Spring Boot, not necessarily the ones contributed by 3rd party. I might be confused, but it looks like this behavior is intentional in OTel. The OTel support configures itself after the simple metrics support and the Boot simple metrics auto-configurations only backs off if there's a registry already present. I find this quite odd, but I might be missing some context here.
Comment From: ViliusS
Thank you for your response @bclozel. I will try to bring this question up with OTel community but, before I'll try chasing ghost, it would be ideal to have a confirmation that Simple Micrometer backend doesn't drop metrics under memory pressure.
Just for the record, if you say opentelemetry-spring-boot-starter is not supported, what is the proper way to have OpenTelemetry auto instrumented in Spring Boot 2.x series? The only other project I found is this one https://github.com/spring-projects-experimental/spring-cloud-sleuth-otel but it is unclear if that is also supported, or if it will ever be integrate into general Spring Cloud release train.
Yes, I know Spring Boot 3.0 have OTel support out-of-the-box, however we still need an interim solution while we migrate to Java 17.
Comment From: jonatan-ivanov
The real issue must still be somewhere in how Micrometer Simple in-memory backend works.
Could you please check if it is still the SimpleMeterRegistry that is used and the OTel agent does not replace it or proxy it or instrument it in some other way? :) Because I think it does and I would guess that's where the issue lies.
what is the proper way to have OpenTelemetry auto instrumented in Spring Boot 2.x series? The only other project I found is this one https://github.com/spring-projects-experimental/spring-cloud-sleuth-otel but it is unclear if that is also supported, or if it will ever be integrate into general Spring Cloud release train.
Sleuth is for tracing, not metrics. Is your question about tracing, metrics, or both?
Metrics: Micrometer has OtlpMeterRegistry, you can use it to publish metrics in the OTel format using the Micrometer API (Spring Boot 2.x is instrumented with this API)
Tracing: Spring Cloud Sleuth Otel is in the experimental org because OTel is not really stable yet (see this and this). Moving it into the Spring Cloud release train can be a subject of future discussions (/cc @marcingrzejszczak) especially if OTel becomes stable (imho: I don't see this being likely in the timeframe of the OSS support for Sleuth). Right now I would say if you are adventurous enough to use OTel in prod, you can use Sleuth-OTel too (btw Sleuth-OTel is GA).
Comment From: ViliusS
@jonatan-ivanov you are probably right. I'm not a Java expert, but at least looking at loaded beans via Spring Boot Admin looks like OTel agent uses Micrometer Shim library which replaces SimpleMeterRegistry with OpenTelemetryMeterRegistry https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/micrometer/micrometer-1.5/library/src/main/java/io/opentelemetry/instrumentation/micrometer/v1_5 I will take this with OTel guys now.