Spring Webflux applications do not report exemplars in their Prometheus Metrics endpoint.
To reproduce
- start.spring.io -> create a webflux application with observability, prometheus, security and actuator
- Add a simple test controller or something else to hit
- Run your application with
management.endpoints.web.exposure.include=* - Hit your endpoint a couple times
- Check
curl -H 'Accept: application/openmetrics-text; version=1.0.0' localhost:8080/actuator/prometheus | grep trace_id
You will see no results are returned.
The same issue does not occur on servlet.
Sample
https://github.com/braunsonm/spring-boot-exemplars-issue
Comment From: jonatan-ivanov
TL;DR: try to enable histograms.
management.metrics.distribution.percentiles-histogram.http.server.requests=true
or, for everything:
management.metrics.distribution.percentiles-histogram.all=true
If you are interested in the details:
Prometheus implemented this based on the OpenMetrics specification. OpenMetrics does not support Exemplars for Summary and Histogram only supports it on the buckets. The reason why you don't see any exemplar is because the Observation (Micrometer) will create a Timer (Micrometer) that will create a Summary (Prometheus). If you enable histogram, you will see exemplars on its buckets. Also, if you create a Counter and/or emit an .event(...) on the Observation, you should also see Exemplars since counters are supported via OpenMetrics/Prometheus.
Future developments: we discussed this with the Prometheus team and implemented support for any time series (already merged). Once Prometheus will support this, we should implement support in Micrometer so you will see Exemplars on Prometheus summaries.
I'm closing this issue because in order to do this, there is no need to change anything in Boot (other than maybe a property to turn it on off).
Comment From: braunsonm
Interesting! @jonatan-ivanov Do you know why servlet and reactive are different? That property doesn't seem to be required on servlet
PS: Do you have an issue for micrometer supporting exemplars on Summary and Histogram that I can track?
Comment From: jonatan-ivanov
They should not be, you should see the same behavior: no exemplars unless you turn on histogram or create a Counter.
Right now there is a PR for count and sum but I think sum is not needed and this should be implemented slightly differently. Also, there is an issue for max.
Comment From: braunsonm
You might want to reopen this issue then because that is not the case. The following are lists of metric names which have exemplars.
Servlet without property:
spring_security_filterchains_DisableEncodeUrlFilter_after_total
spring_security_filterchains_HeaderWriterFilter_after_total
spring_security_filterchains_SecurityContextHolderFilter_after_total
spring_security_filterchains_WebAsyncManagerIntegrationFilter_after_total
spring_security_filterchains_ExceptionTranslationFilter_after_total
spring_security_filterchains_BasicAuthenticationFilter_before_total
logback_events_total
spring_security_filterchains_DefaultLogoutPageGeneratingFilter_before_total
spring_security_filterchains_BasicAuthenticationFilter_after_total
spring_security_filterchains_DefaultLoginPageGeneratingFilter_after_total
spring_security_filterchains_CorsFilter_before_total
spring_security_filterchains_LogoutFilter_before_total
spring_security_filterchains_UsernamePasswordAuthenticationFilter_before_total
spring_security_filterchains_SecurityContextHolderFilter_before_total
spring_security_filterchains_DefaultLoginPageGeneratingFilter_before_total
spring_security_filterchains_HeaderWriterFilter_before_total
spring_security_filterchains_DefaultLogoutPageGeneratingFilter_after_total
spring_security_filterchains_CsrfFilter_after_total
spring_security_filterchains_UsernamePasswordAuthenticationFilter_after_total
spring_security_filterchains_AnonymousAuthenticationFilter_before_total
spring_security_filterchains_AuthorizationFilter_after_total
spring_security_filterchains_SecurityContextHolderAwareRequestFilter_after_total
spring_security_filterchains_WebAsyncManagerIntegrationFilter_before_total
spring_security_filterchains_CsrfFilter_before_total
spring_security_filterchains_AnonymousAuthenticationFilter_after_total
spring_security_filterchains_SecurityContextHolderAwareRequestFilter_before_total
spring_security_filterchains_CorsFilter_after_total
spring_security_filterchains_AuthorizationFilter_before_total
spring_security_filterchains_ExceptionTranslationFilter_before_total
spring_security_filterchains_LogoutFilter_after_total
spring_security_filterchains_DisableEncodeUrlFilter_before_total
spring_security_filterchains_RequestCacheAwareFilter_before_total
spring_security_filterchains_RequestCacheAwareFilter_after_total
Webflux without property
<NONE>
Servlet with property
spring_security_filterchains_AuthorizationFilter_before_total
spring_security_filterchains_SecurityContextHolderFilter_after_total
spring_security_filterchains_RequestCacheAwareFilter_before_total
spring_security_filterchains_LogoutFilter_after_total
spring_security_filterchains_UsernamePasswordAuthenticationFilter_after_total
spring_security_filterchains_CorsFilter_after_total
spring_security_authorizations_seconds_bucket
spring_security_authorizations_seconds_bucket
spring_security_authorizations_seconds_bucket
spring_security_filterchains_ExceptionTranslationFilter_after_total
spring_security_filterchains_SecurityContextHolderAwareRequestFilter_before_total
spring_security_filterchains_AnonymousAuthenticationFilter_before_total
spring_security_filterchains_UsernamePasswordAuthenticationFilter_before_total
spring_security_filterchains_CorsFilter_before_total
spring_security_filterchains_DefaultLogoutPageGeneratingFilter_before_total
spring_security_filterchains_AnonymousAuthenticationFilter_after_total
spring_security_filterchains_CsrfFilter_before_total
spring_security_filterchains_WebAsyncManagerIntegrationFilter_before_total
spring_security_filterchains_SecurityContextHolderFilter_before_total
spring_security_filterchains_WebAsyncManagerIntegrationFilter_after_total
spring_security_filterchains_HeaderWriterFilter_before_total
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_AuthorizationFilter_after_total
spring_security_filterchains_DisableEncodeUrlFilter_after_total
spring_security_filterchains_DefaultLoginPageGeneratingFilter_after_total
spring_security_authentications_seconds_bucket
spring_security_authentications_seconds_bucket
spring_security_filterchains_BasicAuthenticationFilter_after_total
spring_security_filterchains_DisableEncodeUrlFilter_before_total
spring_security_filterchains_HeaderWriterFilter_after_total
spring_security_filterchains_BasicAuthenticationFilter_before_total
spring_security_filterchains_DefaultLogoutPageGeneratingFilter_after_total
spring_security_filterchains_CsrfFilter_after_total
spring_security_filterchains_SecurityContextHolderAwareRequestFilter_after_total
spring_security_filterchains_RequestCacheAwareFilter_after_total
spring_security_filterchains_DefaultLoginPageGeneratingFilter_before_total
logback_events_total
spring_security_filterchains_ExceptionTranslationFilter_before_total
spring_security_http_secured_requests_seconds_bucket
spring_security_filterchains_LogoutFilter_before_total
Webflux with property
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_filterchains_seconds_bucket
spring_security_http_secured_requests_seconds_bucket
spring_security_authentications_seconds_bucket
spring_security_authentications_seconds_bucket
spring_security_authorizations_seconds_bucket
spring_security_authorizations_seconds_bucket
spring_security_authorizations_seconds_bucket
There is definitely some inconsistent behaviour here where the Observation's are missing exemplars. Please reopen when you have a chance! @jonatan-ivanov
Comment From: jonatan-ivanov
Servlet without property: I think this seems ok. Time series in Prometheus post-fixed by _total are counters which means they can have exemplars (see above). I'm not sure if Boot has to do anything with Spring Security creating or not creating those counters in case of MVC/WebFlux. I think that inconsistency should be reported to Spring Security.
Webflux without property: This is fishy but I'm not sure if it is your project setup or Spring Security not creating these counters (Observation Events).
@jzheaux Do you happen to know it similar observation events are emitted in terms of MVC and WebFlux? It seems there are no events for WebFlux (might be intentional, I don't remember).
Servlet with property: At this point I'm very suspicious that something is wrong with your project setup. Here's a sample where you can see this working: https://github.com/micrometer-metrics/micrometer-samples/tree/main/micrometer-samples-boot3-web Do you happen to ignore the http server Observations in your project? :)
Webflux with property: Same as servlet, the same samples repo I linked above has a webflux sample too.
Comment From: braunsonm
I think that inconsistency should be reported to Spring Security.
I suppose I could but I don't think that Spring Security controls the creation of exemplars, isn't that handled by Micrometer?
Do you happen to ignore the http server Observations in your project? :)
Ha-ha, but no I am not ignoring any Observations in this project, the only code is provided above.
I'm not sure if it is your project setup
At this point I'm very suspicious that something is wrong with your project setup.
I really don't understand these comments. The project could not be simpler and is provided above. The sample application you linked actually does more than this simple start.io project does. What more can I do to convince you that this is a problem we are facing? Can this issue be reopened?
Comment From: braunsonm
Just to make this extremely clear, I created a repo here that demonstrates the issue: https://github.com/braunsonm/spring-boot-exemplars-issue
Comment From: jonatan-ivanov
I suppose I could but I don't think that Spring Security controls the creation of exemplars, isn't that handled by Micrometer?
I did not say Spring Security controls the creation of exemplars, I said if there is an inconsistency between Spring Security counters in case of an MVC app and a WebFlux app, that should be reported to Spring Security.
Ha-ha, but no I am not ignoring any Observations in this project, the only code is provided above.
👍🏼
I really don't understand these comments. The project could not be simpler and is provided above. The sample application you linked actually does more than this simple start.io project does. What more can I do to convince you that this is a problem we are facing? Can this issue be reopened?
From here, I don't know exactly what you are doing. What I know is what I think you are trying to do should work and we have tests that should cover a subset of it. So I'm guessing what could go wrong. The sample app I linked is actually simpler from a point of view: it does not have Spring Security. Having lots of controller methods does not really make a difference.
Just to make this extremely clear, I created a repo here that demonstrates the issue: https://github.com/braunsonm/spring-boot-exemplars-issue
Thank you very much! This helps a lot to reproduce the issue.
I looked into your app and there is no way you can see exemplars on http_server_requests_... because you forgot to add histogram support (see above):
management.metrics.distribution.percentiles-histogram.all=true
After I added it, exemplars were still missing (that's an issue!), but it seems this has nothing to do with Spring Security, if I remove it, exemplars are still missing. Though if I use brave instead of otel:
micrometer-tracing-bridge-brave
instead of
micrometer-tracing-bridge-otel
I have exemplars. It seems the cause of this is when we try to get the current span, it is null. I debugged this a little and it seems the span is null right after we put it into scope here. I feel this issue is not in Boot but in Micrometer/OTel. Let me reopen it till I figure out what's the problem.
In the meantime, could you please double check the Servlet with property scenario? We have tests for that one you should see exemplars on http_server_requests_....
Comment From: jonatan-ivanov
/cc @marcingrzejszczak It seems we have an issue with creating the scope with OTel (it seems Brave works).
Comment From: braunsonm
and there is no way you can see exemplars on http_server_requests_... because you forgot to add histogram support
That was done on purpose to show how all 4 of the different scenarios are inconsistent with eachother. It isn't just about http_server_requests being missing.
In the meantime, could you please double check the Servlet with property scenario?
The issue remains. You can test this on the sample by changing the webflux starter to web and removing Mono in the controller.
It seems we have an issue with creating the scope with OTel (it seems Brave works).
Interesting! Thanks for reopening!
Comment From: jonatan-ivanov
Since it seems the root cause is in Micrometer Tracing, I opened an issue there: https://github.com/micrometer-metrics/tracing/issues/207 (it also references a fix which I'm not 100% satisfied with right now but hopefully we can improve it).
Right now as a workaround, you can either:
- Use Brave instead of OTel
- You can override TracingAwareMeterObservationHandler and apply the current fix (it's a one-liner)