It seems that 3.3.0 introduced a regression during the migration of Prometheus 1.x: exemplars are missing from _count time series.

I think there are two bugs that are causing this: 1. It seems PrometheusPropertiesConfigAdapter does not respect the Micrometer defaults for "Prometheus Properties" (these) when there are no user-defined custom properties set: https://github.com/spring-projects/spring-boot/blob/82d8222f56361079daae64eec4056fe4d7ce6bc9/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/export/prometheus/PrometheusPropertiesConfigAdapter.java#L65-L74 It seems Micrometer defaults (PrometheusConfig.super.prometheusProperties()) are only respected and merged with user-defined custom properties when they present, otherwise Micrometer defaults are ignored. This is a problem since Micrometer sets some defaults which can be ignored or not ignored depending on the presence of user-defined properties. 2. Ignoring Micrometer defaults could have been worked around by manually setting them from user-properties (or by setting any custom Prometheus property) but another bug prevents them to take effect on the exporter. It seems PrometheusOutputFormat does not use these properties when it initializes the exporters: https://github.com/spring-projects/spring-boot/blob/82d8222f56361079daae64eec4056fe4d7ce6bc9/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/metrics/export/prometheus/PrometheusOutputFormat.java#L81 Since PrometheusOutputFormat calls ExpositionFormats.init() instead of ExpositionFormats.init(ExporterProperties), no property set by Micrometer/Boot/user can have any effect on the exporters since they are ignored (in comparison, Micrometer does this).

One workaround could be using the Prometheus property loading mechanism. Adding a prometheus.properties file to the classpath for example placing it to the resources folder with the following content brings exemplars on _count back:

io.prometheus.exporter.exemplarsOnAllMetricTypes=true

In order to reproduce it, you need to ask for the OpenMetrics format from Boot:

http :8080/actuator/prometheus 'Accept: application/openmetrics-text; version=1.0.0' | grep 'trace_id'

and you will need Micrometer's (1.13.x) Prometheus (1.x) registry and Micrometer Tracing: start.spring.io example

Comment From: mhalbritter

Hello Jonatan,

for 1: fromPropertiesMap is used in 

public Properties prometheusProperties() {
  return get(this::fromPropertiesMap, PrometheusConfig.super::prometheusProperties);
}

and this get method does this:

protected final <V> V get(Function<T, V> getter, Supplier<V> fallback) {
  V value = getter.apply(this.properties);
  return (value != null) ? value : fallback.get();
}

So if getter returns null, it invokes fallback.

With our invocation, if fromPropertiesMap returns null, it uses super.prometheusProperties().

Looking at fromPropertiesMap:

Map<String, String> map = prometheusProperties.getProperties();
if (map.isEmpty()) {
  return null;
}

So if there are no user-defined properties, we return null, and get will use super.prometheusProperties().

This looks fine to me, or have I overlooked something?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

Comment From: mhalbritter

The bot reminded me that there's something pending. I gave it another look, and it looks like that point 1 that @jonatan-ivanov has raised isn't valid, but the 2nd is.

I have something in https://github.com/mhalbritter/spring-boot/tree/mh/40904-prometheus-exemplars-are-missing-from-_count, but I need to play around with it a bit more.

Comment From: mhalbritter

I also added a smoke test to verify that it doesn't break in the future.

Comment From: jonatan-ivanov

Sorry for the late response, you are right about PrometheusProperties, I checked it in the ctor of PrometheusMeterRegistry, the property is there, sorry for the confusion.

I looked at your fix in PrometheusScrapeEndpoint, it looks great, I have a quite minor comment for this section: https://github.com/spring-projects/spring-boot/blob/7d96789869497c29881f694b4c58a835966d7733/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/metrics/export/prometheus/PrometheusScrapeEndpoint.java#L73-L78

What do you think about renaming expositionFormatsProperties to exporterProperties since that's the name Prometheus itself uses for these properties, see prometheusProperties.getExporterProperties() and the docs.

I also checked the latest snapshots, it works, thank you very much!

Comment From: mhalbritter

Thanks for the feedback! Sounds good, will rename the parameter.