For a Spring Boot Application 2.5.6 we have configured Prometheus metrics endpoint /actuator/prometheus. The application is configured to expose two ports: 8080 - application port, 7080 - management port.

The problem is that when calling this /actuator/prometheus endpoint we see only metrics for endpoint exposed on management port (/health, /prometheus) instead our actual application endpoints:

HELP http_server_requests_seconds TYPE http_server_requests_seconds summary http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 2495.0 http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/prometheus",} 5.503918268 http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/health",} 52399.0 http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/health",} 68.726239855

The application is using WebFlux as web framework (in other applications when using standard WebMVC framework we don't have this issue on same Spring Boot version). We are using micrometer: <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency>

Is there something I'm missing in terms of configuration?

Comment From: wilkinsona

I cannot reproduce the behaviour that you have described. The Prometheus endpoint produces http server request metrics for both application and actuator endpoints when the two are running on separate ports:

http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics",} 2.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics",} 0.085208598
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/example",} 1.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/example",} 0.004481142
http_server_requests_seconds_count{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics/{requiredMetricName}",} 2.0
http_server_requests_seconds_sum{exception="None",method="GET",outcome="SUCCESS",status="200",uri="/actuator/metrics/{requiredMetricName}",} 0.019497845

If you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.

Comment From: rzukowski-equinix

So I prepared an example application to replicate this problem (attached).

What I forgot to mention before is that we are using WebFluxTagsContributor to add additional tags to metrics. To be specific we are searching for specific headers in HTTP requests and if they exist, we are adding them and their values to metrics.

So the example application exposes two ports: - 8080 as an application port with GET /example endpoint - 8081 as a management port with GET /actuator/prometheus endpoint

Here are my findings (for each of the below cases I restarted the application): 1. When I call both /example and /actuator/prometheus endpoint with no custom heades, many times, in whatever order, then I get all the metrics (both for /prometheus and for /example endpoints) 2. When I call /example and /actuator/prometheus endpoint both with my custom header "Some-Header" (that WebFluxTagsContributor is looking for), then I get the same behaviour as above (metrics are correct). 3. When only one of the endpoints contains this header "Some-Header" (doesn't matter which one) then from now on, the metrics will contain endpoint which was called first. For example: a. I call /example endpoint FIRST then call /actuator/prometheus - in the output I will see metrics for /example. Calling /acutator/prometheus again and again will not give me metrics for /actuator/prometheus. b. I call /acutator/prometheus FIRST then call /example - in the output I will see metrics for /acutator/prometheus. Calling /example again and again will not give me metrics for /example.

This is very weird and I'm not sure if the problem is on my side, maybe I don't understand how WebFluxTagsContributor works, but from my undertanding it should enrich default tags with the ones contributor returns.

Here is the code for tags contributor:

@Configuration
public class WebFluxMetricsConfiguration {

    @Bean
    public WebFluxTagsContributor webFluxTagsContributor() {
        return (exchange, ex) -> createHeaderTags(exchange, List.of("Some-Header"));
    }

    private Iterable<Tag> createHeaderTags(ServerWebExchange exchange, List<String> headers) {
        return headers
                .stream()
                .map(header -> createHeaderTag(exchange, header))
                .filter(Optional::isPresent)
                .map(Optional::get)
                .collect(Collectors.toList());
    }

    private Optional<Tag> createHeaderTag(ServerWebExchange exchange, String headerName) {
        return Optional
                .of(exchange.getRequest())
                .map(HttpMessage::getHeaders)
                .map(headers -> headers.getFirst(headerName))
                .map(headerValue -> Tag.of(headerName, headerValue));
    }
}

metrics-bug.zip

Comment From: wilkinsona

Thanks for the sample and the additional details. Prometheus requires that all meters with the same name have the same set of tag keys. Your tags contributor does not meet this requirement as the Some-Header tag may not always be present. Micrometer enforces Prometheus's requirement here. You may want to consider always setting that tag, even when the header is not present, for example by using a custom value to indicate the header's absence.