SpringBoot Metrics aren't pushing to New Relic after migration to ObservationConvention

Hello, after migrating to SpringBoot 3.0.2 I had to rewrite the configuration for the metric's filter on a new ObservationConvention logic. I applied the filter on reactive WebClient directly, now it became observationRegistry() method. Old implementation:

import org.springframework.boot.actuate.metrics.AutoTimer;
import org.springframework.web.reactive.function.client.WebClient;

private final MeterRegistry meterRegistry;

  public WebClient client(ApiClientConfig config) {
    return WebClient.builder()
        .baseUrl(config.getBaseUrl())
        .clientConnector(new ReactorClientHttpConnector(createHttpClient(config)))
        .filter(metricsFilter())
        .build();
  }

  private MetricsWebClientFilterFunction metricsFilter() {
    return new MetricsWebClientFilterFunction(
        meterRegistry, new CustomWebClientExchangeTagsProvider(), "operation", AutoTimer.ENABLED);
  }

public class CustomWebClientExchangeTagsProvider implements WebClientExchangeTagsProvider {
  private static final String URI_TEMPLATE_ATTRIBUTE = WebClient.class.getName() + ".uriTemplate";

  @Override
  public Iterable<Tag> tags(ClientRequest request, ClientResponse response, Throwable throwable) {
    Tag uri = getUri(request);
    Tag clientName = WebClientExchangeTags.clientName(request);
    Tag status = WebClientExchangeTags.status(response, throwable);

    return Arrays.asList(uri, clientName, status);
  }

  Tag getUri(ClientRequest request) {
    String uri =
        (String) request.attribute(URI_TEMPLATE_ATTRIBUTE).orElseGet(() -> request.url().getPath());
    return Tag.of("uri", uri);
  }
}

New implementation:

private final ObservationRegistry observationRegistry;

  public WebClient client(ApiClientConfig config) {
    return WebClient.builder()
        .baseUrl(config.getBaseUrl())
        .clientConnector(new ReactorClientHttpConnector(createHttpClient(config)))
        .observationRegistry(observationRegistry)
        .observationConvention(new CustomWebClientObservationConvention())
        .build();
  }
  ```

  ```java
  public class CustomWebClientObservationConvention extends DefaultClientRequestObservationConvention
    implements GlobalObservationConvention<ClientRequestObservationContext> {

    @Override
    public KeyValues getLowCardinalityKeyValues(ClientRequestObservationContext context) {
      KeyValues lowCardinalityKeyValues = super.getLowCardinalityKeyValues(context);
      KeyValue statusKeyValue =
          lowCardinalityKeyValues.stream()
              .filter(keyValue ->
   keyValue.getKey().equals(LowCardinalityKeyNames.STATUS.asString()))
              .findAny()
              .get();
      KeyValue uriKeyValue =
          lowCardinalityKeyValues.stream()
              .filter(keyValue -> keyValue.getKey().equals(LowCardinalityKeyNames.URI.asString()))
              .findAny()
              .get();
      return KeyValues.of(statusKeyValue, uriKeyValue);
    }

    @Override
    public KeyValues getHighCardinalityKeyValues(ClientRequestObservationContext context) {
      KeyValues highCardinalityKeyValues = super.getHighCardinalityKeyValues(context);
      KeyValue clientNameKeyValue =
          highCardinalityKeyValues.stream()
              .filter(
                  keyValue ->
                      keyValue.getKey().equals(HighCardinalityKeyNames.CLIENT_NAME.asString()))
              .findFirst()
              .get();
      return KeyValues.of(clientNameKeyValue);
    }
}

Before migration, calls made using this WebClient successfully generated metrics and were pushed to NewRelic using Micrometer statsd . After migration, even though the project started and I don't see any errors in logs, metrics aren't pushing to NewRelic as they did before. I haven't found any good examples of creating some basic ObservationRegistry to work with WebClient, also I haven't found anything that replaces the old AutoTimer.ENABLED property in MetricsWebClientFilterFunction. It could be why my metrics aren't pushing and I would be grateful for any help you can provide.

My statsd configuration:

management:
    endpoint:
      metrics:
        enabled: true
    statsd:
      metrics:
        export:
          host: gostatsd.kube-system.svc
          port: 7777
          step: 1s
          flavor: datadog
          enabled: true

Comment From: wilkinsona

Why have you created your own ObservationRegistry rather than injecting the auto-configured registry? I'm not sure if it's the cause of the problem, but it's an unusual thing to be doing.

Comment From: Ardreon

Thanks for the heads up. I have changed the implementation according to your advice, but it didn't solve the problem. Forward-thinking, is it possible to define the **metricName** and **AutoTimer** of the deprecated MetricsWebClientFilterFunction() in the new implementation? I'm still feeling like the problem lies somewhere there, but haven't found in the documentation how to migrate the logic behind these parameters correctly

Comment From: Ardreon

Falsely closed

Comment From: wilkinsona

Thanks, that eliminates one possible cause.

Can you please provide a minimal sample that reproduces the problem? Unfortunately, the snippets above leave too many unknowns. For example, I would like to be able to see if the WebClient metrics are available in the Actuator's metrics endpoint and it's only an export problem or if the meters aren't being registered at all.

Comment From: wsaccentedsg

Not sure if this is related to the issue, but in order to get the tags like client_name, and http_url and etc added I had to do the below (using kotlin). My issue was with prometheus, but I hope this helps resolve your issues @Ardreon. When I used the DefaultMeterObservationHandler it did not pick up the high card values as it is not implemented in the code that way. If there is a better way to get this to work I would be happy to hear it, but there was no documentation regarding this so I just had to do the below.

create a custom DefaultMeterObservationHandler (added @component) to the class
change the "createTags" function in the custom DefaultMeterObservationHandler, like the below by using allKeyValues

private fun createTags(context: Observation.Context): Tags {
        return Tags.of(
            context.allKeyValues.stream().map { tag: KeyValue ->
                Tag.of(
                    tag.key,
                    tag.value
                )
            }.collect(Collectors.toList())
        )
    }

For meter registry in my bean class I used the below:

fun meterRegistry(): PrometheusMeterRegistry {
        return PrometheusMeterRegistry(PrometheusConfig.DEFAULT)
    }

Then created an observationRegistry function like the below:

@Bean
    fun observationRegistry(): ObservationRegistry {
        val observationRegistry = ObservationRegistry.create()
        observationRegistry.observationConfig()
            .observationHandler(CustomDefaultMeterObservationHandler(meterRegistry()))
        return observationRegistry
    }

After that I added to my client like the below:

  @Bean
    fun myAccountWebClient(
        webClientBuilder: Builder,
        myAccountClientConfig: MyAccountClientConfig,
        codecConfig: CodecConfig,
        observationRegistry: ObservationRegistry
    ): WebClient {
        return webClientBuilder
            .clientConnector(ReactorClientHttpConnector(createClient(myAccountClientConfig.connectTimeout,
                myAccountClientConfig.readTimeout)))
            .baseUrl("${myAccountClientConfig.baseUrl}/${myAccountClientConfig.path}")
            .codecs(clientCodec(codecConfig))
            .observationRegistry(observationRegistry)
            .build()
    }

Example Output:

http_client_requests_seconds_count{application="test-api",client_name="localhost",error="none",exception="none",method="GET",outcome="SUCCESS",status="200",uri="none",} 1.

Example application.yml:

spring:
  application:
    name: test-api
  mvc:
    log-resolved-exception: true
management:
  endpoints:
    web:
      exposure:
        include: '*'
  metrics:
    distribution:
      percentiles:
        http.server.requests: 0.5, 0.75, 0.95, 0.98, 1.0
        method.requests:  0.5, 0.75, 0.95, 0.98, 1.0
    tags:
      application: ${spring.application.name}

server:
  error:
    include-message: always

Comment From: Ardreon

Hello, as I had thought I had to define the name of the metrics somewhere. Adding:

  @Override
  public String getName() {
    return "operation"; // the name of my metric
  }

in my CustomWebClientObservationConvention helped to solve the issue. I think adding this point to the migration guide would be helpful for the other users migrating from the old MetricsWebClientFilterFunction. What I haven't accounted though, is that for each request, metrics would be pushed two or three times, some of them with none values. My current implementation for overriding low and high cardinality values:

  @Override
  public KeyValues getLowCardinalityKeyValues(ClientRequestObservationContext context) {
    if (context.getRequest() == null) return super.getLowCardinalityKeyValues(context);
    KeyValue statusKeyValue =
        KeyValue.of(
            LowCardinalityKeyNames.STATUS.asString(),
            Integer.toString(context.getResponse().statusCode().value()));
    KeyValue uriKeyValue =
        KeyValue.of(LowCardinalityKeyNames.URI.asString(), context.getRequest().url().getPath());
    return KeyValues.of(statusKeyValue, uriKeyValue);
  }
  ```
  ```java
    @Override
  public KeyValues getHighCardinalityKeyValues(ClientRequestObservationContext context) {
    if (context.getRequest() == null) return super.getHighCardinalityKeyValues(context);
    KeyValue clientNameKeyValue =
        KeyValue.of(
            HighCardinalityKeyNames.CLIENT_NAME.asString(),
            context.getRequest().url().getAuthority());
    return KeyValues.of(clientNameKeyValue);
  }
  ```
I have noticed that the metrics for my request are pushed multiple times, checking in debug, I noticed that these overridden methods are executed multiple times for a single request, the first one with a `null` values in `request` and `response` tabs, after that, an empty metric is pushed with `none` values. Later, for the same request, these methods receive a Context with a filled Request, from where I can take useful fields - URL, status, etc for my monitoring.
1. screenshot with an empty request in Context:
<img width="833" alt="image" src="https://user-images.githubusercontent.com/43916845/216112012-48633eba-94d6-42cb-8c1e-786b6fd720e2.png">
2. screenshot of the Context with request
<img width="833" alt="image" src="https://user-images.githubusercontent.com/43916845/216112660-a9984976-2a4a-47e4-9e50-0665f3304ecf.png">
All this happened in the scope of a single call from a WebClient, metrics pushed twice. 
Maybe I don't understand the concept of Observability to its fullest, but shouldn't these overridden methods be executed and metrics pushed only when the Context has a request?


**Comment From: bclozel**

Hello @Ardreon 

The fact that your custom convention is called twice is an expected behavior in Micrometer Observations: conventions are called [once when the observation `start()`, and once more when it's `stop()`](https://github.com/micrometer-metrics/micrometer/blob/4c00160d34d490e2599002a6e8e5d1a7405160ce/micrometer-observation/src/main/java/io/micrometer/observation/SimpleObservation.java#L140-L180). I think this is required for traces, where you need at least *some* metadata before propagating the trace. Metrics are only sent once.

This might not be a well-known fact for developers working on instrumentation - maybe you could [ask for a documentation improvement in Micrometer](https://github.com/micrometer-metrics/micrometer/issues)?

As for the second point, you're right, it doesn't really make sense to start the observation without the request being present in the context. This has been raised in spring-projects/spring-framework#29880 and will be fixed in Spring Framework directly. I'm closing this issue as a duplicate then.

Thanks for your report!

**Comment From: Ardreon**

Hello, the reason my metrics were pushed twice - is a discrepancy between the type of returned KeyValues in low and high cardinality values. I noticed that for the same request, the `Context` object used during different stages of initializing these methods is the same. Because the observation started with an empty request, in my previous implementation:
```java
  @Override
  public KeyValues getLowCardinalityKeyValues(ClientRequestObservationContext context) {
    if (context.getRequest() == null) return super.getLowCardinalityKeyValues(context);
    KeyValue statusKeyValue =
        KeyValue.of(
            LowCardinalityKeyNames.STATUS.asString(),
            Integer.toString(context.getResponse().statusCode().value()));
    KeyValue uriKeyValue =
        KeyValue.of(LowCardinalityKeyNames.URI.asString(), context.getRequest().url().getPath());
    return KeyValues.of(statusKeyValue, uriKeyValue);
  }

instead of updating low and high cardinality values with new data from Context, they were pushed to NewRelic. (because super.getLowCardinalityKeyValues(context) returns 5 other fields, and I needed only status and uri) After unifying the return type, double pushes stopped occurring. My example, I hope it will help someone:

public class CustomWebClientObservationConvention extends DefaultClientRequestObservationConvention
    implements GlobalObservationConvention<ClientRequestObservationContext> {

  private static final String UNDEFINED_VALUE = "undefined";
  private static final String UNDEFINED_STATUS_VALUE = "500";
  private static final String METRIC_NAME = "operation";

  @Override
  public String getName() {
    return METRIC_NAME;
  }

  @Override
  public KeyValues getLowCardinalityKeyValues(ClientRequestObservationContext context) {
    return KeyValues.of(status(context), uri(context));
  }

  @Override
  public KeyValues getHighCardinalityKeyValues(ClientRequestObservationContext context) {
    return KeyValues.of(clientName(context));
  }

  // TODO: In future spring versions low/high methods won't initiate when request is still null.
  //  Remove request/response is null check then
  // https://github.com/spring-projects/spring-framework/issues/29880
  @Override
  protected KeyValue status(ClientRequestObservationContext context) {
    return KeyValue.of(
        LowCardinalityKeyNames.STATUS.asString(),
        Optional.ofNullable(context.getResponse())
            .map(response -> Integer.toString(response.statusCode().value()))
            .orElse(UNDEFINED_STATUS_VALUE));
  }

  @Override
  protected KeyValue uri(ClientRequestObservationContext context) {
    return KeyValue.of(
        LowCardinalityKeyNames.URI.asString(),
        Optional.ofNullable(context.getRequest())
            .map(request -> request.url().getPath())
            .orElse(UNDEFINED_VALUE));
  }
}

In my implementation, there're no cases when uri will have an "undefined" value, although there are some rare cases when response will be null, and status will have a "500" value. The default implementation of the status method doesn't fit me, because in case of exception it returns a text, and in NewRelic there'll be a lot of nasty errors when filtering and gathering metrics, if they are of the wrong type. So, it's better to return some kind of numeric value, wrapped in String. In this caseerror field will contain WebClientRequestException, but it's no longer a problem of double pushes.

Comment From: bclozel

Thanks very much for the analysis @Ardreon - I'm sure this will be useful to others!

Comment From: Wallman

Im also having problems migrating from MetricsWebClientFilterFunction along with an AutoTimer, previously used for getting metrics with histogram percentiles for a WebClient when trying to upgrade to SB3. Is there a guide for this or does someone have any recommendations?