Micrometer 2.0.0 changes

Micrometer 2.0.0 comes with the following changes

Moving part of Micrometer Core to Micrometer API
Introduction of the Observation API
Tracing (see https://github.com/spring-projects/spring-boot/issues/30156)

Observation API

The most notable change is adding the ObservationRegistry interface. MeterRegistry implements it so there wouldn't be any change there. The thing that does change is that MeterRegistry now has a observationConfig configuration that allows to plug in ObservationHandlers.

Required by Boot AutoConfiguration

To create an Observation one has to call factory methods on Observation (start or createNotStarted). The MeterRegistry#ObservationConfig can have a list of ObservationPredicate injected to define whether for given Observation name and metadata an Observation should be created or a NoOp observation should be returned. Example use case is AutoTimer in Spring Boot. Now AutoTimer is an inbuilt feature of the Observation API.

`ObservationHandler`

An ObservationHandler allows plugging in into the lifecycle of an Observation. That means that we can react to events such as observation was started, stopped, there was an error etc.

Micrometer comes with an interface MeterObservationHandler whose default implementation is TimerObservationHandler. That in turn will create a Timer when an Observation was started and will stop it when the Observation was finished. MeterRegistry comes with a method withTimerObservationHandler that will by default create this handler. Theoretically there might be case that someone adds Micrometer but doesn't want to create a Timer when an Observation is created.

Micrometer Tracing comes with an interface TracingObservationHandler that contains additional logic for tracing related handlers.

Micrometer comes with 2 additional interfaces that group ObservationHandlers. First is ObservationHandler.FirstMatchingCompositeObservationHandler . It will group ObservationHandlers and pick the first one whose supportsContext method returns true. Second is ObservationHandler.AllMatchingCompositeObservationHandler. It will group ObservationHandlers and pick all whose supportsContext method returns true.

After having grouped the handlers they have to be injected to the ObservationRegistry via the ObservationRegistry#observationConfig()#observationHandler method.

Required by Boot AutoConfiguration

The following rules of handler groupings should take place:

All MeterObservationHandler implementations should be automatically grouped in a ObservationHandler.FirstMatchingCompositeObservationHandler.
All TracingObservationHandler implementations should be automatically grouped in ObservationHandler.FirstMatchingCompositeObservationHandler.
All ObservationHandler implementations should be grouped in ObservationHandler.AllMatchingCompositeObservationHandler (without duplicates)

Example:

We have a MeterObservationHandler bean called Meter coming from the framework
We have a TracingObservationHandler bean called Tracing coming from the framework
We have a ObservationHandler bean called Handler coming from the framework
We have a FirstMatchingCompositeObservationHandler bean called First coming from the framework
We have a AllMatchingCompositeObservationHandler bean called All coming from the framework
We have a FirstMatchingCompositeObservationHandler bean called First from the user coming from the user
We have a AllMatchingCompositeObservationHandler bean called All from the user coming from the user
We have a MeterObservationHandler bean called Meter from the user coming from the user
We have a TracingObservationHandler bean called Tracing from the user coming from the user
We have a ObservationHandler bean called Handler from the user coming from the user

We should have the following setup of beans by Boot

FirstMatchingCompositeObservationHandler created by Boot that groups all meter related handlers
MeterObservationHandler called Meter
MeterObservationHandler caled Meter from the user (order can vary depending on Ordered)
FirstMatchingCompositeObservationHandler created by Boot that groups all tracing related handlers
TracingObservationHandler called Tracing
TracingObservationHandler caled Tracing from the user (order can vary depending on Ordered)
ObservationHandler bean called Handler (order can vary depending on Ordered)
ObservationHandler bean called Handler from the user (order can vary depending on Ordered)
FirstMatchingCompositeObservationHandler bean called First (order can vary depending on Ordered)
FirstMatchingCompositeObservationHandler bean called First from the user (order can vary depending on Ordered)
AllMatchingCompositeObservationHandler bean called All (order can vary depending on Ordered)
AllMatchingCompositeObservationHandler bean called All from the user (order can vary depending on Ordered)

ObservationRegistry.observationConfig().observationHandler() will be called to register the beans presented above.

Following conditionals have to be applicable

When this feature is disabled (no Micrometer on the classpath / a property is explicitly disabled) nothing is created
FirstMatchingCompositeObservationHandler created by Boot that groups all meter related handlers
supportsContext should return false when a property to disable metrics is turned on (Micrometer on the classpath / a property for metrics enabling is explicitly disabled) - we can toggle this on and off at runtime
FirstMatchingCompositeObservationHandler created by Boot that groups all tracing related handlers
should not be created when Micrometer Tracing is not on the classpath
supportsContext should return false when a property to disable tracing is turned on (Micrometer & Micrometer Tracing on the classpath / a property for tracing enabling is explicitly disabled) - we can toggle this on and off at runtime

GlobalTagsProvider

A user can provide a GlobalTagsProvider that can be applied to all observations. We would need Boot to autowire a list of GlobalTagsProvider and put them on the ObservationRegistry via the ObservationRegistry.tagsProvider(...) method.

Comment From: wilkinsona

I've only taken a first pass through this so my thoughts probably aren't yet complete. Things I've spotted thus far:

MeterRegistry#ObservationConfig can have a list of BiPredicate<String, Observation.Context> injected. … If a BiPredicate is in your opinion not a good contract for this we can introduce a new, dedicated interface.

If we want Boot's auto-configuration to be able to find the predicates as beans and inject them into the config, I think a dedicated interface would be better. For example, it makes it easier to identify the beans the should be injected without having to worry about generics.

With regards to the composite observation handlers, what should Boot do with the 3 that it creates? I am guessing that they need to be registered with Micrometer but I didn't see that mentioned above. Apologies if I missed it. Also, what should happen with the normal, non-composite handler beans? Do they also need to be registered or should it only be the composites that are registered with Micrometer?

When this feature is disabled (no Micrometer on the classpath / a property is explicitly disabled) nothing is created

Generally speaking, we try to avoid having properties to disable features. I wonder if the classpath could be used as a signal instead but that would depend on the structure of the code in Micrometer.

With regards to the tracing bridges, do you have a feel for the dependency management implications? I imagine that we'll want to manage the version of the Brave and OTel dependencies. Are their release schedules and approach to maintenance amenable to that?

Comment From: marcingrzejszczak

If we want Boot's auto-configuration to be able to find the predicates as beans and inject them into the config, I think a dedicated interface would be better. For example, it makes it easier to identify the beans the should be injected without having to worry about generics.

Makes sense https://github.com/micrometer-metrics/micrometer/issues/3003

With regards to the composite observation handlers, what should Boot do with the 3 that it creates? I am guessing that they need to be registered with Micrometer but I didn't see that mentioned above. Apologies if I missed it. Also, what should happen with the normal, non-composite handler beans? Do they also need to be registered or should it only be the composites that are registered with Micrometer?

I've forgotten to put it there - I've updated the description. Yes, they will need to be registered in ObservationRegistry#observationConfig.

Generally speaking, we try to avoid having properties to disable features. I wonder if the classpath could be used as a signal instead but that would depend on the structure of the code in Micrometer.

That should happen for sure. The thing is that people might state that e.g. tracing is lowering performance and they want to disable just that at runtime.

Comment From: wilkinsona

This is blocked for the same reasons as https://github.com/spring-projects/spring-boot/issues/29753.

Comment From: mhalbritter

Hey @marcingrzejszczak, got a question about the grouping logic:

All ObservationHandler implementations should be grouped in ObservationHandler.AllMatchingCompositeObservationHandler

Why is this necessary? Handlers added via observationRegistry.observationConfig().observationHandler(...); are essentially in a AllMatchingCompositeObservationHandler, why wrap them again?

I imagine our code for registration something like this:

    private void registerHandlers(List<ObservationHandler<?>> observationHandlers, ObservationRegistry observationRegistry) {
        // API is forcing us to use raw types - need to create an issue for that
        List<ObservationHandler> meterHandlers = new ArrayList<>();
        List<ObservationHandler> tracingHandlers = new ArrayList<>();

        for (ObservationHandler<?> observationHandler : observationHandlers) {
            if (observationHandler instanceof MeterObservationHandler) {
                meterHandlers.add(observationHandler);
            } else if (observationHandler instanceof TracingObservationHandler) {
                tracingHandlers.add(observationHandler);
            } else {
                observationRegistry.observationConfig().observationHandler(observationHandler);
            }
        }

        if (!meterHandlers.isEmpty()) {
            observationRegistry.observationConfig().observationHandler(new FirstMatchingCompositeObservationHandler(meterHandlers));
        }
        if (!tracingHandlers.isEmpty()) {
            observationRegistry.observationConfig().observationHandler(new FirstMatchingCompositeObservationHandler(tracingHandlers));
        }
    }

Does that work or am I'm missing something?

Comment From: mhalbritter

I edited your ticket and moved the changes for Micrometer Tracing in a separate issue, https://github.com/spring-projects/spring-boot/issues/30156

Comment From: marcingrzejszczak

Yes, that's OK (I've forgotten to update the ticket with that suggestion).

Comment From: mhalbritter

Thanks for confirmation. I opened an issue regarding the use of raw types (https://github.com/micrometer-metrics/micrometer/issues/3064)

Comment From: marcingrzejszczak

I see one problem actually. Tracing might not be in the classpath. That means that if it's not then accessing the Tracing handler will lead to class not found exception

Comment From: mhalbritter

Yes, we will handle that with some @Conditionals. Thanks for the hint.

Comment From: mhalbritter

Hey @marcingrzejszczak, regarding the option to disable metrics and just leave tracing: Ideally, Micrometer's modules would be structured such that we can figure this out just from what's on the classpath. We don't really like having configuration options if we can avoid it. So if a user wants tracing, they include micrometer-tracing-api. If they don't want metrics, they don't include the micrometer-xxx dependency, whatever that may be.

Comment From: mhalbritter

For further discussions on the implementation, let's please use https://github.com/spring-projects/spring-boot/pull/30186 - this makes it easier to have them in one place and I can mark them as resolved. Thanks!

Comment From: marcingrzejszczak

If I add micrometer-tracing-api I automatically get micrometer-core so there's no way to have only tracing

Comment From: mhalbritter

And to get tracing, but not metrics, we must not call MeterRegistry.withTimerObservationHandler, correct?

Comment From: marcingrzejszczak

Yes, calling withTimerObservationHandler adds automatic metric creation. If it's not called - no metrics will be added

Comment From: mhalbritter

So, if the user really want to use tracing but not metrics and there's no way to model that with the classpath, how about this somewhere in a @Configuration in user code:

@Bean
TimerObservationHandlerMeterRegistryCustomizer timerObservationHandlerDisabled() {
  return TimerObservationHandlerMeterRegistryCustomizer.DISABLED;
}

https://github.com/spring-projects/spring-boot/pull/30186/files#diff-628ba9979d62b94641893b3587d788ee5cac335ea2e52193bb5aa047aaf59094R133

Comment From: spencergibb

What's wrong with properties? That seems like a convoluted way to disable something.

Comment From: marcingrzejszczak

I really like what you did there with the properties @mhalbritter !

Comment From: mhalbritter

Yep, added a property for it, letting user define beans for disabling features felt indeed wrong. We have to discuss in our next team meeting how we want to name them: metrics.observation feels wrong, it should be observation.metrics, but then we have inconsistencies with our existing properties under metrics.*.

Comment From: mhalbritter

PR is done from my side. Will now work on micrometer tracing integration.

Comment From: mhalbritter

We'll delay the merge until the 3.0.0-M2 release is done, as this depends on micrometer snapshots.

Comment From: mhalbritter

3.0.0-M2 is released, no longer blocking the merge.

Comment From: mhalbritter

Build for the PR is currently broken, as Spring Boot uses Micrometer 2.0.0-SNAPSHOT and Spring Batch uses Micrometer 2.0.0-M3, which is incompatible. We need to wait for https://github.com/spring-projects/spring-batch/pull/4081 to be merged, then we can build against Spring Batch 5.0.0-SNAPSHOT.

Comment From: mhalbritter

Spring Batch now builds against micrometer 2.0.0-SNAPSHOT, i've updated the PR, it now builds. No longer blocked.

Comment From: mhalbritter

We need to wait until all the projects in the release train build against micrometer 2.0.0-SNAPSHOT until we can merge this.

SpringBoot Add auto-configuration for Micrometer 2.0 Observation API

Micrometer 2.0.0 changes

Observation API

Required by Boot AutoConfiguration

ObservationHandler

Required by Boot AutoConfiguration

GlobalTagsProvider

`ObservationHandler`