Met with a team today that is configuring two PrometheusMeterRegistry
instances, one for platform metrics and the other for application metrics. They expose two different scrape endpoints from the application. The platform metrics should include framework-level metrics like CPU, GC, http request timings, etc. The application metrics endpoint is reserved for teams to record business metrics.
Both of these registries wind up under the CompositeMeterRegistry
Spring Boot creates. (I hope you enjoy my senseless ascii art ;) ).
Composite
----------/ \
/ \
Prometheus (platform) Prometheus (app)
The problem is that PropertiesMeterFilter
is added to the composite in addition to both sub-registries. So if an app team adds a property like management.metrics.enable.tomcat=false
the composite denies Tomcat metrics before it ever reaches the platform registry.
I suggest we weaken PropertiesMeterFilter
and only apply it to sub-registries. In this way, a user-crafted registry could add meter filters that take precedence over PropertiesMeterFilter
, like this:
@Bean
PrometheusMeterRegistry platformRegistry() {
PrometheusMeterRegistry prometheusMeterRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
prometheusMeterRegistry
.config()
.meterFilter(MeterFilter.acceptNameStartsWith("tomcat"))
.meterFilter(MeterFilter.acceptNameStartsWith("jvm"));
return prometheusMeterRegistry;
}
cc / @wangyf2010 cc / @shakuzen
Comment From: philwebb
I'm a little worried that MeterRegistryConfigurer
can make the assumption that a CompositeMeterRegistry
can be skipped because it is only composed of other beans. The user could register their own CompositeMeterRegistry
which is composed of instance that we have no visibility on.
Perhaps we need a AutoConfiguredCompositeMeterRegistry
class so that we know it's the one that we auto-configured and it only contains beans?
Comment From: jkschneider
I agree @philwebb, I think that makes sense.
Comment From: jkschneider
Thanks for the quick action @wilkinsona. Could you try this out @wangyf2010?