Bug description
The ObservationEnvironmentRepositoryObservationConvention sets the tags for environment class, profile, application, and label on the observations only when the respective values are non-blank.
This is a problem for the Prometheus registry as Prometheus expects the same set of tags to be set with every observation. When an observation with a different set of tags compared to previous observations is registered, the new observation will be dropped.
An example would be a client request that asks for specific label after previously only requests without a specific label have been requested. When the Prometheus registry is used, the requests for a specific label will be dropped in this scenario.
Steps to reproduce
A full minimal complete verifiable example can be found here: https://github.com/hpoettker/config-server-reproducer
In short, a reproducing app can be created as follows:
- create a new project with the following dependencies:
- Spring Cloud Config Server
- Spring Boot Starter Actuator
- Prometheus Registry
- add
@EnableConfigServer
- expose the actuator's Prometheus end-point in the
application.yml
Then, start up the application in the profile native
, and execute the following requests:
curl http://localhost:8080/application/default
followed by
curl http://localhost:8080/application/default/label
The client will receive the expected responses but the the following warning will be logged:
The meter (MeterId{name='spring.cloud.config.environment.find.active', tags=[tag(spring.cloud.config.environment.application=application),tag(spring.cloud.config.environment.class=org.springframework.cloud.config.server.environment.SearchPathCompositeEnvironmentRepository),tag(spring.cloud.config.environment.label=label),tag(spring.cloud.config.environment.profile=default)]}) registration has failed:
Prometheus requires that all meters with the same name have the same set of tag keys.
There is already an existing meter named 'spring.cloud.config.environment.find.active' containing tag keys [spring.cloud.config.environment.application, spring.cloud.config.environment.class, spring.cloud.config.environment.profile].
The meter you are attempting to register has keys [spring.cloud.config.environment.application, spring.cloud.config.environment.class, spring.cloud.config.environment.label, spring.cloud.config.environment.profile].
Note that subsequent logs will be logged at debug level.
When checking the response of the end-point for the meter spring_cloud_config_environment_find_active
, the second request with the label is missing from the statistics.