Hello,
We are facing an issue with Spring metrics not being uploaded to datadog. After debugging with help of datadog support team, the problem seems to be that the percentile metrics emitted by SpringBoot are not in the standard prometheus format and hence datadog can't read them properly. More details are below -
We are using spring boot version 2.2.6.RELEASE in our project. The below setting is enabled
management.metrics.distribution.percentiles.all=0.5,0.9,0.95,0.99
management.metrics.distribution.percentiles-histogram.all=true
Once server is started with above settings, the metrics are published in this format.
# HELP http_server_requests_seconds
# TYPE http_server_requests_seconds histogram
http_server_requests_seconds{exception="None",method="DELETE",outcome="SUCCESS",status="204",uri="/v1/data/{type}/{id}",quantile="0.5",} 0.0
http_server_requests_seconds_bucket{exception="None",method="DELETE",outcome="SUCCESS",status="204",uri="/v1/data/{type}/{id}",le="0.001",} 0.0
http_server_requests_seconds_count{exception="None",method="DELETE",outcome="SUCCESS",status="204",uri="/v1/data/{type}/{id}",} 127.0
http_server_requests_seconds_sum{exception="None",method="DELETE",outcome="SUCCESS",status="204",uri="/v1/data/{type}/{id}",} 0.96087724
Per prometheus histogram definition, the base metric with name http_server_requests_seconds is not expected for a histogram type. This is throwing off datadog collectors and causing to ignore the metric, as they expect it to follow the standard prometheus definition for a histogram.
Is it possible to fix the metrics to follow the standard prometheus formats when percentiles are used?
Regards, Aditya
Comment From: wilkinsona
Thanks for the report. Spring Boot isn't really involved at this level. It passes the histogram configuration to Micrometer and it and the Prometheus library are then responsible for creating the text format that Prometheus scrapes.
Spring Boot 2.2 is out of OSS support so, first of all, it would be worth upgrading to a supported version of Spring Boot (the latest 2.3.x or 2.4.x release). This will also upgrade to Micrometer 1.5 (Boot 2.3) or 1.6 (Boot 2.4) and pick up any fixes that have been made there. If the problem persists, we'll need to get some help from the Micrometer team (/cc @shakuzen).
Comment From: AdityaBhansali1
@wilkinsona : Thanks for the feedback. We will try with latest version and see if it helps and provide an update here. If updating doesn't solve the problem, do you suggest moving this issue to another repo, or can we continue the discussion here?
Comment From: snicoll
@AdityaBhansali1 after a quick chat with @shakuzen, he suggests to follow up on the micrometer issue tracker. Unfortunately, we can't move this issue across orgs so you'll have to recreate it.