When we upgraded from spring boot release 2.2.5 to 2.3.2 we noticed out MultiGauge metrics stopped working as expected. After getting some initial values for a set of tags, they never changed so we ended up with flat dashboard graphs even though we could see from our application logs that the gauges were updated multiple times to new values.

Our application is running java 11 (openjdk).

I have tested below versions so it seems the issue have been introduced with the 2.2.7 release: 2.2.5 -> works 2.2.6 -> works 2.2.7 -> fails 2.3.0 -> fails 2.3.1 -> fails 2.3.2 -> fails

The problem we observe is the following. We have several MultiGauge meters which we update periodically using the content of a Map with AtomicInteger where the Metadata type contains a unique set of tags we wish to report the metric with. 

private static final String TAG1 = "tag1";
private static final String TAG2 = "tag2";
private static final String TAG3 = "tag3";

private Map<Metadata, AtomicInteger> pendingChanges;
private MultiGauge gauge;

...
// Called periodically and the content of the pendingChanges map is slightly different on each iteration.
this.gauge.register(pendingChanges.entrySet().stream().map(t -> MultiGauge.Row.of(
                Tags.of(
                        TAG1, t.getKey().tag1,
                        TAG2, t.getKey().tag2,
                        TAG3, t.getKey().tag3),
                t.getValue())).collect(Collectors.toList()), true);

Furthermore, we use micrometer-registry-prometheus so that our metrics scraper can extract metrics from the /actuator/prometheus endpoint. However, what we observe is that the overwrite:true flag of above MultiGauge.register(...) no longer appears to work as expected. Once we have reported a unique set of tags with e.g. tag1="a", tag2="b", tag3="c" and value Atomicinteger(37), it appears we are no longer able to overwrite this value. It stays 37 for the lifetime of the application even though we make multiple attempts to overwrite it for the same set of tags.

We have not been able to reproduce the problem in our unit tests, so it appears to work when we update the gauges multiple times and after verify the values from the global metric registry using: Metrics.globalRegistry.find("metricName")

The affected 'fronzen' metric values are seen on both /actuator/prometheus as well as /actuator/metrics/, so it does not appear to be specific to prometheus, but rather appears like the internal in-memory metric registry for some reason won't be updated to the new values.

Comment From: wilkinsona

Thanks for the report. MultiGauge is part of Micrometer which is managed as a separate project. We upgraded to Micrometer 1.3.8 in Spring Boot 2.2.7 and I suspect that this is the source of the change in behaviour. Can you please open a Micrometer issue so that the team can investigate?

If it turns out that a change is necessary is Spring Boot to fix the problem we can re-open this issue.

Comment From: bertiljeppsson

upgraded to Micrometer 1.3.8

Thanks, I moved the issue there https://github.com/micrometer-metrics/micrometer/issues/2250