We did spring boot version updates in the last weeks from 2.2 to 2.3 to 2.4.

Since version 2.3 we can see huge negative values for the tomcatGlobalRequest metric in our graphite (v1.0.2).

This negative values (peaks) appear when we deploy (restart) our application via CI pipeline

tomcatGlobalRequest_negative-metric (see screenshot - blue dotted line is the deployment marker which is set when the service is up again).

We already tried to use the graceful shutdown configuration introduced with spring boot 2.3 with no success.

System setup: Service is Running on debian stretch 9.13 (VM) jdk: zulu-11 Spring Boot 2.4.2 kotlin 1.4.21 micrometer-core/graphite-registry 1.6.4 (provided by spring dependency management plugin 1.0.11.RELEASE)

Expected behavior: No negative values are measured in graphite when reboot a spring boot application

Comment From: snicoll

@PH1P5 thanks for the report. Would be able to share a small sample that reproduces the behavior you've described that work with 2.2 but does not by changing the version to 2.3? You can do so by attaching a zip to this issue or by sharing a link to a GitHub repository.

Comment From: PH1P5

@snicoll I created a sample project to reproduce this behavior with spring boot 2.4. For instruction how to reproduce please see the README.

Thanks in advance for your support.

Comment From: wilkinsona

Thanks for the sample, @PH1P5. I think this is happening when tomcat.global.request drops to zero. You can simulate this happening with JConsole by invoking the resetCounters() operation on the GlobalRequestProcessor MBean.

tomcat.global.request will drop to zero if Micrometer fails to query Tomcat's MBean. Any failure to retrieve a long results in 0 being returned:

   private long safeLong(Callable<Object> callable) {
        try {
            return Long.parseLong(callable.call().toString());
        } catch (Exception e) {
            return 0;
        }
    }

My theory is that a failure sometimes occurs when the application is being shut down. If Tomcat deregisters the MBean that Micrometer is using before Micrometer's scheduler that's publishing the metrics has been stopped, the attempt to query the MBean will fail and zero will be returned for the metric.

Comment From: wilkinsona

When the request count drops to zero this code in Micrometer will increment the rate with a negative value.

@shakuzen Can you please take a look at this one? Is this behaviour a Micrometer bug?

Comment From: shakuzen

Thank you for the report and project to reproduce, @PH1P5. Thank you for the analysis @wilkinsona. We can take over handling this issue in Micrometer. I've created micrometer-metrics/micrometer#2489 for that. Returning zero (after a value greater than zero has been returned) is breaking the contract of a monotonically increasing function; so we'll need to figure out what to do about this.

Comment From: wilkinsona

Thanks, Tommy. Closing in favour of micrometer-metrics/micrometer#2489.