SpringBoot Counts in ClassicHistogramBuckets cannot be negative issue after upgrade to spring boot 3.3

We are seeing a similar issue to https://github.com/spring-projects/spring-boot/issues/40552 after upgrading to spring boot 3.3 (it contains already the micrometer 1.13.0), I've tried even the workaround mentioned in the referenced issue without any help.

Stacktrace:

j.l.IllegalArgumentException: Counts in ClassicHistogramBuckets cannot be negative.
    at i.p.m.m.s.ClassicHistogramBuckets.validate(ClassicHistogramBuckets.java:147)
    at i.p.m.m.s.ClassicHistogramBuckets.sortAndValidate(ClassicHistogramBuckets.java:108)
    at i.p.m.m.s.ClassicHistogramBuckets.of(ClassicHistogramBuckets.java:53)
    at i.m.p.PrometheusMeterRegistry.lambda$addDistributionStatisticSamples$29(PrometheusMeterRegistry.java:504)
    at i.m.p.MicrometerCollector.collect(MicrometerCollector.java:77)
    at i.p.m.m.r.PrometheusRegistry.scrape(PrometheusRegistry.java:72)
    at i.p.m.m.r.PrometheusRegistry.scrape(PrometheusRegistry.java:57)
    at o.s.b.a.m.e.p.PrometheusScrapeEndpoint.scrape(PrometheusScrapeEndpoint.java:59)
    at j.i.r.GeneratedMethodAccessor128.invoke(Unknown Source)
    at j.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at j.l.reflect.Method.invoke(Method.java:568)
    at o.s.u.ReflectionUtils.invokeMethod(ReflectionUtils.java:281)
    at o.s.b.a.e.i.r.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
    at o.s.b.a.e.a.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
    at o.s.b.a.e.w.s.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:327)
    at o.s.b.a.e.w.s.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:434)
    at j.i.r.GeneratedMethodAccessor110.invoke(Unknown Source)
    at j.i.r.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at j.l.reflect.Method.invoke(Method.java:568)
    at o.s.w.m.s.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)
    at o.s.w.m.s.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)
    at o.s.w.s.m.m.a.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
    at o.s.w.s.m.m.a.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)
    at o.s.w.s.m.m.a.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)
    at o.s.w.s.m.m.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
    at o.s.b.a.a.w.s.CompositeHandlerAdapter.handle(CompositeHandlerAdapter.java:58)
    at o.s.w.s.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
    at o.s.w.s.DispatcherServlet.doService(DispatcherServlet.java:979)
    at o.s.w.s.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
    at o.s.w.s.FrameworkServlet.doGet(FrameworkServlet.java:903)
    ... 29 frames truncated

We are observing this error several times per day on our production after the upgrade. The application is standard REST application using Mongo DB and calling external services via feign client, where we are creating some custom metrics with histograms enabled. The metric is updated using code looking like this:

        meterRegistry.counter(counter_metric_name, tags.stream().collect(Collectors.toList()).increment();
        Timer.builder(timer_metric_name)
                .tags(tags.stream().collect(Collectors.toList()))
                .publishPercentileHistogram()
                .register(meterRegistry).record(duration);

Regarding prometheus (if I don't count the attempt with the workaround config), we have there only management.endpoint.prometheus.enabled: true option defined. We use spring-boot in combination with jetty, some dependencies which might be relevant: =>

org.springframework.boot:spring-boot-starter-web:jar:3.3.0:compile
org.springframework.boot:spring-boot-starter-jetty:jar:3.3.0:compile
io.micrometer:micrometer-core:jar:1.13.0:compile
io.micrometer:micrometer-registry-prometheus:jar:1.13.0:compile
io.micrometer:micrometer-tracing-bridge-brave:jar:1.3.0:compile

I wasn't able to reproduce the issue outside of the production environment. Any suggestions for a fix or what could have caused it?

Note with Spring boot 3.2.6 this issue wasn't present.

Comment From: mhalbritter

This looks similar to https://github.com/micrometer-metrics/micrometer/issues/4988 - could you please open an issue on the Micrometer issue tracker for that?

Comment From: rhatlapa

Thanks, I've tried with downgraded micrometer version and I was no longer able to reproduce the issue thus confirming the issue is in micrometer, here is the filed issue there in case someone would stumble on this report: https://github.com/micrometer-metrics/micrometer/issues/5193