Once we've upgraded some of our services to 3.3.0 we observed some of our services failing after multiple hours at runtime with the following exception; rendering the scraping of metrics broken.
I'm trying to create a reproduceable test case in the meantime; hoping the stacktrace already helps you guys to pinpoint the problem.
java.lang.ClassCastException: class io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot cannot be cast to class io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot (io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot and io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot are in unnamed module of loader org.springframework.boot.loader.launch.LaunchedClassLoader @34c4973)
io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeClassicHistogramBuckets(OpenMetricsTextFormatWriter.java:125)
io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeHistogram(OpenMetricsTextFormatWriter.java:120)
io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.write(OpenMetricsTextFormatWriter.java:72)
org.springframework.boot.actuate.metrics.export.prometheus.PrometheusOutputFormat$2.write(PrometheusOutputFormat.java:64)
org.springframework.boot.actuate.metrics.export.prometheus.PrometheusScrapeEndpoint.scrape(PrometheusScrapeEndpoint.java:60)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:281)
org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:327)
org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:434)
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
org.springframework.boot.actuate.autoconfigure.web.servlet.CompositeHandlerAdapter.handle(CompositeHandlerAdapter.java:58)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108)
org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231)
org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:479)
org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:340)
org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:82)
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:128)
org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:137)
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:240)
org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:227)
Comment From: knoobie
Test Case; multiple methods annotated with @Timed, the same value to group them together and enable histogramonly on some.. This worked in previous versions.
@Service
public class TimedAnnotationService {
private final AtomicInteger atomicInteger = new AtomicInteger();
@Timed(value = "timed.increase")
public int timedIncrease() {
return atomicInteger.incrementAndGet();
}
@Timed(value = "timed.increase", histogram = true)
public int timedHistogramIncrease() {
return atomicInteger.incrementAndGet();
}
}
Test:
@ExtendWith(SpringExtension.class)
@SpringBootTest
@AutoConfigureObservability
class MicrometerTest {
@Autowired
private MeterRegistry meterRegistry;
@Autowired
private TimedAnnotationService timedService;
@Test
void testMicrometerTimedHistogramAnnotation() throws Exception {
assertThat(timedService.timedIncrease()).isEqualTo(1);
assertThat(timedService.timedHistogramIncrease()).isEqualTo(2);
((PrometheusMeterRegistry) meterRegistry).scrape("application/openmetrics-text; version=1.0.0; charset=utf-8");
}
}
Results in:
java.lang.ClassCastException: class io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot cannot be cast to class io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot (io.prometheus.metrics.model.snapshots.SummarySnapshot$SummaryDataPointSnapshot and io.prometheus.metrics.model.snapshots.HistogramSnapshot$HistogramDataPointSnapshot are in unnamed module of loader 'app')
at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeClassicHistogramBuckets(OpenMetricsTextFormatWriter.java:125)
at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.writeHistogram(OpenMetricsTextFormatWriter.java:120)
at io.prometheus.metrics.expositionformats.OpenMetricsTextFormatWriter.write(OpenMetricsTextFormatWriter.java:72)
at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:167)
at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:163)
at io.micrometer.prometheusmetrics.PrometheusMeterRegistry.scrape(PrometheusMeterRegistry.java:136)
Comment From: wilkinsona
This can be reproduced without any involvement from Spring Boot:
package com.example;
import io.micrometer.core.instrument.Timer;
import io.micrometer.core.instrument.Timer.Sample;
import io.micrometer.prometheusmetrics.PrometheusConfig;
import io.micrometer.prometheusmetrics.PrometheusMeterRegistry;
public class Reproducer {
public static void main(String[] args) throws InterruptedException {
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(new PrometheusConfig() {
@Override
public String get(String key) {
return null;
}
});
timed(registry, false, "one");
timed(registry, true, "two");
System.out.println(registry.scrape("application/openmetrics-text; version=1.0.0; charset=utf-8"));
}
static void timed(PrometheusMeterRegistry registry, boolean histogram, String methodName) throws InterruptedException {
Sample sample = Timer.start(registry);
Thread.sleep(500);
sample.stop(Timer.builder("example").publishPercentileHistogram(histogram).tag("method-name", methodName).register(registry));
}
}
Please report the problem to the Micrometer project.
Comment From: knoobie
Thanks Andy!