https://github.com/spring-projects/spring-boot/pull/21296 describes a use case for streaming a response from an actuator web endpoint rather than buffering the whole response in memory and then writing it out. While it's already possible with a web-stack-specific endpoint, such as @RestControllerEndpoint, it would be good to have something that's web-stack-agnostic that can be returned from an endpoint operation as Resource, for example, can be used today.
Comment From: bric3
Hi I just came from #21296, I started to use spring actuators endpoint to expose prometheus metrics. However I noticed a noticeable increase in humongous allocations (G1GC), almost all due to spring related classes. This is causing regular unresponsiveness in the application(s) (pauses up-to ~100ms).
This is with Spring Boot 2.2.8.
There's a few Spring Framework in cause too. Should I open a ticket there ?
I can share the flamegraph privately, as there is other private flames triggering humongous allocations.
Comment From: wilkinsona
Three of the humongous allocations appear to be related to Prometheus so they should be addressed by the enhancement that's proposed here. The other two appear to be released to HTTP message conversion in Spring Framework. I think it would be worth opening a Framework issue for those so that they can investigate.
Comment From: bric3
OK let's do it. Thanks for the quick feedback.
Comment From: bric3
While the stack is pointing to springboot actuator, it leverages the prometheus simpleclient TextFormat.write004. I think the issue could be resolved by passing a StringWriter with enough capacity maybe as a function of the number of metrics. And by not returning a String, but then the StringWriter may be not the right type form the start.
Comment From: ITman1
Still an issue with new Spring and Spring boot, having this payload on our side is definitelly a weird/bug, but still Spring could handle streaming in better way:
Comment From: bclozel
@ITman1 are you sure this is related? The data share was about flame graphs showing that writing large responses was taking significant CPU time because of large allocations and GC pauses. Here you seem to share a snapshot of heap memory that sticks around at runtime.
Could you elaborate on what your graph is showing and your interpretation here?
Comment From: ITman1
Yes, it is related, high CPU time and GC pauses are symtoms of too large payloads/responses allocated on Heap due to usage of byte[] (ByteArrayOutputScream) in Prometheus Scrape endpoint. Heap dump is actually crash dump which was dumped after reaching OutOfMemoryError. Anyway I just found a root cause why huge payloads were even created - https://github.com/spring-projects/spring-amqp/issues/2914.
I am only noting here that crash place was in Spring Boot Prometheus Scape Endpoint due to lack streaming:
So I am only suggesting performance improvement here as this usually means issue somewhere else, now in Spring AMQP, but still in standard cases Prometheus scape endpoint might produce payloads around 1MB, reaching already half of G1 region size when having <4GB of heap and I think it is better to stream such payloads instead of puting those on heap.
Comment From: bclozel
We'll consider this performance improvement. As for the Spring AMQP issue, this looks like a cardinality explosion to me, so not strictly related to large responses. Even with small responses, unbounded metric tags will overflow memory, which explains the OutOfMemoryError.
Comment From: marschall
While the stack is pointing to springboot actuator, it leverages the prometheus simpleclient
TextFormat.write004. I think the issue could be resolved by passing a StringWriter with enough capacity maybe as a function of the number of metrics. And by not returning aString, but then theStringWritermay be not the right type form the start.
The latest Prometheus Java client versions should avoid the single char writes to OutputStreamWriter
https://github.com/prometheus/client_java/pull/1241 https://github.com/prometheus/client_java/pull/1248