Hi,
I recently noticed that our Spring based applications publish a lot of metrics and traced this back to the lettuce metrics taking a large portion of up to 50-60%. (E.g ~7800 of ~14800 metrics on a single app server that go over the wire are just generated by this within 60 seconds)
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
prefix.lettuce.command.completion.percentile.command.CLIENT.local.local:any.phi.0.5.remote....
The list goes on and on with the different percentiles and their respective histogram data for the various commands. (I've stripped the IPs and actual values btw. but I hope it's clear nonetheless)
27865 added the metrics autoconfiguration for Lettuce. Unfortunately, with the histogram option being set to true which is not in line with the default from Lettuce.
This PR adds the option to enable the histogram on demand and aligns the default value to false. Given that this breaks behaviour it's probably something for 3.x, but I could imagine seeing a backport to 2.7.x where the default is simply set to true in the new MetricsProperties$Lettuce class. A backport would be at least highly appreciated.
Let me know what you think. Cheers, Christoph
Comment From: wilkinsona
Thanks, @dreis2211. I consider the mismatched default for generating histograms to be a bug. I don't think we can fix that bug directly in 2.x, but I think it would be reasonable to introduce a property so that it's easier to switch it back. We can then flip the default in 3.0 as you suggest.
Comment From: wilkinsona
Directly related to histograms, there's also no option for configuring the target percentiles. Beyond that, min latency, max latency, local distinction, and tags also can't be configured. Tags can't really be configured via properties, but the others can. That's quite a big change for 2.x though. Rather than introducing the properties now, I wonder if we'd be better making MicrometerOptions a bean and allowing users to provide their own.
Comment From: dreis2211
@wilkinsona Thought about opening up all of the available options, but decided against it to increase the likelihood of the change being backported. 3.x is unfortunately far away for most of these applications.
But it would be definitely better to offer all of the configuration options in the future
Comment From: wilkinsona
If we want to do something for 2.6.x and 2.7.x, it could just be this:
diff --git a/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/redis/LettuceMetricsAutoConfiguration.java b/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/redis/LettuceMetricsAutoConfiguration.java
index d7490d76b5..1cebf5ca54 100644
--- a/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/redis/LettuceMetricsAutoConfiguration.java
+++ b/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/redis/LettuceMetricsAutoConfiguration.java
@@ -27,6 +27,7 @@ import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.AutoConfigureBefore;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
+import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.data.redis.ClientResourcesBuilderCustomizer;
import org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration;
import org.springframework.context.annotation.Bean;
@@ -47,8 +48,13 @@ import org.springframework.context.annotation.Configuration;
public class LettuceMetricsAutoConfiguration {
@Bean
- public ClientResourcesBuilderCustomizer lettuceMetrics(MeterRegistry meterRegistry) {
- MicrometerOptions options = MicrometerOptions.builder().histogram(true).build();
+ @ConditionalOnMissingBean
+ MicrometerOptions lettuceMicrometerOptions() {
+ return MicrometerOptions.builder().histogram(true).build();
+ }
+
+ @Bean
+ public ClientResourcesBuilderCustomizer lettuceMetrics(MeterRegistry meterRegistry, MicrometerOptions options) {
return (client) -> client.commandLatencyRecorder(new MicrometerCommandLatencyRecorder(meterRegistry, options));
}
An alternative would be to do nothing for the maintenance releases but that would require LettuceMetricsAutoConfiguration to be excluded in favour of your own ClientResourcesBuilderCustomizer bean that adds a command latency recorder.
Comment From: wilkinsona
We're going to make MicrometerOptions a bean. We'll flip the default for histogram in 3.0.0-RC2.
Comment From: wilkinsona
Closing in favor of https://github.com/spring-projects/spring-boot/issues/32985. Thanks for bringing this to our attention, @dreis2211.