Micrometer allows to monitor the executor states using ExecutorServiceMetrics
ref: https://github.com/micrometer-metrics/micrometer/blob/master/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/jvm/ExecutorServiceMetrics.java
It allows to monitor the usage of executors with metrics like:
"async.executor",
"async.executor.active",
"async.executor.completed",
"async.executor.idle",
"async.executor.pool.core",
"async.executor.pool.max",
"async.executor.pool.size",
"async.executor.queue.remaining",
"async.executor.queued",
To achieve that we have to wrap our executors with something like this:
/**
* <p>A default task executor.</p>
* <p>The {@link Executor} instance to be used when processing async
* method invocations.</p>
*/
@Bean(DEFAULT_TASK_EXECUTOR_BEAN_NAME)
@Override
public Executor getAsyncExecutor() {
// Using a thread-pooling TaskExecutor implementation,
// in particular for executing a large number of short-lived tasks.
final ThreadPoolTaskExecutor executor = new TaskExecutorBuilder()
.corePoolSize(100) // With unlimited queue
.allowCoreThreadTimeOut(true)
.threadNamePrefix("task-")
.build();
executor.initialize();
return ExecutorServiceMetrics.monitor(
meterRegistry,
executor.getThreadPoolExecutor(),
"AsyncExecutor",
"async",
tags);
}
(I am creating the tags from MetricsProperties#getTags)
It might be great if an AutoConfiguration could do it magically (it's why we love Spring-Boot). Not sure how it could be implemented in a Spring-Boot way properly.
We can find such request in various places:
- https://stackoverflow.com/questions/56176852/how-to-enable-executorservicemetrics-in-springboot-2-1-2
- https://github.com/micrometer-metrics/micrometer/issues/1430
cc @snicoll
Comment From: snicoll
Thanks for the suggestion. I think it would be worthwhile to have a MeterBinder for TaskExecutor and TaskScheduler as those have a different hierarchy structure. I've triaged this for our next feature release.
Comment From: aheritier
I agree @snicoll it makes sense to me. thanks a lot
Comment From: aheritier
My approach seems to be wrong @snicoll I just discovered that I am loosing the default metrics from spring boot
I also notice that it added a lot of warnings is not eligible for getting processed by all BeanPostProcessors
2020-10-29 19:41:53.110 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration' of type [org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdMetricsExportAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.117 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'management.metrics.export.statsd-org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdProperties' of type [org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.117 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'statsdConfig' of type [org.springframework.boot.actuate.autoconfigure.metrics.export.statsd.StatsdPropertiesConfigAdapter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.118 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration' of type [org.springframework.boot.actuate.autoconfigure.metrics.MetricsAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.118 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'micrometerClock' of type [io.micrometer.core.instrument.Clock$1] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.120 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'statsdMeterRegistry' of type [io.micrometer.statsd.StatsdMeterRegistry] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-10-29 19:41:53.124 INFO 71216 --- [ restartedMain] trationDelegate$BeanPostProcessorChecker : Bean 'management.metrics-org.springframework.boot.actuate.autoconfigure.metrics.MetricsProperties' of type [org.springframework.boot.actuate.autoconfigure.metrics.MetricsProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
Something like this is helping
// https://stackoverflow.com/questions/57607445/spring-actuator-jvm-metrics-not-showing-when-globalmethodsecurity-is-enabled
// Do not loose default metrics
@Bean
InitializingBean forcePostProcessor(BeanPostProcessor meterRegistryPostProcessor) {
return () -> meterRegistryPostProcessor.postProcessAfterInitialization(meterRegistry, "");
}
I still have the post processing errors but the the default metrics are back
Comment From: snicoll
@aheritier yes, we're not going to implement that feature this way. Spring Boot rather injects the component (i.e. thread pool) later on and register metrics on it, which avoids causing unnecessary early initialisations due to the dependency on the MeterRegistry. If you have questions about registering custom metrics, please follow-up on StackOverflow or come chat with the community on Gitter.
Comment From: aheritier
It's good I better understand now. Because I also implements AsyncConfigurer
Your approach using a MeterBinder similar to what is done for Cache and others will be the best approach
Thanks
Comment From: snicoll
I've added a proposal in 910e14f. Unfortunately, it is too late for 2.5.x but I'll do my best to get that in for 2.6.0-M1. It handles both task executors and task schedulers using Micrometer's ExecutorServiceMetrics.
Comment From: aheritier
Awesome. Thanks a lot @snicoll
Comment From: aheritier
@snicoll I imported your code in my 2.5.x codebase and it's working fine.
I just had to be sure that my Beans were defined as ThreadPoolTaskExecutor and not simple Executor
Well done!
Comment From: snicoll
Thanks for testing Arnaud. Your feedback on the bean type is interesting, I might improve the auto-configuration to take that into account transparently.
Comment From: aheritier
yes not sure if you can make it transparent. For sure I was a bit confused because it wasn't working at the beginning with my Executors then I read the code and the doc to understand that using ThreadPoolTaskExecutor was the solution
Comment From: csdbianhua
@snicoll I have a question after reading https://github.com/spring-projects/spring-boot/commit/910e14ffae0218236899cebb61c853d9fae3840d. I need to use the TimedExecutorService returned by ExecutorServiceMetrics#monitor to wrap the tasks I submitted, then I can get task execution time (executor.execution) and idle time (executor.idle). If you just iterate over ThreadPoolTaskExecutor and invoke ExecutorServiceMetrics#monitor, won't these metrics be lost?
Am I missing something?
Comment From: snicoll
Those two are unrelated, I think. You can give 2.6.0-SNAPSHOT a try. If that doesn't work, feel free to create a new issue with a small sample that allows us to reproduce the problem in the form of a sample project and we can take a look.
Comment From: artemik
@snicoll Hi, what's eventually the status of it? Is there a documentation on how to properly setup monitoring of task executors?
Comment From: bclozel
Yes this has been available since 2.6.0. Auto-configuration (https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/task/TaskExecutorMetricsAutoConfiguration.java) and docs (https://docs.spring.io/spring-boot/reference/actuator/metrics.html#actuator.metrics.supported.tasks) are present.