We ran into an issue with the PrometheusPushGateway during shutdown of our our Spring Cloud Task applications. Specifically, any task that leveraged Spring Integration would result in an error on shutdown when the shutdown operation was set to "PUSH":
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'nullChannel': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:212) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:623) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:611) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBeansOfType(AbstractApplicationContext.java:1242) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at org.springframework.integration.config.IntegrationManagementConfigurer.lambda$registerComponentGauges$1(IntegrationManagementConfigurer.java:444) ~[spring-integration-core-5.2.6.RELEASE.jar:5.2.6.RELEASE]
at io.micrometer.core.instrument.internal.DefaultGauge.value(DefaultGauge.java:54) ~[micrometer-core-1.3.8.jar:1.3.8]
at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$newGauge$3(PrometheusMeterRegistry.java:245) [micrometer-registry-prometheus-1.3.8.jar:1.3.8]
at io.micrometer.prometheus.MicrometerCollector.collect(MicrometerCollector.java:69) ~[micrometer-registry-prometheus-1.3.8.jar:1.3.8]
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190) ~[simpleclient-0.7.0.jar:na]
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223) ~[simpleclient-0.7.0.jar:na]
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144) ~[simpleclient-0.7.0.jar:na]
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22) ~[simpleclient_common-0.7.0.jar:na]
at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:310) ~[simpleclient_pushgateway-0.7.0.jar:na]
at io.prometheus.client.exporter.PushGateway.pushAdd(PushGateway.java:182) ~[simpleclient_pushgateway-0.7.0.jar:na]
at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusPushGatewayManager.push(PrometheusPushGatewayManager.java:108) ~[spring-boot-actuator-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusPushGatewayManager.shutdown(PrometheusPushGatewayManager.java:146) ~[spring-boot-actuator-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at org.springframework.boot.actuate.metrics.export.prometheus.PrometheusPushGatewayManager.shutdown(PrometheusPushGatewayManager.java:136) ~[spring-boot-actuator-2.2.7.RELEASE.jar:2.2.7.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_252]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_252]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_252]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_252]
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:339) ~[spring-beans-5.2.6.RELEASE.jar:5.2.6.RELEASE]
After investigating, you can see that the "push" event is being triggered by a "pre-destroy" event on the PrometheusGatewayManager and that, in turn, is attempting to get meter values from singleton beans that have already been destroyed. To fix this, the shutdown can instead be triggered from a ContextClosedEvent (prior to any beans being destroyed)
This is similar to what was discussed in this stack overflow thread:
https://stackoverflow.com/questions/52785517/beancreationnotallowedexception-from-predestroy-annotation-from-prometheuspushg
Comment From: tkvangorder
This fixed a majority of our issues, however, there is still an edge case where the ScheduledFuture is in mid-execution (on a different thread) and it encounters the same problem. We really need to cancel any future scheduled events, wait for the last scheduled event to complete, and then continue with the shutdown. This manifests in an exception being written to the log @ Warning level from micrometer but appears to be harmless.
Comment From: snicoll
@tkvangorder thank you for the report and the PR. The proposed fix seems a bit strange to me considering that shutdown is inferred as a bean destroy method so it would be called (perhaps not at the right time). I'd like to understand the problem a bit more. Could you share a small sample that demonstrates the problem? Given that we have no test to exercise the scenario you've described, I am not keen to make that change as is.
Comment From: tkvangorder
@snicoll I will work to get a small sample to you.
Comment From: tkvangorder
@snicoll I have a sample that reproduces the problem here: https://github.com/tkvangorder/spring-task-issue
Comment From: tkvangorder
I also included the code we are currently using to get around the problem. The readme in that project has more details. Thanks!
Comment From: wilkinsona
Thanks for the sample, @tkvangorder.
Spring Integration registers 3 gauges where the value is calculated by calling getBeansOfType() on the application context for MessageChannel, MessageHandler, and MessageSource beans. It does so in such a way that there are no bean dependency relationships between the meter registry and the MessageChannel, MessageHandler, and MessageSource beans. During context close processing, this allows those beans to be destroyed before the meter registry is destroyed. As a result query the gauges in the meter registry can result in bean creation being attempted and a BeanCreationNotAllowedException is then thrown.
I don't think Spring Boot is the right place to fix this. The problem may occur in any application that uses Spring Integration and Micrometer where an attempt is made to retrieve metrics from the meter registry during application context close processing. I've opened https://github.com/spring-projects/spring-integration/issues/3376 so that the Integration team can take a look.
Thanks anyway for the PR.