As part of our effort to clean up vulnerabilities, we upgraded our Spring Boot application from the latest 2.X to 3.X version.
We use Spring Actuator for monitoring and managing our application in various production environments. In our application.properties, we have a liveness health group configured as follows:
management.health.livenessstate.enabled=true
management.endpoint.health.group.liveness.include=livenessState,rabbitMqState
management.endpoint.health.group.liveness.show-details=always
While the 'livenessState' probe is provided by Spring Actuator, we have a custom health indicator for RabbitMQ represented by the following bean:
@Bean
public RabbitMqStateHealthIndicator rabbitMqStateHealthIndicator(RabbitMqEndpoint endpoint) {
return new RabbitMqStateHealthIndicator(endpoint);
}
The RabbitMqStateHealthIndicator is defined in an external library and implements the HealthIndicator interface:
@RequiredArgsConstructor
@Slf4j
public class RabbitMqStateHealthIndicator implements HealthIndicator {
private final RabbitMqEndpoint endpoint;
@Override
public Health health() {
return new Health.Builder()
.status(getStatus())
// Looks like "java.util.concurrent.ThreadPoolExecutor@34e1411[Running, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 3]"
.withDetail("executor", endpoint.getMessageDispatcher().getTaskExecutor().getThreadPoolExecutor().toString())
.withDetail("isQueueInitializingFailed", endpoint.isQueueInitializingFailed())
.withDetails(getConnectionDetails())
.withDetails(getSessionDetails())
.build();
}
private Status getStatus() {
JmsConnection connection = endpoint.getConnection();
Status result = Status.UP;
// Consider dispatcher healthy while we are disconnected (session.isClosed) to avoid unnecessary restarts.
boolean isDispatcherHealthy = endpoint.getSession().isClosed() ||
endpoint.getMessageDispatcher().allConsumerThreadsAreAlive();
if (connection == null ||
connection.isFailed() || // If we have unlimited reconnect attempts, isFailed will always be false
endpoint.isQueueInitializingFailed() ||
!isDispatcherHealthy) {
result = Status.DOWN;
log.warn("""
Returning {} status in health indicator \
[connection is null? {}, connection.isFailed? {}, \
isQueueInitializingFailed? {}, isDispatcherHealthy? {}]\
""",
result,
connection == null,
connection != null && connection.isFailed(),
endpoint.isQueueInitializingFailed(),
isDispatcherHealthy);
}
return result;
}
private Map<String, ?> getConnectionDetails() {
return Optional.ofNullable(endpoint.getConnection()).map(connection -> Map.of(
"connection.isClosed", connection.isClosed(), // seems to always be false
"connection.isFailed", connection.isFailed(), // true: when all reconnection attempts failed
"connection.isConnected", connection.isConnected(), // false: when all reconnection attempts failed
"connection.isStarted", connection.isStarted() // false: when all reconnection attempts failed
))
.orElse(Collections.emptyMap());
}
private Map<String, ?> getSessionDetails() {
return Optional.ofNullable(endpoint.getSession())
// true: when trying to reconnect or when all reconnection attempts failed
.map(session -> Collections.singletonMap("session.isClosed", session.isClosed()))
.orElse(Collections.emptyMap());
}
}
Before the Spring Boot upgrade, everything worked perfectly. However, after upgrading to Spring Boot 3.X, I encounter the following error during application startup:
error - APPLICATION FAILED TO START
Description:
Included health contributor 'rabbitMqState' in group 'liveness' does not exist
Action:
Update your application to correct the invalid configuration.
You can also set 'management.endpoint.health.validate-group-membership' to false to disable the validation.
Given that this configuration worked seamlessly in Spring Boot 2.X, I'm uncertain why it's failing now in 3.X.
Questions:
-
Is there a new way to register or reference custom health indicators in Spring Boot 3.X?
-
Are there any additional configurations or steps required for Actuator to recognize custom health indicators in the new version?
Any insights or suggestions would be greatly appreciated.
Comment From: wilkinsona
It looks like the rabbitMqStateHealthIndicator bean hasn't actually been defined. Perhaps it's defined in an auto-configuration that hasn't been updated to use the new imports file? Unfortunately, the snippets above leave too many unknowns so I can't tell whether this is the case.
Nothing else that I can recall has changed in this area. If the above doesn't help and you would like us to spend some more time investigating, please spend some time providing a complete yet minimal sample that reproduces the problem. You can share it with us by pushing it to a separate repository on GitHub or by zipping it up and attaching it to this issue.
Comment From: NimrodGolan
I didn't provide the implementation of the configuration class that defines the rabbitMqStateHealthIndicator bean.
it uses @ConditionalOnProperty(name = "rabbitmq.enabled", havingValue = "true"). I guess it could be related, but on the other hand before the migration I had not problem to start the application with rabbitmq.enabled=false and had no indication for errors.
Comment From: wilkinsona
Since Spring Boot 3.1, health group membership has been validated by default. You can disable this validation by setting management.endpoint.health.validate-group-membership to false.
We recommend upgrading in small steps and reading the release notes for each step as you go. For example, if you're upgrading from 2.7 to 3.3, you should upgrade to 3.0, then 3.1, then 3.2, and lastly to 3.3.
I'm going to close this one now as I can't see any sign of a problem with Spring Boot. If following the advice above does not help and you would like us to spend some more time investigating, please provide the requested sample and we can re-open the issue and take another look. Beyond this, if you have further questions, please follow up on Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.