Recently, there have been a lot of issues with the liveness and readiness probes in Spring Boot, see for instance https://github.com/spring-projects/spring-boot/issues/22562
After I switched from 2.3.1 to 2.3.2, I had to include the groups explicitly in my application.yml, like so:
management:
endpoints:
web:
exposure:
include: '*'
endpoint:
health:
group:
liveness:
include: livenessStateProbeIndicator
readiness:
include: readinessStateProbeIndicator
This would make the liveness and readiness probes show up in /actuator/health and make them available under /actuator/health/liveness and /actuator/health/readiness respectively.
With version 2.3.3, I can remove these explicit groups and configure it like so:
management:
endpoint:
health:
probes:
enabled: true
health:
livenessstate:
enabled: true
readinessstate:
enabled: true
Now, if you look at /actuator/health, you see the groups liveness and readiness listed. So far, so good.
But here's the kicker, the URLs /actuator/health/liveness and /actuator/health/readiness don't work and return a 404! After stepping through the actuator code in my debugger, I found that under the hood, these groups are referenced as 'livenessState' and 'readinessState' respectively.
So I tried the URLs /actuator/health/livenessState and /actuator/health/readinessState and they both worked!
I don't know if this is a misunderstanding on my part, or whether the groups listed in health need to be renamed or the endpoints need to be made available under the old names again.
Comment From: snicoll
But here's the kicker, the URLs /actuator/health/liveness and /actuator/health/readiness don't work and return a 404!
What you've done in your configuration above is to enable the individual health indicators but that's not going to enable the groups. You should remove the workaround for the issue that got fixed in 2.3.3 now that you've upgraded to that. The only thing to get those groups to show up is management.endpoint.health.probes.enabled, as documented in the reference guide.
If is currently only documented in the Kubernetes section and I wonder if we shouldn't change that and make it a bit more explicit outside of that context.
Comment From: Davio
I had added that to my application.yml, but skipped over it in copy-pasting it to my post in this issue, I added it above for clarification. But the issue remains the same.
To clarify my conclusion: the groups are enabled and listed in /health, but they aren't available at the path I would expect them. I would expect them under /liveness and /readiness, but they are made available under /livenessState and /readinessState instead. So the names of the groups as they are listed in /health are different from the actual endpoints.
The documentation you refer to specifies:
They will also be exposed as separate HTTP Probes using Health Groups: "/actuator/health/liveness" and "/actuator/health/readiness"
But that is simply not true as they are exposed as "/actuator/health/livenessState" and "/actuator/health/readinessState"
Comment From: snicoll
But that is simply not true as they are exposed as "/actuator/health/livenessState" and "/actuator/health/readinessState"
I can't reproduce what you're describing. I can however reproduce exactly what is described in the doc using the configuration you've shared. To make some progress, please share a small sample that we can run ourselves, either attached as a zip to this issue or in the form of a GitHub repo.
Comment From: Davio
Haven't been able to reproduce it with a simple demo yet, but in my simplified demo, both the endpoints /liveness and /livenessState are working and return the same result.
In my actual application, it seems that something is dropping / blocking the /liveness endpoint and only /livenessState still works.
Okay, I finally found it, I overlooked something. I still had
management:
endpoint:
health:
group:
liveness:
include: livenessStateProbeIndicator
readiness:
include: readinessStateProbeIndicator
floating around somewhere in a profile and this messes up the paths and oddly enough makes /liveness not work, but still lets /livenessState work.
So if you remove these 2.3.2 workarounds, everything should be working as it should again. Maybe it can't hurt to put this in the release notes as a migration tip?
Comment From: bclozel
I've carefully read this thread and it seems that #22562 created a lot of confusion and frustration, I'm really sorry about that. I'm not aware of other issues on that matter.
This issue seems to really be about the pain of applying the workaround and removing it afterwards. All the behavior described here (including the 404 when a configured group has no existing indicator) has been discussed already in #22562.
I've updated the release notes to underline that issue. I don't think we should update other types of documentation as it would be even more confusing for developers.
@Davio I'd really like to know if removing the workaround fixed this problem for you. Are there remaining issues with the probes?
Comment From: YatishKumar
I have the same issue, in version 2.3.1. RELEASE I was able to access /actuator/health/liveness and /actuator/health/readiness but after changing to 2.3.3. RELEASE I changed my application.yml file as below
management:
endpoint:
health:
probes:
enabled: true
health:
livenessstate:
enabled: true
readinessstate:
enabled: true
/actuator/health i receive below JSON response
{
"status": "UP",
"components": {
"diskSpace": {
"status": "UP",
"details": {
"total": *******,
"free": *******,
"threshold": ********,
"exists": true
}
},
"livenessStateProbeIndicator": {
"status": "UP"
},
"ping": {
"status": "UP"
},
"readinessStateProbeIndicator": {
"status": "UP"
}
},
"groups": Array[2][
"liveness",
"readiness"
]
}
but when I request the URLs /actuator/health/liveness and /actuator/health/readiness don't work and return a 404! I even tried "/actuator/health/livenessState" and "/actuator/health/readinessState" but i get 404!
/actuator/health/livenessStateProbeIndicator and /actuator/health/readinessStateProbeIndicator returns 200 with status UP
Comment From: Davio
Removing the configuration which we needed to add in 2.3.2 fixed my issue.
I just found it weird that /livenessState worked in all cases with all configurations and /liveness only worked after removing the 2.3.2 configuration.
I still don't understand why that happens exactly. I would have expected that the probes would either work under /liveness or not at all. That they can appear under /livenessState instead, surprised me
Op di 25 aug. 2020 12:29 schreef Brian Clozel notifications@github.com:
I've carefully read this thread and it seems that #22562 https://github.com/spring-projects/spring-boot/issues/22562 created a lot of confusion and frustration, I'm really sorry about that. I'm not aware of other issues on that matter.
This issue seems to really be about the pain of applying the workaround and removing it afterwards. All the behavior described here (including the 404 when a configured group has no existing indicator) has been discussed already in #22562 https://github.com/spring-projects/spring-boot/issues/22562.
I've updated the release notes to underline that issue https://github.com/spring-projects/spring-boot/releases/tag/v2.3.3.RELEASE. I don't think we should update other types of documentation as it would be even more confusing for developers.
@Davio https://github.com/Davio I'd really like to know if removing the workaround fixed this problem for you. Are there remaining issues with the probes?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/spring-projects/spring-boot/issues/23035#issuecomment-679942600, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH633ZNGRG5KUCJ6JFNJZ3SCOG77ANCNFSM4QHA3GBQ .
Comment From: snicoll
I just found it weird that /livenessState worked in all cases with all configurations and /liveness only worked after removing the 2.3.2 configuration.
Rather than letting those groups have their default configuration, that explicit override was requesting the inclusion of livenessStateProbeIndicator. Such a bean doesn't exist in 2.3.3. The group is empty and therefore 404 when you try to access it. Same for readiness.
That being said, @bclozel was asking you in context of
there have been a lot of issues with the liveness and readiness probes in Spring Boot
Comment From: bclozel
@YatishKumar the behavior you're describing is consistent with #22562, so you're probably running Spring Boot 2.3.2.RELEASE. Could you ensure that you're running 2.3.3 as mentioned in your comment?
Comment From: bclozel
@Davio It seems you're confused by the difference between "health indicators" and "health groups". We've tried to improve the documentation on Kubernetes Probes in the latest release. Feel free to suggest improvements if you think this is not clear.
I guess the Kubernetes Probes make the Health Groups feature front and center, whereas developers might have not noticed it previously.
Comment From: Davio
I just found it weird that /livenessState worked in all cases with all configurations and /liveness only worked after removing the 2.3.2 configuration.
Rather than letting those groups have their default configuration, that explicit override was requesting the inclusion of
livenessStateProbeIndicator. Such a bean doesn't exist in2.3.3. The group is empty and therefore 404 when you try to access it. Same for readiness.That being said, @bclozel was asking you in context of
there have been a lot of issues with the liveness and readiness probes in Spring Boot
I think I understand now.
The group include just replaces the regular thing with a non existing bean so by accessing that group you get the 404. But /livenessState is a thing which always exists even if it doesn't belong to the /liveness group.
My minimal example to get the /liveness URL working without a Kubernetes environment is as follows:
management:
endpoint:
health:
probes:
enabled: true
Comment From: bclozel
Alright I'm closing this issue for now since the docs and the release notes are now in order. Thanks!