We are using spring boot actuator for exposing liveness and readiness endpoints when running in kubernetes cluster. By default spring boot actuator exposes the end point on the default standard http server port where the request are served by Tomcat/Jetty server acceptor and worker thread pools. We recently ran into an issue during our stress testing where all threads in worker pool were busy and new requests were getting queued. This caused the pod to crash in kubernetes cluster as the liveness probes started failing.
I am considering to expose actuator on management port. I wanted to check on the following
a) Are the requests on management port served a separate worker thread pool (from that of standard server port ) ?
b) If answer to a) is no, is there a way i can configure spring boot to use a seperate thread pool for management port (we are using tomcat/jetty and reacitve netty servers across our different micro services)
Comment From: wilkinsona
Yes, when you are using a separate management port requests are handled by a separate embedded server and, therefore, by a separate thread pool. This is mentioned, along with some potential caveats, in the reference documentation about the probes:
If your Actuator endpoints are deployed on a separate management context, be aware that endpoints are then not using the same web infrastructure (port, connection pools, framework components) as the main application. In this case, a probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections).
If you have any further questions, please follow up on Stack Overflow or Gitter. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements.
Comment From: arifiqbal81
Thanks @wilkinsona