If a @RestController uses DeferredResult with timeout, the controller sends a status code HTTP/1.1 503 and closes the connection after the timeout. If the request times out within the graceful shutdown phase, the timeout action does not trigger and no response is sent to the requesting client.

Desired behavior: The timeout action should trigger even if the application is in graceful shutdown phase.

Steps to reproduce: TimeoutController:

@RestController
public class TimeoutController {
    public TimeoutController() {
    }

    @GetMapping("/async-deferredresult")
    public DeferredResult<ResponseEntity<?>> handleReqDefResult() {
        DeferredResult<ResponseEntity<?>> output = new DeferredResult<>(20000L); // wait 20 secs before timing out
        return output;
    }
}

application.properties:

spring.lifecycle.timeout-per-shutdown-phase=60s
server.shutdown=graceful
  • Start App
  • curl -v localhost:8080/async-deferredresult
  • In other terminal send SIGTERM to app: kill $(ps aux |grep java | grep 'DemoApplication' | tr -s ' ' | cut -d ' ' -f 2)

app output:

2022-01-05 11:18:43.893  INFO 127830 --- [ionShutdownHook] o.s.c.support.DefaultLifecycleProcessor  : Failed to shut down 1 bean with phase value 2147483647 within timeout of 60000ms: [webServerGracefulShutdown]
2022-01-05 11:18:43.927  INFO 127830 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown aborted with one or more requests still active

curl output:

➜  ~ curl -v localhost:8080/async-deferredresult
*   Trying 127.0.0.1:8080...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /async-deferredresult HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server

Versions used: Spring Boot 2.6.2 Java 11

Please find attached the deferred-result-app.zip to reproduce the issue.

Comment From: wilkinsona

Thanks for the sample. I've reproduced the problem with Tomcat. Interestingly, the problem does not occur when using Jetty or Undertow so it would appear that there's a problem with how we're gracefully shutting Tomcat down or there is a bug in Tomcat that prevents async requests from timing out once we've initiated the shutdown.

@markt-asf To initiate a graceful shutdown of Tomcat, we pause each Connector and gracefully close its server socket:

https://github.com/spring-projects/spring-boot/blob/ccc924b9083f604bdc04fd534aa8efa4a7b38a98/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/web/embedded/tomcat/GracefulShutdown.java#L91-L92

From what I can tell, it appears that this has had the unwanted side-effect of ignoring async request timeouts. As a result, the graceful shutdown period elapses and we abort the shut down attempt with the async request that should have timed out still active. The connection's then dropped, returning an empty response to the client.

Should we be doing something differently with Tomcat to allow async timeouts to be honoured while also preventing new requests from being accepted?

Comment From: markt-asf

Do you have a preference for what happens in this case? Should the async requests timeout prematurely or should the graceful shutdown keep the requests open until either app completes them or the timeout is reached? My main concern with the latter is how long that timeout might be.

Comment From: wilkinsona

I'd like the requests to be kept open until the app completes them or things time out. That timeout could be the async request's timeout or the graceful shutdown timeout, whichever occurs first. The graceful shutdown timeout (handled by Boot) already works fairly well as we stop waiting for the async requests to complete and the shutdown then proceeds, dropping the connection in the process. The main problem here is that the async request's timeout is ignored once graceful shutdown's begun.

Comment From: markt-asf

Understood.

The root cause appears to be that pausing the Connector also stops the thread that handles async timeouts. Consensus amongst the Tomcat team appears to be that that isn't what was intended. Pausing the Connector should stop new requests and disable keep-alive for existing connections but allow in progress requests to complete (and them the associated connection should close). Assuming that consensus holds, this should be fixed in the next round of Tomcat releases.

Comment From: wilkinsona

Thanks, Mark. Is there an issue or mailing list thread that I can follow to see if the consensus holds and how the fix progresses?

Comment From: markt-asf

The discussion is on the mailing list: https://lists.apache.org/thread/fxbddjy8j204p30rohkm6k6l3ok32kjl

Fixing this is on my TODO list for the January release round. Hopefully I'll be tagging in the next few days.

Comment From: wilkinsona

Thanks, Mark.

Comment From: wilkinsona

For future reference, https://github.com/apache/tomcat/commit/6f2f76d664c146276adebfdcf6847342ca903b08 is the change in Tomcat 9.0.x. We'll pick up the releases in due course.