Comment From: wilkinsona

The improved diagnostics have shown that we seem to be suffering from a problem where the process isn't being scheduled:

org.awaitility.core.ConditionTimeoutException: Lambda expression in org.springframework.boot.devtools.tests.RemoteApplicationLauncher that uses java.io.File, java.io.Fileorg.springframework.boot.devtools.tests.JvmLauncher$LaunchedJvm: expected the predicate to return <true> but it returned <false> for input of <Application launched at 2020-08-24T03:41:29.831658Z produced output:   

      .   ____          _            __ _ _ 
     /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \    
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \   
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )  
      '  |____| .__|_| |_|_| |_\__, | / / / /   
     =========|_|==============|___/=/_/_/_/    
     :: Spring Boot ::       (v2.4.0-SNAPSHOT)  

    2020-08-24 03:42:21.846  INFO 28096 --- [  restartedMain] com.example.DevToolsTestApplication      : Starting DevToolsTestApplication using Java 11.0.8 on e46ff2f5-306d-42bb-6146-b7e441ea79b8 with PID 28096 (/tmp/junit14818993685356428482/app started by root in /tmp/build/6da1d659/git-repo/spring-boot-project/spring-boot-devtools)    
    2020-08-24 03:42:21.947  INFO 28096 --- [  restartedMain] com.example.DevToolsTestApplication      : No active profile set, falling back to default profiles: default   
    2020-08-24 03:42:22.715  INFO 28096 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable  
    2020-08-24 03:42:22.716  INFO 28096 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'

The above shows that the process was launched at 03:41:29 but its first log output didn't appear until almost a minute later at 03:42:21. The first retry of the entire test passed in 17 seconds. I don't think there's much we can do about process scheduling/noisy neighbour problems on CI, but we could further increase the period that we wait for the server port.

Comment From: wilkinsona

The latest changes seem to have worked. They were made on 25 August and the last flaky test was on 24 August.