With https://github.com/spring-projects/spring-framework/issues/31178, ReactorResourceFactory is now lifecycle-aware. But on Spring Boot side, NettyReactiveWebServerFactory#createHttpServer currently retrieve the LoopResources instance, which leads to the following error when restarting the application context after a JVM Checkpoint / Restore as pointed out by @violetagg:
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[netty-common-4.1.97.Final.jar!/:4.1.97.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) ~[netty-transport-4.1.97.Final.jar!/:4.1.97.Final]
at reactor.netty.transport.TransportConnector.doInitAndRegister(TransportConnector.java:293) ~[reactor-netty-core-1.1.10.jar!/:1.1.10]
at reactor.netty.transport.TransportConnector.bind(TransportConnector.java:87) ~[reactor-netty-core-1.1.10.jar!/:1.1.10]
at reactor.netty.transport.ServerTransport.lambda$bind$0(ServerTransport.java:115) ~[reactor-netty-core-1.1.10.jar!/:1.1.10]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58) ~[reactor-core-3.6.0-M2.jar!/:3.6.0-M2]
at reactor.core.publisher.Mono.subscribe(Mono.java:4495) ~[reactor-core-3.6.0-M2.jar!/:3.6.0-M2]
at reactor.core.publisher.Mono.block(Mono.java:1737) ~[reactor-core-3.6.0-M2.jar!/:3.6.0-M2]
at reactor.netty.transport.ServerTransport.bindNow(ServerTransport.java:149) ~[reactor-netty-core-1.1.10.jar!/:1.1.10]
at reactor.netty.transport.ServerTransport.bindNow(ServerTransport.java:134) ~[reactor-netty-core-1.1.10.jar!/:1.1.10]
at org.springframework.boot.web.embedded.netty.NettyWebServer.startHttpServer(NettyWebServer.java:149) ~[spring-boot-3.2.0-SNAPSHOT.jar!/:3.2.0-SNAPSHOT]
The LoopResources retrieval should probably be moved at a point where it will be re-evaluated after ReactorResourceFactory#start has been invoked.
Comment From: wilkinsona
I don't think I understand the problem. As far as I can tell, things are already being stopped and started in the required order. I've modified SampleWebFluxApplication to run the app, and then stop and start its context:
ConfigurableApplicationContext context = application.run(args);
context.stop();
context.start();
I've also enabled debug logging for DefaultLifecycleProcessor. The following output is produced:
2023-09-07T17:56:42.845+01:00 INFO 90878 --- [ main] s.webflux.SampleWebFluxApplication : Started SampleWebFluxApplication in 2.341 seconds (process running for 2.77)
2023-09-07T17:56:42.849+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147483647
2023-09-07T17:56:42.849+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'applicationTaskExecutor' completed its stop procedure
2023-09-07T17:56:42.849+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147482623
2023-09-07T17:56:42.850+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'webServerGracefulShutdown' completed its stop procedure
2023-09-07T17:56:42.851+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 2147481599
2023-09-07T17:56:42.852+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'webServerStartStop' completed its stop procedure
2023-09-07T17:56:42.852+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase 0
2023-09-07T17:56:44.917+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Successfully stopped bean 'reactorResourceFactory'
2023-09-07T17:56:44.918+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Stopping beans in phase -2147483647
2023-09-07T17:56:44.918+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Bean 'springBootLoggingLifecycle' completed its stop procedure
2023-09-07T17:56:44.919+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase -2147483647
2023-09-07T17:56:44.919+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Successfully started bean 'springBootLoggingLifecycle'
2023-09-07T17:56:44.919+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2023-09-07T17:56:44.919+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Successfully started bean 'reactorResourceFactory'
2023-09-07T17:56:44.919+01:00 DEBUG 90878 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 2147481599
2023-09-07T17:56:44.926+01:00 WARN 90878 --- [ main] io.netty.channel.AbstractChannel : Force-closing a channel whose registration task was not accepted by an event loop: [id: 0xa7ebf375]
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:934) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:351) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:344) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute0(SingleThreadEventExecutor.java:827) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:817) ~[netty-common-4.1.97.Final.jar:4.1.97.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:483) ~[netty-transport-4.1.97.Final.jar:4.1.97.Final]
at reactor.netty.transport.TransportConnector.doInitAndRegister(TransportConnector.java:293) ~[reactor-netty-core-1.1.11-SNAPSHOT.jar:1.1.11-SNAPSHOT]
at reactor.netty.transport.TransportConnector.bind(TransportConnector.java:87) ~[reactor-netty-core-1.1.11-SNAPSHOT.jar:1.1.11-SNAPSHOT]
at reactor.netty.transport.ServerTransport.lambda$bind$0(ServerTransport.java:115) ~[reactor-netty-core-1.1.11-SNAPSHOT.jar:1.1.11-SNAPSHOT]
at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:58) ~[reactor-core-3.6.0-SNAPSHOT.jar:3.6.0-SNAPSHOT]
at reactor.core.publisher.Mono.subscribe(Mono.java:4495) ~[reactor-core-3.6.0-SNAPSHOT.jar:3.6.0-SNAPSHOT]
at reactor.core.publisher.Mono.block(Mono.java:1737) ~[reactor-core-3.6.0-SNAPSHOT.jar:3.6.0-SNAPSHOT]
at reactor.netty.transport.ServerTransport.bindNow(ServerTransport.java:149) ~[reactor-netty-core-1.1.11-SNAPSHOT.jar:1.1.11-SNAPSHOT]
at reactor.netty.transport.ServerTransport.bindNow(ServerTransport.java:134) ~[reactor-netty-core-1.1.11-SNAPSHOT.jar:1.1.11-SNAPSHOT]
at org.springframework.boot.web.embedded.netty.NettyWebServer.startHttpServer(NettyWebServer.java:149) ~[main/:na]
at org.springframework.boot.web.embedded.netty.NettyWebServer.start(NettyWebServer.java:100) ~[main/:na]
at org.springframework.boot.web.reactive.context.WebServerManager.start(WebServerManager.java:55) ~[main/:na]
at org.springframework.boot.web.reactive.context.WebServerStartStopLifecycle.start(WebServerStartStopLifecycle.java:41) ~[main/:na]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:258) ~[spring-context-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:441) ~[spring-context-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:230) ~[spring-context-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
at org.springframework.context.support.DefaultLifecycleProcessor.start(DefaultLifecycleProcessor.java:157) ~[spring-context-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
at org.springframework.context.support.AbstractApplicationContext.start(AbstractApplicationContext.java:1444) ~[spring-context-6.1.0-SNAPSHOT.jar:6.1.0-SNAPSHOT]
at smoketest.webflux.SampleWebFluxApplication.main(SampleWebFluxApplication.java:38) ~[main/:na]
As the logging shows, the web server is stopped and then the resource factory is stopped. Symmetrically, when restarting, the resource factory is started and then the web server is started.
Comment From: wilkinsona
Ah, I guess it's because stopping and then starting the ReactorResourceFactory changes the underling LoopResources instances. The doesn't align well with HttpServer being immutable as it means that the HttpServer instance that we're working with will change.
I guess we might be able to tolerate that but it feels like quite a nasty side-effect of implementing Lifecycle. Can't the same LoopResources instance be kept across stop() and start()?
Comment From: wilkinsona
https://github.com/wilkinsona/spring-boot/tree/gh-37209 fixes things in Boot. However, rather than merging those changes, I still think it would be better if this was addressed in ReactorResourceFactory such that implementing Lifecycle wasn't a breaking change.
Comment From: sdeleuze
As far as I can tell, LoopResources can be disposed but not reactivated, so we can't reuse the instance. If Reactor team is ok to introduce this capability, we could adapt Spring Framework and not require a change on Boot side. If they don't, I think our current implementation is the best we can do.
@violetagg @OlegDokuka Any thoughts?
Comment From: violetagg
It is intentionally created like this
On Fri, 8 Sep 2023 at 9:27, Sébastien Deleuze @.***> wrote:
As far as I can tell, LoopResources can be disposed but not reactivated, so we can't reuse the instance. If Reactor team is ok to introduce this capability, we could adapt Spring Framework and not require a change on Boot side. If they don't, I think our current implementation is the best we can do.
@violetagg https://github.com/violetagg @OlegDokuka https://github.com/OlegDokuka Any thoughts?
— Reply to this email directly, view it on GitHub https://github.com/spring-projects/spring-boot/issues/37209#issuecomment-1711143006, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFKCVJSAXRS7MFNFG2E6DLXZK3D3ANCNFSM6AAAAAA4NJDFSA . You are receiving this because you were mentioned.Message ID: @.***>
Comment From: sdeleuze
Thanks for your feedback Violeta, so I would hope that it will be possible to merge https://github.com/wilkinsona/spring-boot/tree/gh-37209. Conceptually, it can of make sense to me that some adjustment are required when we introduce support of checkpoint/restore and this one (defer accessing loop resources until web server is being started) kind of make sense to me.
I have improved ReactorResourceFactory API documentation to mention those requirements when a Lifecycle stop/restart happens.