SpringBoot Unable to create Channel from class class io.netty.channel.epoll.EpollSocketChannel

It has happened like the below. It suddenly occur in product environment. There are the similar issue in this git repository. But, the issues are closed and I don't know how to explain and solve this issue.

Can I get some help to solve it?

1. envronment:

Recently, we have upgraded this version from 2.1.7.RELEASE & Greenwich.SR2. java: 8, spring boot: 2.3.3.RELASE spring cloud: Hoxton.SR8

2. action to reslove reboot application (spring boot)

3. exception log:

20201016 01:09:55.372 ERROR [xxx-service,3b9b41a15864af68,e0eccbddc316c0b0,true] 1 [tp-epoll-3]: [703e1221-16621]  500 Server Error for HTTP POST "/v2/some/xxx" @a.w.r.e.AbstractErrorWebExceptionHandler
io.netty.channel.ChannelException: **Unable to create Channel from class class io.netty.channel.epoll.EpollSocketChannel**
        at io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:46)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        |_ checkpoint ? Request to POST http://xxx-service/v1/some/yyyy [DefaultWebClient]
        |_ checkpoint ? Handler com.xxx.micro.xxxservice.controller.XxxxController#saveOrderInfo(SaveOrderInfoReq) [DispatcherHandler]
        |_ checkpoint ? org.springframework.cloud.sleuth.instrument.web.TraceWebFilter [DefaultWebFilterChain]
        |_ checkpoint ? org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
        |_ checkpoint ? HTTP POST "/v2/some/xxx" [ExceptionHandlingWebHandler]
Stack trace:
                at io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:46)
                at io.netty.bootstrap.AbstractBootstrap.initAndRegister(AbstractBootstrap.java:310)
                at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:155)
                at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:116)
                at reactor.netty.resources.PooledConnectionProvider$PooledConnectionAllocator.lambda$connectChannel$0(PooledConnectionProvider.java:243)
                at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)
                at reactor.core.publisher.Mono.subscribe(Mono.java:4213)
                at reactor.core.publisher.Mono.subscribeWith(Mono.java:4324)
                at reactor.core.publisher.Mono.subscribe(Mono.java:4184)
                at reactor.core.publisher.Mono.subscribe(Mono.java:4120)
                at reactor.core.publisher.Mono.subscribe(Mono.java:4092)
                at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.drainLoop(SimpleDequePool.java:260)
                at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.drain(SimpleDequePool.java:204)
                at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool.doAcquire(SimpleDequePool.java:199)
                at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.request(AbstractPool.java:378)
                at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onSubscribe(PooledConnectionProvider.java:532)
                at reactor.netty.internal.shaded.reactor.pool.SimpleDequePool$QueueBorrowerMono.subscribe(SimpleDequePool.java:509)
                at reactor.netty.resources.PooledConnectionProvider.disposableAcquire(PooledConnectionProvider.java:218)
                at reactor.netty.resources.PooledConnectionProvider.lambda$acquire$3(PooledConnectionProvider.java:182)
                at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)
                at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
                at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:327)
                at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)
                at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
               client_loop: send disconnect: Connection reset by peerxRetryWhen.java:79)
.. (skip)... 

Comment From: snicoll

@jeehunseo please refrain from commenting in multiple places. Someone is trying to give you support elsewhere.

If you want support, please take the time to build a small sample that reproduces the problem. You can do so by attaching a zip to this issue or sharing a link to a GitHub repository.

Comment From: jeehunseo

I expect to know how to solve it, if this issue is the closed issue. Are there the same case like this issue?

Anyway, I'll try the sample project to reproduce and share issue.

Thanks for your quick help.

Comment From: jeehunseo

We don't know the correct reason but fixed after remove the custom 'ClientConnector' in WebClient.builder(). Maybe there is some leak because of async task.

We testing with monitoring 'opend file count'.

I change the status of this issue to closed