We recently upgraded to 2.4.5, in order to get the fix for DefaultPartHttpMessageReader #26736 that was delivered in Spring 5.3.6.
We face the following exception in our apps now :
22-Apr-2021 17:11:05.479 SEVERE [http-nio-127.0.0.1-9999-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at java.base/sun.nio.ch.Net.accept(Native Method)
at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:305)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:463)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.base/java.lang.Thread.run(Thread.java:832)
22-Apr-2021 17:11:05.507 SEVERE [http-nio-127.0.0.1-9999-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at java.base/sun.nio.ch.Net.accept(Native Method)
at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:305)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:463)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.base/java.lang.Thread.run(Thread.java:832)
22-Apr-2021 17:11:06.440 WARNING [Catalina-utility-2] org.apache.catalina.users.MemoryUserDatabase.backgroundProcess Failed to close [conf/tomcat-users.xml]
java.io.FileNotFoundException: /data/apache-tomcat-9.0.24/conf/tomcat-users.xml (Too many open files)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:211)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:153)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:108)
at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:189)
at org.apache.catalina.users.MemoryUserDatabase.backgroundProcess(MemoryUserDatabase.java:685)
at org.apache.catalina.realm.UserDatabaseRealm.backgroundProcess(UserDatabaseRealm.java:141)
at org.apache.catalina.realm.CombinedRealm.backgroundProcess(CombinedRealm.java:299)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1137)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:832)
^[[28~22-Apr-2021 17:12:17.459 SEVERE [http-nio-127.0.0.1-9999-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at java.base/sun.nio.ch.Net.accept(Native Method)
at java.base/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:305)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:463)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:73)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:95)
at java.base/java.lang.Thread.run(Thread.java:832)
Comment From: wilkinsona
Thanks for the report, but I don't think we'll be able to diagnose this with the information provided. When the problem occurs, can you please look at the files that Tomcat has open? That may help to pinpoint the cause of the problem.
Comment From: saad14092
@wilkinsona Thanks for the quick reply. I've been digging more into the problem since I've discovered it. It is not container-specific if you will. It is reproduced in both external Tomcat and embedded Netty (MVC and Webflux apps). I'm not sure but I think it happens when a @Scheduled task is executed.
I'm working on pin-pointing the cause.
Also, using the lsof command I can see there is a node that repeats so many times "NODE 8656 anon_inode" for each java app that is running (including Tomcat)
Comment From: spring-projects-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: spring-projects-issues
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.
Comment From: JanHron
I think this was closed by mistake - the OP provided the requested information, but a bot closed the issue. We've been having similar issues when using Apache HttpClient 5.1 as a backend for the Reactive WebClient. From my research the behavior is very similar to this issue: https://issues.apache.org/jira/browse/HTTPCORE-631 which was however fixed in the version we're using. Is it certain that the WebClient is properly closing the underlying CloseableHttpAsyncClient? If so, I'd lean in the direction of yet another bug in the used external Client, not sure if OP's situation is similar to ours though.
Comment From: bclozel
@JanHron Could be. Are you redeploying the same application in the same Servlet container instance multiple times (i.e. keeping the same JVM instance running)?
Comment From: JanHron
@bclozel No, its a stand-alone Spring Boot application, but we're creating multiple WebClients with different settings during the application's lifetime. I noticed that for every WebClient we create, pipes and anon_inodes remain for what is either infinite or a long time, which leads me to think that the HttpClient does not get closed properly.
Comment From: bclozel
@JanHron I've just raised spring-projects/spring-framework#27032 but I don't think it will directly solve the problem you're seeing.
If you're instantiating CloseableHttpAsyncClient instances yourself because you want to customize their configuration, I think that in this case you're responsible for closing them when they need to be recycled.
A better approach would be to reuse the same connector instance for all WebClient instances if you can. With spring-projects/spring-framework#27032, you'll be in a position to close the underlying client when you need to dispose of the client.
In all cases, creating an unbounded number of clients in an application is a problem and you should look into it. Usually clients should be bound to beans and their lifecycle. I'm not familiar with your use case but I suspect that not being in control of the lifecycle of WebClient instances will be a problem anyway.
- Could you elaborate a bit about the
WebClientinstances lifecycle, how many are created, disposed of and when? - Would the Spring Framework issue I've just created solve the problem you're facing?
Comment From: JanHron
@bclozel Thank you for your quick replies! We have an application which collects data from various REST APIs, each of which needs a specific configuration. So we organized it into a set of separate libraries with a common API which is used to get a suitable WebClient for each of the API. The main application then obtains a WebClient which is pre-configured by the library and uses it to fetch data from an endpoint.
All of this happens in a fairly complicated parallel Flux, but the bottom line is that since the configuration of the endpoints can change between data collection cycles (for example due to change of credentials), we allow the WebClients to get garbage collected at the end of the data collection cycle. I think this is where the problem is - the file descriptors are not being released during garbage collection.
If I understand your suggestion correctly, it could help us. What I expect from garbage collected instances is that they clean up their resources when they are no longer needed. We previously used the async Netty client and it was working just fine without depleting file descriptors, but after an update of spring-boot-sarter-webflux to a newer version (specifically its update of the reactor-netty dependency), our Fluxes started to hang and we were unable to pinpoint the root cause. After hours of fruitless debugging, we decided to try an alternative client - the Apache one worked well and did not cause the Flux to hang, so we thought we would be able to stick with it before we're able to figure out what's wrong with reactor-netty.
Comment From: JanHron
On a side note, why is it so wrong to create new WebClients all the time? I would expect that it would only potentially cause performance problems due to costly creation of the clients, and I can easily imagine that many people just create a new WebClient for every request.
Comment From: bclozel
@JanHron In your case, you could maybe start off by getting the WebClient.Builder bean auto-configured by Spring Boot, which has sensible defaults for codecs and is instrumented for metrics. From there you can apply additional defaults and get a WebClient.
Since your application needs slightly different instances depending on the case, you could mutate the base WebClient and apply local customizations (base URL, authentication and more).
This would solve a lot of issues: * you'd get the builder prepared by Spring Boot * you'd ensure that all client instances still share the same connector, so descriptors leaks won't happen * each library could implement a callback that is used to customized the shared instance; as far as I understand right now, each library can choose to use a different client under the hood (reactor netty, jetty, http components) which can be wasteful and brittle
WebClient instances are cheap to create, only the connectors can be costly and should be reused as much as possible. Creating a WebClient instance is wasteful; we've provided a lot of infrastructure around it so that you don't need to use that pattern. In general we're advising developers to avoid creating an unbounded number of clients for this very reason: if the connector instances are not managed properly, this can lead to performance issues like this.
In your case, I guess libraries are always creating new instances by calling new HttpComponentsClientHttpConnector() which effectively creates a new HttpClient and never releases the resources. Even with the Spring Framework fix in, manually creating many connector instances and never calling close() on them will still get you in trouble.