spring-cloud 1.3.6 I have a high availability set up for eureka-service-registry. We have thus two instances of Eureka Services that act as peers of each other. For most part this keeps on working fine. Every now and then I see the following exception in the logs
The logs seem benign since work is not impacted and services are up. I believe the following is happening since few of the connections are stuck in CLOSE_WAIT state.
I was reading that there might be a way to have PoolingHttpClientConnectionManager validate connections after inactivity. Is there a way to control this through a configuration. Is this something you have heard about before and do you know the solution to this or is it a bug that is currently being addressed.
c.n.e.c.ReplicationTaskProcessor-Network level connection to peer x.y.z.com; retrying after delay
com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: x.y.x.db.com:8044 failed to respond
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar!/:1.19.1]
at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48) ~[eureka-core-1.7.2.jar!/:1.7.2]
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.7.2.jar!/:1.7.2]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.1.jar!/:1.19.1]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.1.jar!/:1.19.1]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.1.jar!/:1.19.1]
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) ~[jersey-client-1.19.1.jar!/:1.19.1]
at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:116) ~[eureka-core-1.7.2.jar!/:1.7.2]
at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:71) [eureka-core-1.7.2.jar!/:1.7.2]
at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:187) [eureka-core-1.7.2.jar!/:1.7.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
Caused by: org.apache.http.NoHttpResponseException: x.y.z.com:8044 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:294) ~[httpcore-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:230) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.11.jar!/:4.4.11]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.11.jar!/:4.4.11]
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:679) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:481) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.8.jar!/:4.5.8]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.8.jar!/:4.5.8]
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173) ~[jersey-apache-client4-1.19.1.jar!/:1.19.1]
I tried following property by setting it to 5 seconds but that did not help
eureka.client.eureka-connection-idle-timeout-seconds
https://cloud.spring.io/spring-cloud-netflix/reference/html/appendix.html
Comment From: ryanjbaxter
I assume you mean Spring Cloud Netflix 1.3.6?
If so that is part of Spring Cloud Dalston which is no longer supported. Please update to Spring Cloud Hoxton or Spring Cloud 2020.0.1 and reopen the issue if you still encounter the problem.
Comment From: soodrah
I was wrong..we have
<modelVersion>4.0.0</modelVersion>
<artifactId>spring-cloud-netflix</artifactId>
<version>1.4.7.RELEASE</version>
<packaging>pom</packaging>
I guess the above too has reached end of Life based on below, can you confirm
| Release Train | Boot Version |
|---|---|
| 2020.0.x aka Ilford | 2.4.x |
| Hoxton | 2.2.x, 2.3.x (Starting with SR5) |
| Greenwich | 2.1.x |
| Finchley | 2.0.x |
| Edgware | 1.5.x |
| Dalston | 1.5.x |
Spring Cloud Dalston, Edgware, Finchley, and Greenwich have all reached end of life status and are no longer supported.
Comment From: ryanjbaxter
Yes that would be Edgware, which is also end of life
Comment From: soodrah
hi Ryan I updated the spring-cloud.version to Hoxton.SR9 and spring-boot to 2.3.9. I also added the following properties. I am still getting the above error reproduced below again, can you please advise.
eureka.client.eureka-connection-idle-timeout-seconds=5
eureka.server.peer-node-read-timeout-ms=10000
One thing I would like to point out -
server 1 is abc.com
server 2 is xyz.com
The errors are always when abc.com tries to connect to xyz.com but not the other way. xyz.com does not have issues talking to abc.com.
Once again to remind, these are two instances of Eureka-service running on two different machines and participating in an eureka high availability peer network cluster. See exception below
c.n.e.c.ReplicationTaskProcessor-Network level connection to peer xyz.com; retrying after delay
com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: xyz.com:8044 failed to respond
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187) ~[jersey-apache-client4-1.19.1.jar!/:1.19.1]
at com.netflix.eureka.cluster.DynamicGZIPContentEncodingFilter.handle(DynamicGZIPContentEncodingFilter.java:48) ~[eureka-core-1.10.7.jar!/:1.10.7]
at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27) ~[eureka-client-1.10.7.jar!/:1.10.7]
at com.sun.jersey.api.client.Client.handle(Client.java:652) ~[jersey-client-1.19.4.jar!/:1.19.4]
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682) ~[jersey-client-1.19.4.jar!/:1.19.4]
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) ~[jersey-client-1.19.4.jar!/:1.19.4]
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570) ~[jersey-client-1.19.4.jar!/:1.19.4]
at com.netflix.eureka.transport.JerseyReplicationClient.submitBatchUpdates(JerseyReplicationClient.java:117) ~[eureka-core-1.10.7.jar!/:1.10.7]
at com.netflix.eureka.cluster.ReplicationTaskProcessor.process(ReplicationTaskProcessor.java:80) [eureka-core-1.10.7.jar!/:1.10.7]
at com.netflix.eureka.util.batcher.TaskExecutors$BatchWorkerRunnable.run(TaskExecutors.java:190) [eureka-core-1.10.7.jar!/:1.10.7]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]
Caused by: org.apache.http.NoHttpResponseException: xyz.com:8044 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.14.jar!/:4.4.14]
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:294) ~[httpcore-4.4.14.jar!/:4.4.14]
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:230) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.14.jar!/:4.4.14]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.14.jar!/:4.4.14]
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:679) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:481) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118) ~[httpclient-4.5.13.jar!/:4.5.13]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar!/:4.5.13]
at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173) ~[jersey-apache-client4-1.19.1.jar!/:1.19.1]
... 10 more
Comment From: ryanjbaxter
If it is working in one direction but not the other than I would think it has nothing to do with Spring Cloud Netflix and likely a networking issue
Comment From: mahesh539dev
@soodrah Did you find any solution for the error because i have the same issue!!
Comment From: soodrah
this was a network issue and we left it alone, since it was not impacting functionality and we have been running fine since then