Spring Cloud Netflix Sporadic timeout from eureka client during heartbeat to eureka server

We are seeing intermittent (no specific schedule or cadence) errors when eureka clients are sending heartbeats to eureka server. Whenever it fails the 1st retry is always successful. This issue is happening only in our production environment but not in any other environments. We had restarted/redeployed the apps in lower environment (not prod) but cannot replicate this behavior. Please look into it and let me know if you need any additional information to triage it.

Fyi - this error pops up with all eureka clients and below I am providing stack from gateway.

Snippet from client POM 2021.0.0

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        <exclusions>
            <exclusion>
                <groupId>com.google.code.gson</groupId>
                <artifactId>gson</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Error stack

2022-06-06 15:41:27.478 INFO 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RedirectingEurekaHttpClient : Request execution error. endpoint=DefaultEndpoint{ serviceUrl='/eureka/} exception=I/O error on PUT request for "/eureka/apps/GATEWAY-SERVER/gateway-server-green-bf567745-hpx67:gateway-server:8080": :443 failed to respond; nested exception is org.apache.http.NoHttpResponseException: :443 failed to respond stacktrace=org.springframework.web.client.ResourceAccessException: I/O error on PUT request for "/eureka/apps/GATEWAY-SERVER/gateway-server-green-bf567745-hpx67:gateway-server:8080": :443 failed to respond; nested exception is org.apache.http.NoHttpResponseException: :443 failed to respond at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602) at org.springframework.cloud.netflix.eureka.http.RestTemplateEurekaHttpClient.sendHeartBeat(RestTemplateEurekaHttpClient.java:99) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92) at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:91) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92) at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92) at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77) at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89) at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:893) at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1457) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.http.NoHttpResponseException: :443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776) ... 19 more

2022-06-06 15:41:27.478 WARN 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution failed with message: I/O error on PUT request for "/eureka/apps/GATEWAY-SERVER/gateway-server-green-bf567745-hpx67:gateway-server:8080": :443 failed to respond; nested exception is org.apache.http.NoHttpResponseException: :443 failed to respond

2022-06-06 15:41:27.500 INFO 1 --- [tbeatExecutor-0] c.n.d.s.t.d.RetryableEurekaHttpClient : Request execution succeeded on retry #1

Comment From: OlgaMaciaszek

Hello @pgburra , thanks for reporting the issue. Could you provide the Eureka server logs with matching timestamps (with debug logging enabled). Also, without some kind of reproducible sample/ demo, it is going to be very difficult to verify this issue. Please see if you can come up with a reproducible scenario.

Comment From: spring-cloud-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-cloud-issues

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.