Suppose two eureka servers which are not connected together and one client which knows about two servers. When the client starts it registers itself either on server 1 or server 2. If the server on which client is registered would go down then client should automatically register itself on another available server. The client will try to send heartbeat to all known services until it succeeds, if client is not registered in some service it would receive 404 and then would send register request. Due to 404 RestTemplateEurekaHttpClient.sendHeartBeat fails to convert object to InstanceInfo and throws exception here which makes it not possible for DiscoveryClient to register client on another available server.

Setup:

eureka server 1:

spring:
  application:
    name: discovery-server-1

eureka:
  client:
    register-with-eureka: false
    fetch-registry: false

eureka server 2

spring:
  application:
    name: discovery-server-2

eureka:
  client:
    register-with-eureka: false
    fetch-registry: false

eureka client

eureka:
  client:
    register-with-eureka: true
    fetch-registry: true
    serviceUrl:
      defaultZone: http://127.0.0.1:8000/eureka, http://127.0.0.2:8001/eureka

Stack trace:

http-outgoing-9 >> "PUT /eureka/apps/DISCOVERY-CLIENT/localhost:discovery-client:8080?status=UP&lastDirtyTimestamp=1678643765127 HTTP/1.1[\r][\n]"
http-outgoing-9 >> "Accept: application/json, application/*+json[\r][\n]"
http-outgoing-9 >> "Accept-Encoding: gzip, x-gzip, deflate[\r][\n]"
http-outgoing-9 >> "Content-Length: 0[\r][\n]"
http-outgoing-9 >> "Host: 127.0.0.1:8000[\r][\n]"
http-outgoing-9 >> "Connection: keep-alive[\r][\n]"
http-outgoing-9 >> "User-Agent: Apache-HttpClient/5.1.4 (Java/17)[\r][\n]"
http-outgoing-9 >> "[\r][\n]"
http-outgoing-9 << "HTTP/1.1 404 [\r][\n]"
http-outgoing-9 << "Content-Type: application/json[\r][\n]"
http-outgoing-9 << "Transfer-Encoding: chunked[\r][\n]"
http-outgoing-9 << "Date: Sun, 12 Mar 2023 17:57:35 GMT[\r][\n]"
http-outgoing-9 << "Keep-Alive: timeout=60[\r][\n]"
http-outgoing-9 << "Connection: keep-alive[\r][\n]"
http-outgoing-9 << "[\r][\n]"
http-outgoing-9 << "95[\r][\n]"
http-outgoing-9 << "{"timestamp":"2023-03-12T17:57:35.357+00:00","status":404,"error":"Not Found","path":"/eureka/apps/DISCOVERY-CLIENT/localhost:discovery-client:8080"}[\r][\n]"
http-outgoing-9 << HTTP/1.1 404 
http-outgoing-9 << Content-Type: application/json
http-outgoing-9 << Transfer-Encoding: chunked
http-outgoing-9 << Date: Sun, 12 Mar 2023 17:57:35 GMT
http-outgoing-9 << Keep-Alive: timeout=60
http-outgoing-9 << Connection: keep-alive
ex-0000000014 connection can be kept alive for 60 SECONDS
Response 404 NOT_FOUND
Reading to [com.netflix.appinfo.InstanceInfo]

Request execution error. endpoint=DefaultEndpoint{ serviceUrl='http://127.0.0.1:8000/eureka/}, exception=Error while extracting response for type [class com.netflix.appinfo.InstanceInfo] and content type [application/json] stacktrace=org.springframework.web.client.RestClientException: Error while extracting response for type [class com.netflix.appinfo.InstanceInfo] and content type [application/json]
    at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:118)
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1132)
    at org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1115)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:865)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:764)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:646)
    at org.springframework.cloud.netflix.eureka.http.RestTemplateEurekaHttpClient.sendHeartBeat(RestTemplateEurekaHttpClient.java:99)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
    at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.executeOnNewServer(RedirectingEurekaHttpClient.java:121)
    at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:80)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
    at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
    at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
    at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
    at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:837)
    at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1401)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Root name ('timestamp') does not match expected ('instance') for type `com.netflix.appinfo.InstanceInfo`
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:406)
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.read(AbstractJackson2HttpMessageConverter.java:354)
    at org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:103)
    ... 23 more
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Root name ('timestamp') does not match expected ('instance') for type `com.netflix.appinfo.InstanceInfo`
 at [Source: (org.springframework.util.StreamUtils$NonClosingInputStream); line: 1, column: 2] (through reference chain: com.netflix.appinfo.InstanceInfo["timestamp"])
    at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
    at com.fasterxml.jackson.databind.DeserializationContext.reportPropertyInputMismatch(DeserializationContext.java:1781)
    at com.fasterxml.jackson.databind.DeserializationContext.reportPropertyInputMismatch(DeserializationContext.java:1797)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext._unwrapAndDeserialize(DefaultDeserializationContext.java:348)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:320)
    at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2105)
    at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1481)
    at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395)
    ... 25 more

Comment From: ivan-zaitsev

Fixed here but not yet released: #4145