Spring Cloud Netflix Eureka server UI list eureka server instances error:FreeMarker template error (DEBUG mode; use RETHROW in production!)

I have 2 eureka server instances registered to each other run in aws ec2. It is normal when they just start over, but the eureka server UI have error when show eureka instances info after some time(hours or days, not sure). The eureka server web UI error as follows:

FreeMarker template error (DEBUG mode; use RETHROW in production!): The following has evaluated to null or missing: ==> amiCount.key [in template "eureka/status.ftl" at line 35, column 26] ---- Tip: It's the step after the last dot that caused this error, not those before it. ---- Tip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)?? ---- ---- FTL stack trace ("~" means nesting-related): - Failed at: ${amiCount.key} [in template "eureka/status.ftl" at line 35, column 24] ---- Java stack trace (for programmers): ---- freemarker.core.InvalidReferenceException: [... Exception message was already printed; see it above ...] at freemarker.core.InvalidReferenceException.getInstance(InvalidReferenceException.java:134) at freemarker.core.EvalUtil.coerceModelToTextualCommon(EvalUtil.java:467) at freemarker.core.EvalUtil.coerceModelToStringOrMarkup(EvalUtil.java:389) at freemarker.core.EvalUtil.coerceModelToStringOrMarkup(EvalUtil.java:358) at freemarker.core.DollarVariable.calculateInterpolatedStringOrMarkup(DollarVariable.java:96) at freemarker.core.DollarVariable.accept(DollarVariable.java:59) at freemarker.core.Environment.visit(Environment.java:361) at freemarker.core.IteratorBlock$IterationContext.executedNestedContentForCollOrSeqListing(IteratorBlock.java:317) at freemarker.core.IteratorBlock$IterationContext.executeNestedContent(IteratorBlock.java:271) at freemarker.core.IteratorBlock$IterationContext.accept(IteratorBlock.java:242) at freemarker.core.Environment.visitIteratorBlock(Environment.java:635) at freemarker.core.IteratorBlock.acceptWithResult(IteratorBlock.java:107) at freemarker.core.IteratorBlock.accept(IteratorBlock.java:93) at freemarker.core.Environment.visit(Environment.java:361) at freemarker.core.IteratorBlock$IterationContext.executedNestedContentForCollOrSeqListing(IteratorBlock.java:317) at freemarker.core.IteratorBlock$IterationContext.executeNestedContent(IteratorBlock.java:271) at freemarker.core.IteratorBlock$IterationContext.accept(IteratorBlock.java:242) at freemarker.core.Environment.visitIteratorBlock(Environment.java:635) at freemarker.core.IteratorBlock.acceptWithResult(IteratorBlock.java:107) at freemarker.core.IteratorBlock.accept(IteratorBlock.java:93) at freemarker.core.Environment.visit(Environment.java:325) at freemarker.core.Environment.visit(Environment.java:331) at freemarker.core.Environment.visit(Environment.java:331) at freemarker.core.Environment.process(Environment.java:304) at freemarker.template.Template.process(Template.java:382) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:368) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.doRender(FreeMarkerView.java:285) at org.springframework.web.servlet.view.freemarker.FreeMarkerView.renderMergedTemplateModel(FreeMarkerView.java:235) at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167) at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:303) at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1286) at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1041) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:984) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:108) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)
Whitelabel Error Page
This application has no explicit mapping for /error, so you are seeing this as a fallback.

Mon Feb 26 07:56:31 GMT 2018
There was an unexpected error (type=OK, status=200).
The following has evaluated to null or missing: ==> amiCount.key [in template "eureka/status.ftl" at line 35, column 26] ---- Tip: It's the step after the last dot that caused this error, not those before it. ---- Tip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing</#if>. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)?? ---- ---- FTL stack trace ("~" means nesting-related): - Failed at: ${amiCount.key} [in template "eureka/status.ftl" at line 35, column 24] ----

About the relation info as follows: 1. spring cloud version: Edgware.RELEASE 2. application.yaml:

spring:
  application:
    name: eureka-server
  profiles:
    active: ${env:dev}
server:
  port: 8761
eureka:
  instance:
    healthCheckUrlPath: /health
  client:
    use-dns-for-fetching-service-urls: true
    eureka-server-d-n-s-name: xxx.com
    eurekaServerURLContext: eureka
    eureka-server-port: 8761
    region: us-east-1
    availability-zones:
      us-east-1: us-east-1b, us-east-1d
  server:
    wait-time-in-ms-when-sync-empty: 0
    list-auto-scaling-groups-role-name: xxxRole
    binding-strategy: eip
    enable-self-preservation: false

1. java code:

@SpringBootApplication
@EnableEurekaServer
@EnableDiscoveryClient
public class EurekaServerApplication {

  @Value("${server.port:8761}")
  private int port;

  public static void main(String[] args) {
    SpringApplication.run(EurekaServerApplication.class, args);
  }


  @Bean
  @Autowired
  public EurekaInstanceConfigBean eurekaInstanceConfigBean(InetUtils utils) {
    final EurekaInstanceConfigBean instance = new EurekaInstanceConfigBean(utils) {
      @Scheduled(initialDelay = 30000L, fixedRate = 30000L)
      public void refreshInfo() {
        AmazonInfo newInfo = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
        if (!this.getDataCenterInfo().equals(newInfo)) {
          ((AmazonInfo) this.getDataCenterInfo()).setMetadata(newInfo.getMetadata());
          this.setHostname(newInfo.get(AmazonInfo.MetaDataKey.publicHostname));
          this.setIpAddress(newInfo.get(AmazonInfo.MetaDataKey.publicIpv4));
          this.setNonSecurePort(port);
          this.setDataCenterInfo(newInfo);
        }
      }
    };
    AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
    instance.setHostname(info.get(AmazonInfo.MetaDataKey.publicHostname));
    instance.setIpAddress(info.get(AmazonInfo.MetaDataKey.publicIpv4));
    instance.setNonSecurePort(port);
    instance.setDataCenterInfo(info);
    return instance;
  }
}

Someone can help?

Comment From: ryanjbaxter

I really have no idea. Is there anything in the logs on the server?

Comment From: spencergibb

You are not setting all the required information on DataCenterInfo.

Comment From: lishengsam

@ryanjbaxter I had restarted the 2 eureka servers yesterday, but it is normal until now. If it is not normal, I would try to find related logs and offer to you. @spencergibb Thanks for your hint.But the strange things is that If I was not setting all the required information on DataCenterInfo, eureka server UI should be not normal when I start the eureka server.

I will continue to monitor, If I reproduce the scene，I would offer more information to you. If I resolve the error, I would update the reason here ：)

Comment From: denebj

Hello, I am running a similar issue. I made the setup simple to troubleshoot, 1 eureka and 1 service. When the service in not in a docker container, the dashboard works. If I put the service in the container then the dashboard fails. To be noted that the service is working just fine, it is simply the dashboard that is not working -Throwing AMI not found. If I go to /eureka/apps then the XML displays the service information.

Here some details :

Eureka :

eureka:   
  instance:
    hostname: ${EUREKA_HOSTNAME}
  region: us-west-2
  shouldUseDns: false
  datacenter: cloud
  us-west-2:
    availabilityZones: us-west-2b
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${EUREKA_URL}:${EUREKA_PORT_ZONE}/eureka/
      us-west-2b: http://${EUREKA_URL}:${EUREKA_PORT_ZONE}/eureka/
  server:
    waitTimeInMsWhenSyncEmpty: 0   
    enable-self-preservation: false 

server:
  port: ${PORT}

The service : Bean:

     @Bean
     @Profile("!default")
     @Primary
     public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils) {

       EurekaInstanceConfigBean b = new EurekaInstanceConfigBean(inetUtils);
       AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
        b.setDataCenterInfo(info);
        return b;       
     }

Config:

eureka:
  us-west-2:
    availabilityZones: us-west-2b  
  instance:
    instance-id: ${spring.application.name}:${spring.application.instance_id:${random.value}}
    ip-address: ${SERVICE_HOST_NAME}
    nonSecurePort: ${SERVICE_HOST_NAME_PORT}
    preferIpAddress: true
    leaseRenewalIntervalInSeconds: 1
    leaseExpirationDurationInSeconds: 2
    metadataMap:
      zone: zone1
  client:
    registerWithEureka: true
    fetchRegistry: true
    serviceUrl:
      defaultZone: http://${EUREKA_URL}:${EUREKA_PORT}/eureka/
      preferSameZoneEureka: true
    healthcheck:
      enabled: true
    lease:
      duration: 5

server:
  port: ${PORT}

Docker Service :

docker run \
-e SERVICE_HOST_NAME_PORT='8089' \
-e SERVICE_HOST_NAME='172.1.1.1' \
-e PORT='8080' \
-e EUREKA_PORT='8081' \
-e EUREKA_URL='172.1.1.2' \
-p 8089:8080 \
-d \
--name=zipkin \
--hostname='172.1.1.1' \
zipkin

Docker Eureka:

docker run \
-e EUREKA_PORT_ZONE='8081' \
-e EUREKA_URL='172.1.1.2' \
-e EUREKA_HOSTNAME='172.1.1.2' \
-e PORT='8080' \
-e SPRING_APPLICATION_NAME='EUREKA_PRINCIPAL' \
-p 8081:8080 \
-d \
--hostname='172.1.1.2' \
--name=eureka-principal \
eureka

Eureka/App:

<applications>
<versions__delta>1</versions__delta>
<apps__hashcode>UP_1_</apps__hashcode>
<application>
<name>ZIPKIN-SERVER</name>
<instance>
<instanceId>zipkin-server:b8546b19b911631736a81ea19f828a06</instanceId>
<hostName>172.1.1.1</hostName>
<app>ZIPKIN-SERVER</app>
<ipAddr>172.1.1.1</ipAddr>
<status>UP</status>
<overriddenstatus>UNKNOWN</overriddenstatus>
<port enabled="true">8089</port>
<securePort enabled="false">443</securePort>
<countryId>1</countryId>
<dataCenterInfo class="com.netflix.appinfo.AmazonInfo">
<name>Amazon</name>
<metadata>
<instance-id>zipkin-server:b8546b19b911631736a81ea19f828a06</instance-id>
</metadata>
</dataCenterInfo>
<leaseInfo>
<renewalIntervalInSecs>1</renewalIntervalInSecs>
<durationInSecs>2</durationInSecs>
<registrationTimestamp>1522446723466</registrationTimestamp>
<lastRenewalTimestamp>1522446751466</lastRenewalTimestamp>
<evictionTimestamp>0</evictionTimestamp>
<serviceUpTimestamp>1522446723466</serviceUpTimestamp>
</leaseInfo>
<metadata>
<zone>zone1</zone>
</metadata>
<homePageUrl>http://172.1.1.1:8089/</homePageUrl>
<statusPageUrl>http://172.1.1.1:8089/info</statusPageUrl>
<healthCheckUrl>http://172.1.1.1:8089/health</healthCheckUrl>
<vipAddress>zipkin-server</vipAddress>
<secureVipAddress>zipkin-server</secureVipAddress>
<isCoordinatingDiscoveryServer>false</isCoordinatingDiscoveryServer>
<lastUpdatedTimestamp>1522446723467</lastUpdatedTimestamp>
<lastDirtyTimestamp>1522446884284</lastDirtyTimestamp>
<actionType>ADDED</actionType>
</instance>
</application>
</applications>

Same result if Eureka is on a Docker or not. As soon I have the service in a docker I have that issue.

Thanks for the pointers :)

Comment From: ryanjbaxter

If you can find a way to reproduce this without using AWS then it might be easier for us to help. Without more information it is hard to say what might be wrong.

Comment From: denebj

Hello Baxter so I believe i know what's happening ! I just saw that this morning after digging through the code :) The problem was that my docker image run on an AWS EC2 but the docker image itself is not an EC2. So I have the Eureka configuration bean that create an amazon object in the client. Since that docker is not an EC2 it most likely sends null value when Eureka dashboard is trying to read the object, hence the issue in the template ! I actually removed that Eureka configuration bean and the dashboard worked as it should :)

Comment From: lishengsam

@ryanjbaxter I had resolved this problem with changing the 「EurekaInstanceConfigBean」init as follow:

@SpringBootApplication
@EnableEurekaServer
@EnableDiscoveryClient
public class EurekaServerApplication {

  @Value("${server.port:8761}")
  private int port;

  public static void main(String[] args) {
    SpringApplication.run(EurekaServerApplication.class, args);
  }

  @Qualifier("eurekaApplicationInfoManager")
  @Autowired
  ApplicationInfoManager applicationInfoManager;

  @Bean
  @Autowired
  public EurekaInstanceConfigBean eurekaInstanceConfigBean(InetUtils utils) {
    final EurekaInstanceConfigBean instance = new EurekaInstanceConfigBean(utils) {
      @Scheduled(initialDelay = 30000L, fixedRate = 30000L)
      public void refreshInfo() {
        AmazonInfo newInfo = Builder.newBuilder().autoBuild("eureka");
        if (!applicationInfoManager.getInfo().getDataCenterInfo().equals(newInfo)) {
          this.setHostname(newInfo.get(MetaDataKey.publicHostname));
          this.setIpAddress(newInfo.get(MetaDataKey.publicIpv4));
          this.setNonSecurePort(port);
          this.setDataCenterInfo(newInfo);
          InstanceInfo.Builder builder = new InstanceInfo.Builder(applicationInfoManager.getInfo());
          builder.setDataCenterInfo(newInfo);
          applicationInfoManager.getInfo().setIsDirty();
        }
      }
    };
    AmazonInfo info = AmazonInfo.Builder.newBuilder().autoBuild("eureka");
    instance.setHostname(info.get(AmazonInfo.MetaDataKey.publicHostname));
    instance.setIpAddress(info.get(AmazonInfo.MetaDataKey.publicIpv4));
    instance.setNonSecurePort(port);
    instance.setDataCenterInfo(info);
    return instance;
  }
}

Comment From: storoe1992

En mi caso la solución estuvo en eliminar AmazonInfo como objeto proveedor de datos y dejé el por defecto, la implementación de @lishengsam no me funcionó, me di cuenta por el XML de /eureka/apps que los demás servicios estaban usando DefaultDataCenterInfo mientras que este nuevo estaba usando la instancia AmazonInfo

mi configuración quedó así:

@Bean @Profile("docker") public EurekaInstanceConfigBean eurekaInstanceConfig(InetUtils inetUtils) { final EurekaInstanceConfigBean config = new EurekaInstanceConfigBean(inetUtils); Map mapInfo = getPrivateIp(); String valueIp = mapInfo.get("ip"); String hostName = mapInfo.get("host"); int serverPort = ConfigurationsUtil.configuration().getServerPort(); config.setHostname(hostName); config.setIpAddress(valueIp); config.setNonSecurePort(serverPort); return config; }

En mi caso obtengo el ip por mi propia cuenta porque el objeto AmazonInfo no me daba los datos que necesitaba, al menos no de forma correcta me devolvía null.

De igual manera este servicio era un docker en un EC2.