We got SSLPeerUnverifiedException when we try to access secure https REST point from zuul. complete stack trace is attached. error_zuul.txt

zuul configuration:

spring:
  application:
    name: gateway 
server:
  port: 9092
  ssl:    
    enabled: true
    key-store: classpath:keystore.jks
    key-store-password: password
    key-password: password   
    key-alias: xxxx
eureka:
  instance:
    securePort: ${server.port}
    nonSecurePortEnabled: false
    securePortEnabled: true
  client:
    serviceUrl:
      defaultZone: ${EUREKA_URI:http://localhost:8761/eureka}
    registry-fetch-interval-seconds: 15
    register-with-eureka: true
    fetch-registry: true
    heartbeat-executor-thread-pool-size: 5
    eureka-service-url-poll-interval-seconds: 10
zuul:
  prefix: /tree
  routes:
    serv:
      path: /cxf/**
      strip-prefix: false
      serviceId: serv
ribbon:
  IsSecure: true
  IsHostnameValidationRequired: false

service is registered with eureka via POST request. Please find the below sample POST request.

{
"instance": {
    "hostName": "xxx",
    "app": "appname",
    "vipAddress": "appname",
    "secureVipAddress": "appname",
    "ipAddr": "10.xxx.xx.xxx",
    "status": "UP",
    "port": {"$": "8181", "@enabled": "true"},
    "securePort": {"$": "8443", "@enabled": "true"},
    "healthCheckUrl": "http://localhost:8000/cat",
    "statusPageUrl": "http://localhost:8000/cat",
    "homePageUrl": "http://localhost:8000/cat",
    "dataCenterInfo": {
        "@class": "com.netflix.appinfo.InstanceInfo$DefaultDataCenterInfo", 
        "name": "MyOwn"
    }
}

}

if i replace 'serviceId' with the corresponding 'url' in zuul configuration, it works fine. keystore.jks has been under src/main/resources. Also, imported keystore entry into certificate under $JDK_HOME/jre/lib/security/cacerts. Are we missing any other configuration?

Kindly help to resolve the issue.

NOTE: REST endpoint is an OSGI service. spring boot version: v1.5.7.RELEASE We use embedded tomcat.

Comment From: ryanjbaxter

There is no need to add the keystore.jks file that wont do anything. If that is the JDK used to run the app than importing the certificate into the JDK should do the trick.

Comment From: Barathi07

Thank you for your response. I tried importing certificate into $JAVA_HOME/jre/lib/security/cacerts. Even then facing same issue.

Comment From: ryanjbaxter

I went back and looked at the error in the stacktrace

javax.net.ssl.SSLPeerUnverifiedException: Certificate for <xxx> doesn't match common name of the certificate subject: 10.xxx.xx.xx

That sound like the certificate you created is bad.

Comment From: Barathi07

You're right @ryanjbaxter . Certificate which i created doesn't have the field "SubjectAlternativeName". Now i created the certificate where "SubjectAlternativeName" includes CN(Common Name) and IP details and is working fine. Thank you. For someone who is looking for a solution, hope the below steps might help.

1) Generate server key and self signed server certificate keytool -genkey -alias serverkey -keyalg RSA -storetype PKCS12 -keystore serverkeystore.p12 -ext SAN=dns:abc.com,dns:localhost,ip:127.0.0.1

2) Generate client key and self signed client certificate keytool -genkey -alias clientkey -keyalg RSA -storetype PKCS12 -keystore clientkeystore.p12 -ext SAN=dns:def.com,dns:localhost,ip:127.0.0.1

3) Export the server certificate keytool -export -alias serverkey -file servercert.cer -keystore serverkeystore.p12

4) Export the client certificate keytool -export -alias clientkey -file clientcert.cer -keystore clientkeystore.p12

5) Import cert to $JAVA_HOME/jre/lib/security sudo keytool -import -trustcacerts -alias localhost -file localhost.crt -keystore $JAVA_HOME/jre/lib/security/cacerts

application.yml of gateway:

spring:
  application:
    name: gateway
server:
  port: 8443
  ssl:
    enabled: true
    key-store: classpath:serverkeystore.p12
    key-store-password: server
    key-alias: serverkey
eureka:
  instance:
    securePort: ${server.port}
    nonSecurePortEnabled: false
    securePortEnabled: true
    leaseRenewalIntervalInSeconds: 7
    leaseExpirationDurationInSeconds: 9
  client:
    serviceUrl:
      defaultZone: ${EUREKA_URI:http://localhost.com:8761/eureka/}
    registry-fetch-interval-seconds: 5
    register-with-eureka: true
    fetch-registry: true
    heartbeat-executor-thread-pool-size: 5
    eureka-service-url-poll-interval-seconds: 10
zuul:
  prefix: /service
  routes:
    producer:
      path: /employee/**
      strip-prefix: false
      serviceId: producer
ribbon:
  IsSecure: true
logging:
 file: logs/gateway.log
 level.root: INFO
 level.com.fujitsu.fnc.sdnfw.msvc: DEBUG

Comment From: Barathi07

@ryanjbaxter I have two queries.

#Query1 I tried generating wildcard SSL certificate [(i.e.) created certificate with the below mentioned params] to support multiple sub domains on same domain name.

1: ObjectId: 1.2.3.4 Criticality=false

SubjectAlternativeName [ DNSName: *.domain.com DNSName: CNname ]

So that i can make use of same certificate across servers. But, am getting below exception: Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <10.xx.xx.xxx> doesn't match any the subject alternative names: [*.domain.com, CNname]

Complete exception message: Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <10.xx.xx.xxx> doesn't match any of the subject alternative names: [*.domain.com, CNname] at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:467) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:397) at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient$1.doWithRetry(RetryableRibbonLoadBalancingHttpClient.java:92) at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient$1.doWithRetry(RetryableRibbonLoadBalancingHttpClient.java:70) at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:286) at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:163) at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.executeWithRetry(RetryableRibbonLoadBalancingHttpClient.java:109) at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.execute(RetryableRibbonLoadBalancingHttpClient.java:100) at org.springframework.cloud.netflix.ribbon.apache.RetryableRibbonLoadBalancingHttpClient.execute(RetryableRibbonLoadBalancingHttpClient.java:49) at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287) at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231) at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228) at rx.Observable.unsafeSubscribe(Observable.java:10211) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286) at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185) at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180) at rx.Observable.unsafeSubscribe(Observable.java:10211) at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94) at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) at rx.Observable.subscribe(Observable.java:10307) at rx.Observable.subscribe(Observable.java:10274) at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445)

Can you please let me know will spring boot microservices support wildcard certificates. Please assist on this.

NOTE: Gateway has same configuration as mentioned in the previous comment. Additional parameter added under zuul:
sslHostnameValidationEnabled: false

#Query2 Do we need restart microservices upon adding an SecurePortEnabled(https) microservice. The reason why am asking is that , in this case we may have add the new microservice's certificate in the truststore/ $JAVA_HOME/jre/lib/security/cacerts of the existing microservice. Say for example I have 3 microservcies, all are securePortEnabled(https): 1. Eureka 2. Gateway 3. Producer

In the above mentioned scenario works fine, as we would have added the certificate in the $JAVA_HOME/jre/lib/security/cacerts.
Suppose if we add another Producer instance from a new server, we may have to add the certificate in gateway, eureka right? If yes, do we need to restart Eureka/Zuul? If no, will microservices detect "cacerts" changes or addition of new certificate at runtime?

Thank you in advance. I look forward to your response.

Comment From: ryanjbaxter

It is the Apache HTTP Client giving that error not Spring Cloud, we arent doing anything special to validate the certificate. It sounds like you are accessing the app using the 10.x.x.x ip address but that was not used when creating the certificate. You should be able to use wild card certs if the cert is configured properly.

As far as importing a new cert into the JVM is concerned I believe you would need to restart the app before the JVM knows about the new cert. From what I have read online it sounds like the JVM only reads the certs once on startup.

Comment From: Barathi07

Thank you so much @ryanjbaxter . You are absolutely right! I have set "perferIpAddress" field in yml to true. That's why it didn't work. I modified the parameter "eureka.instance.preferIpAddress: false" in the yml file and it is working fine. Once again thank you for the clarification on both queries.

Comment From: Barathi07

@ryanjbaxter Could you please guide me to resolve the following exception?

`2018-01-08 07:24:43.316 DEBUG 9819 --- [https-jsse-nio-8443-exec-3] o.a.tomcat.util.net.SecureNioChannel : The SNI host name extracted for this connection was [null] 2018-01-08 07:24:43.316 DEBUG 9819 --- [https-jsse-nio-8443-exec-3] org.apache.tomcat.util.net.NioEndpoint : Error during SSL handshake javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection? at sun.security.ssl.EngineInputRecord.bytesInCompletePacket(EngineInputRecord.java:156) at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:868) at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:459) at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:232) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1432) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)

2018-01-08 07:24:43.316 DEBUG 9819 --- [https-jsse-nio-8443-exec-3] org.apache.tomcat.util.net.NioEndpoint : Failed to close socket`

Stacktrace of Zuul is attached. [Zuul_Log_SSLException] Zuul_Log_SSLException.txt

Imported wildcard certificate into JVM's cacerts. Able to access the REST end points exposed in the microservice via Zuul. But, couldn't figure out the reason for the above exception.

Certificate content: `[abc@xxx jks]$ keytool -printcert -v -file certificate.cer Owner: CN=.domain.com, OU=ABC, O=ABC Communications, L=Sunnyvale, ST=California, C=US Issuer: CN=.domain.com, OU=ABC, O=ABC Communications, L=Sunnyvale, ST=California, C=US Serial number: 9a4ef456 Valid from: Thu Jan 04 00:56:37 EST 2018 until: Fri Jan 04 00:56:37 EST 2019 Certificate fingerprints: MD5: 70:96:0D:YF:63:F3:BC:FA:CJ:0D:06:88:GJ:31:66:A4 SHA1: 09:F4:73:33:50:11:DF:U2:1I:D9:19:C5:63:22:07:87:2E:F7:E3:0B SHA256: 0B:5E:T4:8E:KC:O5:P6:8F:P9:10:6B:0U:PA:E0:44:B3:B6:20:AD:7C:70:C9:9B:F8:9B:DF:40:L6:SA:19:0Z:C5 Signature algorithm name: SHA256withRSA Version: 3

Extensions:

1: ObjectId: 2.5.29.17 Criticality=false

SubjectAlternativeName [ DNSName: *.domain.com ] ` Is the certificate which i created is correct. Does it require any SNI related params/headers. Please help me to resolve this.

Comment From: ryanjbaxter

I am not sure. And there is nothing indicating Spring Cloud is involved in the stacktrace.

Comment From: testinggithub1222

@Barathi07, would you mind explain me why do we need to generate client certificate? What is the different between server and client certificate? and in case we bought ssl from any website, which would we need to have to generate by ourselves?

Comment From: ryanjbaxter

@testinggithub1222 I dont see where anyone suggested generating a certificate. The problem is almost always that the certificate, whether generated yourself or purchased, is not trusted by the JVM

Comment From: abhigupta13004

try changing the version for "org.apache.httpcomponents:httpclient" to 4.5.2

Spring Cloud Netflix javax.net.ssl.SSLPeerUnverifiedException: Certificate for <xxx> doesn't match common name of the certificate subject

1: ObjectId: 1.2.3.4 Criticality=false

1: ObjectId: 2.5.29.17 Criticality=false