SpringBoot Spring RSocket "SSLHandshakeException: No available authentication scheme" when using PKCS12 Keystore

Spring Boot (starter rsocket) Version 2.6.2 Kotlin 1.6.10 Temurin-17.0.1+12

Hey, i have an issue with the SSL Handshake when using an PKCS12 keystore. This won't happen if you convert the keystore to a JKS type by using keytool -importkeystore -srckeystore rsocket-server.p12 -srcstoretype pkcs12 -srcalias rsocket -destkeystore rsocket-server.jks -deststoretype jks -deststorepass password -destalias rsocket

Steps to reproduce: Create a self signed cert, trust- and keystore:

keytool -genkeypair -alias rsocket -keyalg RSA -keysize 2048 -storetype PKCS12 -validity 3650 -keystore rsocket-server.p12 -storepass password
keytool -exportcert -alias rsocket -keystore rsocket-server.p12 -storepass password -file cert.pem
keytool -importcert -alias rsocket -keystore client.truststore -storepass password -file cert.pem

Enable ssl via application.properties:

spring.rsocket.server.port=6565
spring.rsocket.server.ssl.enabled=true
spring.rsocket.server.ssl.key-store-type=PKCS12
spring.rsocket.server.ssl.key-store=<path to keystore>/rsocket-server.p12
spring.rsocket.server.ssl.key-password=password
spring.rsocket.server.ssl.key-alias=rsocket

Run a simple test:

@SpringBootTest
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
class SslTest(@Autowired private val builder: RSocketRequester.Builder) {
    companion object {
        init {
            System.setProperty("javax.net.ssl.trustStore", "<path to truststore>/client.truststore")
            System.setProperty("javax.net.ssl.trustStorePassword", "password")
        }
    }

    private val requester = builder.transport(
        TcpClientTransport.create(TcpClient.create().host("localhost").port(6565).secure()))

    @Test
    internal fun sslTlsTest() {
        val mono = requester.route("math.server.square")
            .data(ComputationRequestDto(5))
            .retrieveMono<ComputationResponseDto>()
            .doOnNext(::println)

        StepVerifier.create(mono).expectNextCount(1).verifyComplete()
    }
}

If you enable -Djavax.net.debug=all via the VM options, you can see (among other things):

javax.net.ssl|ALL|E1|reactor-http-nio-2|2022-01-07 16:59:26.146 CET|X509Authentication.java:312|rsocket is not a private key entry
javax.net.ssl|WARNING|E1|reactor-http-nio-2|2022-01-07 16:59:26.146 CET|CertificateMessage.java:1084|Unavailable authentication scheme: rsa_pkcs1_sha1
javax.net.ssl|WARNING|E1|reactor-http-nio-2|2022-01-07 16:59:26.146 CET|CertificateMessage.java:1094|No available authentication scheme
javax.net.ssl|ERROR|E1|reactor-http-nio-2|2022-01-07 16:59:26.147 CET|TransportContext.java:362|Fatal (HANDSHAKE_FAILURE): No available authentication scheme (
"throwable" : {
  javax.net.ssl.SSLHandshakeException: No available authentication scheme
  ...

And as a said in the beginning, as soon as you convert your PKCS12 keystore to a type of JKS, it worked as expected:

keytool -importkeystore -srckeystore rsocket-server.p12 -srcstoretype pkcs12 -srcalias rsocket -destkeystore rsocket-server.jks -deststoretype jks -deststorepass password -destalias rsocket

spring.rsocket.server.ssl.key-store-type=jks
spring.rsocket.server.ssl.key-store=<path to keystore>/rsocket-server.jks

Probably not a Spring issue. But maybe you already stumbled across this behaviour and could tell me what i obviously did wrong?

Comment From: philwebb

We don't seem to do a lot other than create the KeyStore and load the file. I don't know why the PKCS-12 file wouldn't work, but I'm not sure there's any bug here for us to fix.

I'm afraid I personally don't have a lot of experience with keystore files. Perhaps someone from the Netty team might be able to explain the "No available authentication scheme" error.

If there is something we need to change in our configuration, I'm more than happy to re-open this issue.