Spring Boot Kafka SSL Issue on EKS Deployment using Spring boot 3.4.3(tried 3.3.5 and 3.4.2) We are facing an issue where the Kafka producer works fine in local environments (IntelliJ, mvn spring-boot:run, and java -jar). However, after deploying to EKS, we encounter the following exception:
Exception occurred: org.apache.kafka.common.config.ConfigException: Invalid value org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory for configuration ssl.engine.factory.class: Class org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory could not be found.. Stack Trace: [org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:778), org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:531), org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:114), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:134), org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:643), org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:295), org.springframework.kafka.core.DefaultKafkaProducerFactory.createRawProducer(DefaultKafkaProducerFactory.java:944), org.springframework.kafka.core.DefaultKafkaProducerFactory.createKafkaProducer(DefaultKafkaProducerFactory.java:826), org.springframework.kafka.core.DefaultKafkaProducerFactory.doCreateProducer(DefaultKafkaProducerFactory.java:793), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:768), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:762), org.springframework.kafka.core.KafkaTemplate.getTheProducer(KafkaTemplate.java:976), org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:828), org.springframework.kafka.core.KafkaTemplate.observeSend(KafkaTemplate.java:805), org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:608),
Issue Details: Kafka producer initialization delayed: Instead of initializing at application startup, it only starts when sending a message, leading to the SslBundleSslEngineFactory class not being found. Temporary workaround: We manually create the producer in a @PostConstruct method inside a KafkaConfig class.
@Slf4j
@Configuration
@RequiredArgsConstructor
public class KafkaConfig {
private final ProducerFactory<String, String> producerFactory;
@PostConstruct
public void initializeProducer() {
try {
Producer<String, String> producer = producerFactory.createProducer(); // Workaround to initialize Kafka producer
log.info("Kafka Producer initialized successfully: {}", producer);
} catch (Exception e) {
log.error("Failed to initialize Kafka Producer", e);
}
}
}
Spring Boot Configuration (application.yml)
spring:
application:
name: test
ssl:
bundle:
pem:
kafkaCert:
keystore:
certificate: classpath:KafkaKeystoreCert.pem
private-key: classpath:KafkaKeystoreKey.pem
truststore:
certificate: classpath:KafkaTruststorePem.pem
kafka:
bootstrap-servers: ${KafkaBootstrapServers}
ssl:
bundle: kafkaCert
security:
protocol: SSL
client-id: client-id
producer:
retries: 3
key-serializer: org.apache.kafka.common.serialization.StringSerializer
value-serializer: org.apache.kafka.common.serialization.StringSerializer
Request for a Proper Solution Do we have a proper fix for this issue instead of the workaround?
Is there a missing dependency or classpath issue specific to the environment? Are there configurations that need adjustment to ensure Kafka SSL settings load correctly? Any best practices for ensuring SslBundleSslEngineFactory is properly available at runtime?
Comment From: mhalbritter
This could be a classloader issue. It does work locally with SSL enabled, too? Or is SSL only enabled in EKS?
Please take the time to provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.
Comment From: kst1980
Yes, it works locally with the same configuration and SSL enabled, but I read that it might be functioning correctly locally due to the JVM.
I have added the following dependency, which provides org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory, and verified its presence in the image after packaging:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
Since the Spring Kafka producer is lazy-initialized, it throws a ClassNotFoundException after the application startup.
Comment From: mhalbritter
Please provide a sample to reproduce.
Comment From: kst1980
kafka-producer.zip - sample
Comment From: mhalbritter
Hey, do you have any instructions how to reproduce the issue?
I've added a compose file and some SSL config (the certs in your sample are all empty), and it works, both in tests and locally when running with java -jar. Do i need to do something else to see it fail?
Since the Spring Kafka producer is lazy-initialized, it throws a ClassNotFoundException after the application startup.
Also I don't see any spring-boot-autoconfigure, which you mentioned here.
Comment From: kst1980
spring-boot-autoconfigure is included in spring-boot-starter, so I didn't add it explicitly. As I mentioned earlier, everything works fine locally. However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.
Comment From: patpatpat123
I do not want to pollute this thread, apologies.
I am facing the same issue with all the above springboot versions mentioned.
For me, I have a case where it is working in "normal" mode and local, but I always encounter this same issue when running with GraalVM native image
Just wondering if I should create a new issue, targeting specifically native images, or if this is related.
Comment From: mhalbritter
@kst1980
However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.
I still think this is a classloader issue. The classloader associated with the thread when using the @PostConstruct workaround is able to the load the class, so it's not a missing dependency.
Is there something different in the dependencies or are any agents attached when running in EKS?
Comment From: mhalbritter
I do not want to pollute this thread, apologies.
I am facing the same issue with all the above springboot versions mentioned.
For me, I have a case where it is working in "normal" mode and local, but I always encounter this same issue when running with GraalVM native image
Just wondering if I should create a new issue, targeting specifically native images, or if this is related.
Please open a new issue, and please attach a reproducer. Thanks!
// Edit: Nevermind, there's a class hint missing. While it's the same error message as this issue, it's not related. I've opened https://github.com/spring-projects/spring-boot/issues/44435 for that.
Comment From: mhalbritter
Instead of putting SslBundleSslEngineFactory as a string in the configuration map, we decided to put the class directly in the config map. This hopefully fixes the classloading issue.
Comment From: kst1980
However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.
I still think this is a classloader issue. The classloader associated with the thread when using the
@PostConstructworkaround is able to the load the class, so it's not a missing dependency.Is there something different in the dependencies or are any agents attached when running in EKS?
Using the Datadog Java Agent https://dtdg.co/latest-java-tracer, when can I test this fix and with which version?
Comment From: mhalbritter
Maybe that's the problem. The fix (hopefully) will be released as 3.3.10 next month (https://calendar.spring.io/), or available on Spring Snapshots repo as 3.3.10-SNAPSHOT.