Spring Boot provides a great helper configuration KafkaProperties class for Kafka.
This class includes two spring-kafka
module classes :
import org.springframework.kafka.listener.ContainerProperties.AckMode;
import org.springframework.kafka.security.jaas.KafkaJaasLoginModuleInitializer;
Due to those imports, in order to leverage Spring Boot's Kafka auto configuration, it is required to import spring-kafka
module which is largely increasing the application size (our projects are using Reactor Kafka).
IMHO, the configuration that depends on spring-kafka
should be moved to spring-kafka
module, allowing Spring Boot's projects to leverage automatic type safe Kafka configuration without importing spring-kafka
.
Comment From: wilkinsona
Thanks for the suggestion. The whole auto-configuration, of which KafkaProperties
is a part, depends on Spring for Apache Kafka. In other words, it's intentionally auto-configuration for Spring for Apache Kafka and not for Kafka itself. The auto-configuration will back off completely in the absence of Spring for Apache Kafka. Also, you shouldn't try to use KafkaProperties
in your own code. As noted in the documentation, all configuration properties classes are for internal use only:
Similar to auto-configuration classes,
@ConfigurationProperties
classes available in Spring Boot are for internal use only. The properties that map to the class, which are configured via properties files, YAML files, environment variables etc., are public API but the content of the class itself is not meant to be used directly.
I don't think it makes sense for Spring Boot to offer configuration for Kafka without using Spring for Apache Kafka. Our opinion is that it makes sense to use the latter when interacting with Kafka in a Spring application. If you have use cases where that's not true, please share them here and we can reconsider or pass them on to the Spring for Apache Kafka team for them to look at.
Comment From: jntakpe
@wilkinsona, all our applications are microservices that relies on Spring Boot + Spring Webflux. All those microservices fully leverages Reactive programming so it was pretty obvious for us to connect to Kafka embracing reactive libraries. That's why we are using reactor-kafka to communicate with Kafka. Even if Spring-kafka supports some async types, IMHO it's not fully embracing reactive programming like reactor-kafka does. We are very pleased by all the effort made so for by Spring teams to push Java reactive support forward, it seems logical that Spring Boot enables easy Kafka configuration through KafkaProperties class letting users choose the fully reactive implementation reactor-kafka or the classic spring-kafka.
Comment From: wilkinsona
Thanks. So it sounds like what you're really looking for is auto-configuration support for Reactor Kafka. Can you share some details of how you're currently using Reactor Kafka and how you're using KafkaProperties
to configure it?
Comment From: jntakpe
Actually that's pretty straightforward, we inject KafkaProperties
to create both KafkaSender
and KafkaReceiver
beans. It looks like, the following sample :
@Bean
fun sender(properties: KafkaProperties): KafkaSender<String, String> {
val options = SenderOptions.create<String, String>(properties.buildProducerProperties())
return KafkaSender.create<String, String>(options)
}
Hence both KafkaSender
and KafkaReceiver
are configured using regular Spring properties.
The full reactor auto configuration support would be great. Nevertheless, being able to simply inject KafkaProperties
without being forced to add spring-kafka to the classpath is our main concern
Comment From: wilkinsona
Thanks for the additional information.
Nevertheless, being able to simply inject KafkaProperties without being forced to add spring-kafka to the classpath is our main concern
This (alone) almost certainly won't happen. As I mentioned above, Boot's own @ConfigurationProperties
classes should be considered internal so you should not be injecting KafkaProperties
. Given that the only supported use of KafkaProperties
is by the auto-configuration for Spring for Apache Kafka, and its needs are met by the current arrangement, it will be difficult to justify modifying KafkaProperties
as you have described.
Comment From: jntakpe
So you intent to provide a reactor-kafka auto configuration mechanism ?
Comment From: wilkinsona
We don't know yet, hence this issue still being labelled as waiting for triage. We can certainly consider it though. In the meantime, I would strongly encourage you to use your own configuration properties and @ConfigurationProperties
class for the configuration of Reactor Kafka.
Comment From: jntakpe
Yes that's what I intent to do. Thanks !
Comment From: wilkinsona
Having discussed this with the Reactor team, we've decided not to add auto-configuration for Reactor Kafka at this time. If the situation changes, we can re-open this issue and reconsider. Thanks for the suggestion, @jntakpe.
Comment From: matthewadams
Understand the decision. Can you provide a quick tldr-style example here (or link thereto) that uses @ConfigurationProperties
?
Comment From: wilkinsona
@matthewadams Unfortunately we don't have such an example to hand. The properties would really depend on your use case and the Reactor Kafka settings that you want to be able to configure using externalised properties.