SpringBoot As a spring kafka user i want to set clusterid filed through yml properties so that Spring Kafka Observation works seemlessly

How to work with Spring Kafka Observation without clusterId ?

Problem: while using spring Kafka observation, KafkaTemplate and KafkaMessageListenerContainer creates KafkaRecordSenderContext & KafkaRecordReceiverContext objects with clusterid derived from KafkaAdmin

But depending on the environment KafkaAdmin may not work expected for example in AWS MSK, we need to have describe cluster permission to fetch the cluster id details and even fetching cluster id value with KafkaAdmin#Describecluster is time consuming process. It will be even bigger issue if for each message application tries to fetch clusterid (say for example no access to describe cluster).

Some times due to repetitive process of fetching cluster id we are observing memory issue and Consumer Lag.

It will be better similar to other properties that user can set the clusterid under prefix spring.kafka.admin, fetch clusterid value only if the the corresponding field is not set in yml properties.

Comment From: philwebb

It looks like we could call KafkaAdmin.setClusterId(...) in KafkaAutoConfiguration to do this. In the meantime, you could probably write a BeanPostProcessor to call this method yourself.

Comment From: wilkinsona

Duplicates #38825.