The Spring Boot 2.6.5 update includes an update for Kafka 3.0.0 to 3.0.1. This update changed the default for spring.kafka.producer.properties.enable.idempotence from false to true. As "idempotence" needs certain privileges on the Kafka servers, our clients stopped working with an ClusterAuthorizationException.
Please add a note to the release notes at https://github.com/spring-projects/spring-boot/releases/tag/v2.6.5 explaining that setting spring.kafka.producer.properties.enable.idempotence=false fixes this problem.
The Kafka change log which mentions this change of defaults is at https://docs.confluent.io/platform/current/release-notes/index.html:
Producer defaults now enable the strongest delivery guarantee (ack=all and idempotence=true). This means that you now get ordering and durability by default.
The error that we got:
12:23:01.324 [kafka-producer-network-thread | xxx-1] [ ] DEBUG org.apache.kafka.clients.NetworkClient - [Producer clientId=xxx-1] Received INIT_PRODUCER_ID response from node -3 for request with header RequestHeader(apiKey=INIT_PRODUCER_ID, apiVersion=4, clientId=xxx-1, correlationId=3): InitProducerIdResponseData(throttleTimeMs=0, errorCode=31, producerId=-1, producerEpoch=-1)
12:23:01.326 [kafka-producer-network-thread | xxx-1] [ ] INFO org.apache.kafka.clients.producer.internals.TransactionManager - [Producer clientId=xxx-1] Transiting to fatal error state due to org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
Comment From: wilkinsona
It looks like they intended to enable idempotence and set acks to all by default in 3.0.0 but only the acks change actually took effect. This was then corrected in 3.0.1, causing the change in behaviour that you observed in Boot 2.6.5.
We have already linked to the Kafka 3.0.0 announcement blog post from Boot 2.6's release notes and the blog post mentions the (ineffective till Kafka 3.0.1) switch to enable.idempotence=true. I'm not sure what, if anything, we should do here. On the one hand, this is quite a nasty breaking change, on the other, we can't describe every change in a third-party dependency in our changelog, even if it's a breaking change, and I'm a bit wary of creating an expectation that we might do so. I'm going to flag this for team attention to see what everyone else thinks.
Comment From: mhalbritter
I guess this will pop up for a lot of users using Kafka. I would add it to the changelog to save peoples time searching a solution for this problem. Maybe with a sentence stating "it came to our attention that ...", which then will hopefully not create the expectation that we can foresee or document every breaking change in 3rd party libs.
I think this is an exception to the rule that we document changes in 3rd party libs, as it was a bug in Kafka that this setting was ineffective and then activated on a Spring Boot maintenance release. I'd see it as a service for our users to save them some time.
Comment From: mbhave
+1 for adding something to our release notes. We've done it in the past with Hibernate Validator. Also, since the default can be flipped using a Spring Boot configuration property, I think it makes sense to add a note about it.
Comment From: wilkinsona
Thanks, both. I've added a section to the release notes and also linked to it from the 2.6.5 changelog.
Comment From: ValentinBossi
most propably you need to add an acl ressource "cluster" with allowed operation "IdempotenceWrite" to get rid of the exception (1). or just via cli (2):
bin/kafka-acls --authorizer-properties zookeeper.connect=localhost:2181 \
--add --allow-principal User:Bob \
--producer --topic test-topic --idempotent
- https://kafka.apache.org/documentation/#operations_resources_and_protocols
- https://docs.confluent.io/5.3.0/kafka/authorization.html#enabling-authorization-for-idempotent-and-transactional-apis