Scala lib version mismatch in Spring Kafka in Spring Boot 2.3.5

Frankly I do not know if this is Spring Boot bug or Spring Kafka Bug. After upgrading to Spring Boot 2.3.5, we started to get following error when starting embedded Kafka Test

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'embeddedKafka': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1794)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:406)
    at org.springframework.kafka.test.context.EmbeddedKafkaContextCustomizer.customizeContext(EmbeddedKafkaContextCustomizer.java:116)
    at org.springframework.boot.test.context.SpringBootContextLoader$ContextCustomizerAdapter.initialize(SpringBootContextLoader.java:302)
    at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:626)
    at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:370)
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
    at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:120)
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:99)
    at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
    ... 50 more
Caused by: java.lang.NoClassDefFoundError: scala/math/Ordering$$anon$7
    at kafka.api.ApiVersion$.orderingByVersion(ApiVersion.scala:45)
    at kafka.api.ApiVersion.compare(ApiVersion.scala:139)
    at kafka.api.ApiVersion.compare$(ApiVersion.scala:138)
    at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
    at kafka.api.KAFKA_2_5_IV0$.compare(ApiVersion.scala:339)
    at scala.math.Ordered.$greater$eq(Ordered.scala:91)
    at scala.math.Ordered.$greater$eq$(Ordered.scala:91)

It's caused by scala library mismatch which is caused by Jackson library upgrade as shown here:

|    +--- org.apache.kafka:kafka_2.12:2.5.1
|    |    +--- org.apache.kafka:kafka-clients:2.5.1 (*)
|    |    +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 -> 2.11.3 (*)
|    |    +--- com.fasterxml.jackson.module:jackson-module-scala_2.12:2.10.2 -> 2.11.3
|    |    |    +--- org.scala-lang:scala-library:2.12.12
|    |    |    +--- com.fasterxml.jackson.core:jackson-core:2.11.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.11.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.11.3 (*)
|    |    |    \--- com.fasterxml.jackson.module:jackson-module-paranamer:2.11.3
|    |    |         +--- com.fasterxml.jackson.core:jackson-databind:2.11.3 (*)
|    |    |         \--- com.thoughtworks.paranamer:paranamer:2.8
|    |    +--- com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.10.2 -> 2.11.3
|    |    |    +--- com.fasterxml.jackson.core:jackson-databind:2.11.3 (*)
|    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.11.3
|    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.11.3
|    |    +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.2 -> 2.11.3 (*)
|    |    +--- net.sf.jopt-simple:jopt-simple:5.0.4
|    |    +--- com.yammer.metrics:metrics-core:2.2.0
|    |    |    \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.30
|    |    +--- org.scala-lang.modules:scala-collection-compat_2.12:2.1.3
|    |    |    \--- org.scala-lang:scala-library:2.12.8 -> 2.12.12
|    |    +--- org.scala-lang.modules:scala-java8-compat_2.12:0.9.0
|    |    |    \--- org.scala-lang:scala-library:2.12.4 -> 2.12.12
|    |    +--- org.scala-lang:scala-library:2.12.10 -> 2.12.12
|    |    +--- org.scala-lang:scala-reflect:2.12.10 -> 2.12.11
|    |    |    \--- org.scala-lang:scala-library:2.12.11 -> 2.12.12
|    |    +--- com.typesafe.scala-logging:scala-logging_2.12:3.9.2
|    |    |    +--- org.scala-lang:scala-library:2.12.7 -> 2.12.12
|    |    |    +--- org.scala-lang:scala-reflect:2.12.7 -> 2.12.11 (*)
|    |    |    \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30

It seems that jackson-databind version is overriden by Spring Boot which in turn introduces newer version of jackson-module-scala which in turn introduces org.scala-lang:scala-library:2.12.12 which is not compatible with 2.12.11 because Scala.

Excluding jackson-module-scala_2.12 from spring kafka dependencies helps. The project is built by Gradle.

Comment From: lukas-krecan

May be related to https://github.com/spring-projects/spring-kafka/commit/5f6ec92ba607991a8c129eb2faffaeac1f815223

Comment From: wilkinsona

introduces org.scala-lang:scala-library:2.12.12 which is not compatible with 2.12.11 because Scala.

This is the problem in a nutshell and I don't think it's something that we should try to address in Spring Boot. One thing we could do would be to adding dependency management for Scala, downgrading it to 2.12.11, however this has a couple of notable downsides:

  1. We'd be stuck on an old version of Scala so users will miss out on fixes in 2.12.12 and later
  2. This is only a problem is you're using Kafka and it doesn't feel write to control the version of Scala that used for every to address a problem that's specific to users of Kafka.

Another option would be to downgrade Jackson to avoid it pulling in the broken version of Scala. This has similar downsides to those above, leaving us stuck on an old version of Jackson with a change that affects everyone rather than only those that are using Kafka.

If you are affected by this problem then, until such a time as the Scala team restores backwards compatibility or the Kafka team works around the breaking change in Scala, I think your best bet is to manage the versions of the Scala dependencies yourself to meet you specific needs.

Comment From: boybundit

The workaround can be found here https://stackoverflow.com/questions/64669007/noclassdeffounderror-scala-math-ordering-with-spring-kafka-test-2-5-7