SpringBoot Provide auto-configuration for kotlinx.serialization and KotlinSerializationJsonHttpMessageConverter

I initially reported this here and here at the Spring Framework project, but the spring.mvc.converters.preferred-json-mapper property is apparently specific to Spring Boot, so I'm reporting it here.

Summary of the problem: when certain serde frameworks, notably kotlinx.serialization, are detected in the classpath by Spring, then the Spring Boot configuration property spring.mvc.converters.preferred-json-mapper (defined in HttpMessageConvertersAutoConfiguration.PREFERRED_MAPPER_PROPERTY) is effectively ignored.

The reason for this is that HttpMessageConvertersAutoConfiguration.PREFERRED_MAPPER_PROPERTY is only evaluated by GsonHttpMessageConvertersConfiguration, JacksonHttpMessageConvertersConfiguration and JsonbHttpMessageConverterConfiguration. There is however no such configuration class for kotlinx.serialization.

Instead, AllEncompassingFormHttpMessageConverter just adds kotlinx.serialization to the standard HttpMessageConverters as provided by HttpMessageConvertersAutoConfiguration.messageConverters(), and it happens to just get a higher priority than for instance Jackson when they are both in the classpath, regardless of whether the value spring.mvc.converters.preferred-json-mapper property is set and regardless of what value it is set to.

To be more specific: bean provider method HttpMessageConvertersAutoConfiguration.messageConverters() invokes one of the constructors of HttpMessageConverters, which does not explicitly set the parameter addDefaultConverters to false, and it remains true by default, which results in a call to the private helper method getCombinedConverters(), which (through some other private methods) eventually defers to WebMvcConfigurationSupport.getMessageConverters(), which calls the protected method addDefaultHttpMessageConverters(), which (among others) adds AllEncompassingFormHttpMessageConverter, which then autodetects a number of hard-coded serde frameworks in its init block.

To reproduce, add the following to the application.yml of a Spring Boot project:

spring:
  mvc:
    converters:
      preferred-json-mapper: jackson

Then add both Jackson and kotlinx.serialization dependencies to the project.

You will find that Spring will automatically pick KotlinSerializationJsonHttpMessageConverter over MappingJackson2HttpMessageConverter when (de)serializing JSON messages, even though Jackson was explicitly specified as the preferred JSON converter/mapper for the project.

Also, according to spring-configuration-metadata.json, the property spring.mvc.converters.preferred-json-mapper only supports the values gson, jackson and jsonb. It doesn't have an option defined to explicitly select kotlinx.serialization or any of the other serde frameworks that Spring autodetects in the static block of the following classes:

AllEncompassingFormHttpMessageConverter.java
BaseDefaultCodecs.java
DefaultRestClientBuilder.java
RestTemplate.java
WebMvcConfigurationSupport.java

This doesn't seem consistent, and it cost me quite a few hours to figure out why the addition of a non-spring library to my Spring project suddenly led to the project switching from Jackson to kotlinx.serialization, without me changing anything to the Spring configuration.

In the meantime, I found that the following workaround worked for me to force Jackson to be used, even if kotlinx.serialization was included in the classpath as a transitive dependency of another library:

import org.springframework.beans.factory.ObjectProvider
import org.springframework.boot.autoconfigure.http.HttpMessageConverters
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.http.converter.AbstractKotlinSerializationHttpMessageConverter
import org.springframework.http.converter.HttpMessageConverter

@Configuration
class MessageConvertersConfig {
    /**
     * Bean provider that filters out standard [HttpMessageConverter]s that we don't want Spring to use, even when the
     * serde frameworks for them are detected by Spring in the classpath.
     *
     * Should override [org.springframework.boot.autoconfigure.http.HttpMessageConvertersAutoConfiguration.messageConverters]
     */
    @Bean
    fun filteredMessageConverters(converters: ObjectProvider<HttpMessageConverter<*>>): HttpMessageConverters =
        HttpMessageConverters(
            false,
            converters.filter { converter ->
                converter !is AbstractKotlinSerializationHttpMessageConverter<*>
            },
        )
}

But I'm reporting this as a bug, since this is not reasonably expected behavior, and it's not consistent (between the various supported serde frameworks) either.

Please let me know in this thread if you need any additional information.

Thank you for your time.

Comment From: wilkinsona

Until we find time to look at this in detail, as an alternative workaround, you may want to consider defining your own HttpMessageConverters bean that overrides postProcessConverters and postProcessPartConverters to filter out the unwanted converter.

Comment From: volkert-fastned

@wilkinsona Thanks for the alternative workaround. Can you maybe explain what the advantage of that approach is over the workaround I shared above?

Comment From: wilkinsona

Using the postProcess hooks points offers more precise control over the end result. Perhaps not needed here, but it's the recommended approach and this sort of situation is why those methods were introduced.

Comment From: volkert-fastned

@wilkinsona Thanks! By the way, when I have overridden postProcessConverters(), I don't also have to override postProcessPartConverters(), correct? The latter only applies in a subset of cases, do I interpret that correctly from the code?

Based on the example you linked to, I ended up with this (in Kotlin), which indeed works as well, at least in our case:

import org.springframework.boot.autoconfigure.http.HttpMessageConverters
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.http.converter.AbstractKotlinSerializationHttpMessageConverter
import org.springframework.http.converter.HttpMessageConverter

@Configuration
class MessageConvertersConfig {
    /**
     * Bean provider that filters out standard [HttpMessageConverter]s that we don't want Spring to use, even when the
     * serde frameworks for them are detected by Spring in the classpath.
     *
     * With thanks to https://github.com/spring-projects/spring-boot/issues/39853#issuecomment-1984360351 and
     * https://github.com/spring-projects/spring-boot/issues/1482#issuecomment-61862787
     *
     * Should override [org.springframework.boot.autoconfigure.http.HttpMessageConvertersAutoConfiguration.messageConverters]
     */
    @Bean
    fun filteredMessageConverters(): HttpMessageConverters =
        object : HttpMessageConverters() {
            override fun postProcessConverters(converters: MutableList<HttpMessageConverter<*>>): MutableList<HttpMessageConverter<*>> {
                converters.removeIf {
                    it is AbstractKotlinSerializationHttpMessageConverter<*>
                }

                return converters
            }
        }
}

Comment From: wilkinsona

By the way, when I have overridden postProcessConverters(), I don't also have to override postProcessPartConverters(), correct? The latter only applies in a subset of cases, do I interpret that correctly from the code?

Yes, that's right. postProcessPartConverters() only applies to the converters that are used by AllEncompassingFormHttpMessageConverter. If your app isn't receiving multi-part requests, the converters used by this converter converter won't need to be customized.

Just to make sure I have understood your situation correctly, am I right in understanding that:

Your application has Jackson on the classpath
Your application has kotlinx.serialization on the classpath
You are (de)serializing a type that kotlinx.serialization supports (kotlinx.serialization.SerializersKt.serializerOrNull returns a KSerializer for the type) but you want Jackson to be used instead

Assuming that the above is correct, is kotlinx.serialization on the classpath for some other reason or is it there accidentally? Just trying to understand the scope of the problem and gauge how likely others are to be affected by it – this is the first time it's been raised in Boot's issues as far as I can remember.

Comment From: volkert-fastned

@wilkinsona You summarized the points correctly, but just to add some extra context to point 2: we are including a library dependency in our project, which uses kotlinx.serialization and pulls it into the project as a transitive dependency. That's why it's showing up in our classpath alongside Jackson.

The application as a whole, however, is still set up to use Jackson for (de)serialization (apart from the library that we're including for one specific purpose). We may decide to migrate the entire project from Jackson to kotlinx.serialization at some point, but that wouldn't be a trivial, so we prefer to maintain the same behavior throughout the application for now.

By the way, thinking about it more, I'm not entirely sure about point 3. It's plausible, but I simply experienced certain integration tests in the application starting to fail, because it switched from Jackson to kotlinx.serialization, just because it found both serde frameworks in the classpath. And I know it did so, because I could see in the stacktraces that kotlinx.serialization was being invoked. So that seems to imply that point 3 is true.

But even if it's not common to have multiple (Spring-supported) serde frameworks in the classspath, wouldn't it be reasonable to expect that the one explicitly selected in spring.mvc.converters.preferred-json-mapper would be used anyway?

Anyway, I hope this answers your question. Just let me know in this thread if more information or clarification is needed. Thanks!

Comment From: be-hase

Hi. I also encountered this issue. In my product, I use Jackson. One day, I added a certain library, and it implicitly depended on kotlinx.serialization. As a result, kotlinx.serialization started being used in Spring Boot's serde as well. Since I had custom settings like snake case in Jackson, it ended up causing a bug.

I'd be happy to see some progress.

Comment From: wilkinsona

Fixing this in a maintenance release isn't straightforward, unfortunately.

If we prioritise Jackson (or Gson, or Jsonb) over kotlinx.serialization it will be a breaking change for those who are relying on the current behavior where kotlinx.serialization is preferred. They could work around this by providing their own HttpMessageConverters instance as described above but this is no different to what's currently necessary for those who want the opposite behavior.

It could be argued that changing the behavior would align it with the documentation that currently states that Jackson is preferred. Unfortunately, given that things have behaved as they currently do ever since Spring Framework introduced support for kotlinx.serialization, I think the runtime behaviour has to trump the documentation.

I think the best way to fix this is to add support for auto-configuring kotlinx.serialization. As part of that, we'd modify HttpMesageConverters so that it was aware of KotlinSerializationJsonHttpMessageConverter and ordered it appropriately. We'd also support it as a value of spring.mvc.converters.preferred-json-mapper so that continuing to use kotlinx.serialization after the ordering change required nothing more than setting the property.

We can tackle this in 3.4. Until then, a custom HttpMessageConverters instance that orders the message converters as required is the recommended approach. I've opened https://github.com/spring-projects/spring-boot/issues/40767 to document it.

Comment From: be-hase

As you mentioned, I agree that changing it is difficult because there are users who depend on the current behavior.

I would appreciate it if there could be some sort of guide in the documentation for now. In the medium to long term, it would be great if we could change it through Spring Boot configuration.

Comment From: volkert-fastned

@wilkinsona I concur that you need to be considerate of people who depend on current behavior.

But I don't think that applies when spring.mvc.converters.preferred-json-mapper is explicitly configured in a project. If this is set to jackson and kotlinx.serialization gets picked over Jackson despite this, then that's a bug, no matter how you slice it.

So a distinction should be made here, IMO:

If spring.mvc.converters.preferred-json-mapper is not explicitly set in a project (or it's set to a value that is not yet supported in the current Spring version), then yes, let's not mess with the current behavior, at least not before 3.4.
If spring.mvc.converters.preferred-json-mapper is set to gson, jackson, or jsonb, then that specified serde framework should really be picked over kotlinx.serialization when both are found in the classpath, and this should be fixed as a bug, without waiting for 3.4. There is no way that this would break "expected" behavior for anyone.

Unless I somehow completely misunderstood what you are trying to say.

Comment From: wilkinsona

Unfortunately, that's easier said than done.

jackson is, effectively, the default value for spring.mvc.converters.preferred-json-mapper so setting it to jackson has no effect on the beans that are configured. Either way, a MappingJackson2HttpMessageConverter is auto-configured. It's picked up by HttpMessageConverters as an additional converter where its internal logic causes it to take precedence over the default Jackson, Gson, and Jsonb converters. To have it take precedence over the default kotlinx.serialization converter would require a change to its ordering logic which would break those who rely on the existing behavior.

We could add additional logic to HttpMessageConverters so that its ordering can be changed when spring.mvc.converters.preferred-json-mapper has been set to jackson. However, that would add complexity and probably public API as well so that those defining an HttpMessageConverters bean themselves could also make use of it. At this point, I think the cure's likely worse than the disease. I continue to believe that the right balance here is to address this through documentation in the short term and by adding complete support for kotlinx.serialization in the longer term.

Comment From: volkert-fastned

Ah, so it's not trivial to distinguish between jackson being an implicit default value and jackson having been set explicitly. Got it. Thanks for clarifying that. Your approach indeed seems prudent then. 👍

Comment From: wilkinsona

Duplicates https://github.com/spring-projects/spring-boot/issues/24238.