Provide auto-configuration for kotlinx.serialization and KotlinSerializationJsonHttpMessageConverter

[volkert-fastned]

I initially reported this here and here at the Spring Framework project, but the spring.mvc.converters.preferred-json-mapper property is apparently specific to Spring Boot, so I'm reporting it here.

Summary of the problem: when certain serde frameworks, notably kotlinx.serialization, are detected in the classpath by Spring, then the Spring Boot configuration property spring.mvc.converters.preferred-json-mapper (defined in HttpMessageConvertersAutoConfiguration.PREFERRED_MAPPER_PROPERTY) is effectively ignored.

The reason for this is that HttpMessageConvertersAutoConfiguration.PREFERRED_MAPPER_PROPERTY is only evaluated by GsonHttpMessageConvertersConfiguration, JacksonHttpMessageConvertersConfiguration and JsonbHttpMessageConverterConfiguration. There is however no such configuration class for kotlinx.serialization.

Instead, AllEncompassingFormHttpMessageConverter just adds kotlinx.serialization to the standard HttpMessageConverters as provided by HttpMessageConvertersAutoConfiguration.messageConverters(), and it happens to just get a higher priority than for instance Jackson when they are both in the classpath, regardless of whether the value spring.mvc.converters.preferred-json-mapper property is set and regardless of what value it is set to.

To be more specific: bean provider method HttpMessageConvertersAutoConfiguration.messageConverters() invokes one of the constructors of HttpMessageConverters, which does not explicitly set the parameter addDefaultConverters to false, and it remains true by default, which results in a call to the private helper method getCombinedConverters(), which (through some other private methods) eventually defers to WebMvcConfigurationSupport.getMessageConverters(), which calls the protected method addDefaultHttpMessageConverters(), which (among others) adds AllEncompassingFormHttpMessageConverter, which then autodetects a number of hard-coded serde frameworks in its init block.

To reproduce, add the following to the application.yml of a Spring Boot project:

spring:
  mvc:
    converters:
      preferred-json-mapper: jackson

Then add both Jackson and kotlinx.serialization dependencies to the project.

You will find that Spring will automatically pick KotlinSerializationJsonHttpMessageConverter over MappingJackson2HttpMessageConverter when (de)serializing JSON messages, even though Jackson was explicitly specified as the preferred JSON converter/mapper for the project.

Also, according to spring-configuration-metadata.json, the property spring.mvc.converters.preferred-json-mapper only supports the values gson, jackson and jsonb. It doesn't have an option defined to explicitly select kotlinx.serialization or any of the other serde frameworks that Spring autodetects in the static block of the following classes:

• AllEncompassingFormHttpMessageConverter.java
• BaseDefaultCodecs.java
• DefaultRestClientBuilder.java
• RestTemplate.java
• WebMvcConfigurationSupport.java

This doesn't seem consistent, and it cost me quite a few hours to figure out why the addition of a non-spring library to my Spring project suddenly led to the project switching from Jackson to kotlinx.serialization, without me changing anything to the Spring configuration.

In the meantime, I found that the following workaround worked for me to force Jackson to be used, even if kotlinx.serialization was included in the classpath as a transitive dependency of another library:

import org.springframework.beans.factory.ObjectProvider
import org.springframework.boot.autoconfigure.http.HttpMessageConverters
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.http.converter.AbstractKotlinSerializationHttpMessageConverter
import org.springframework.http.converter.HttpMessageConverter

@Configuration
class MessageConvertersConfig {
    /**
     * Bean provider that filters out standard [HttpMessageConverter]s that we don't want Spring to use, even when the
     * serde frameworks for them are detected by Spring in the classpath.
     *
     * Should override [org.springframework.boot.autoconfigure.http.HttpMessageConvertersAutoConfiguration.messageConverters]
     */
    @Bean
    fun filteredMessageConverters(converters: ObjectProvider<HttpMessageConverter<*>>): HttpMessageConverters =
        HttpMessageConverters(
            false,
            converters.filter { converter ->
                converter !is AbstractKotlinSerializationHttpMessageConverter<*>
            },
        )
}

But I'm reporting this as a bug, since this is not reasonably expected behavior, and it's not consistent (between the various supported serde frameworks) either.

Please let me know in this thread if you need any additional information.

Thank you for your time.

[wilkinsona]

Until we find time to look at this in detail, as an alternative workaround, you may want to consider defining your own HttpMessageConverters bean that overrides postProcessConverters and postProcessPartConverters to filter out the unwanted converter.

[volkert-fastned]

@wilkinsona Thanks for the alternative workaround. Can you maybe explain what the advantage of that approach is over the workaround I shared above?

[wilkinsona]

Using the postProcess hooks points offers more precise control over the end result. Perhaps not needed here, but it's the recommended approach and this sort of situation is why those methods were introduced.

[volkert-fastned]

@wilkinsona Thanks! By the way, when I have overridden postProcessConverters(), I don't also have to override postProcessPartConverters(), correct? The latter only applies in a subset of cases, do I interpret that correctly from the code?

Based on the example you linked to, I ended up with this (in Kotlin), which indeed works as well, at least in our case:

import org.springframework.boot.autoconfigure.http.HttpMessageConverters
import org.springframework.context.annotation.Bean
import org.springframework.context.annotation.Configuration
import org.springframework.http.converter.AbstractKotlinSerializationHttpMessageConverter
import org.springframework.http.converter.HttpMessageConverter

@Configuration
class MessageConvertersConfig {
    /**
     * Bean provider that filters out standard [HttpMessageConverter]s that we don't want Spring to use, even when the
     * serde frameworks for them are detected by Spring in the classpath.
     *
     * With thanks to https://github.com/spring-projects/spring-boot/issues/39853#issuecomment-1984360351 and
     * https://github.com/spring-projects/spring-boot/issues/1482#issuecomment-61862787
     *
     * Should override [org.springframework.boot.autoconfigure.http.HttpMessageConvertersAutoConfiguration.messageConverters]
     */
    @Bean
    fun filteredMessageConverters(): HttpMessageConverters =
        object : HttpMessageConverters() {
            override fun postProcessConverters(converters: MutableList<HttpMessageConverter<*>>): MutableList<HttpMessageConverter<*>> {
                converters.removeIf {
                    it is AbstractKotlinSerializationHttpMessageConverter<*>
                }

                return converters
            }
        }
}

[wilkinsona]

By the way, when I have overridden postProcessConverters(), I don't also have to override postProcessPartConverters(), correct? The latter only applies in a subset of cases, do I interpret that correctly from the code?

Yes, that's right. postProcessPartConverters() only applies to the converters that are used by AllEncompassingFormHttpMessageConverter. If your app isn't receiving multi-part requests, the converters used by this converter converter won't need to be customized.

Just to make sure I have understood your situation correctly, am I right in understanding that:

1. Your application has Jackson on the classpath
2. Your application has kotlinx.serialization on the classpath
3. You are (de)serializing a type that kotlinx.serialization supports (kotlinx.serialization.SerializersKt.serializerOrNull returns a KSerializer for the type) but you want Jackson to be used instead

Assuming that the above is correct, is kotlinx.serialization on the classpath for some other reason or is it there accidentally? Just trying to understand the scope of the problem and gauge how likely others are to be affected by it – this is the first time it's been raised in Boot's issues as far as I can remember.

[volkert-fastned]

@wilkinsona You summarized the points correctly, but just to add some extra context to point 2: we are including a library dependency in our project, which uses kotlinx.serialization and pulls it into the project as a transitive dependency. That's why it's showing up in our classpath alongside Jackson.

The application as a whole, however, is still set up to use Jackson for (de)serialization (apart from the library that we're including for one specific purpose). We may decide to migrate the entire project from Jackson to kotlinx.serialization at some point, but that wouldn't be a trivial, so we prefer to maintain the same behavior throughout the application for now.

By the way, thinking about it more, I'm not entirely sure about point 3. It's plausible, but I simply experienced certain integration tests in the application starting to fail, because it switched from Jackson to kotlinx.serialization, just because it found both serde frameworks in the classpath. And I know it did so, because I could see in the stacktraces that kotlinx.serialization was being invoked. So that seems to imply that point 3 is true.

But even if it's not common to have multiple (Spring-supported) serde frameworks in the classspath, wouldn't it be reasonable to expect that the one explicitly selected in spring.mvc.converters.preferred-json-mapper would be used anyway?

Anyway, I hope this answers your question. Just let me know in this thread if more information or clarification is needed. Thanks!

[be-hase]

Hi. I also encountered this issue. In my product, I use Jackson. One day, I added a certain library, and it implicitly depended on kotlinx.serialization. As a result, kotlinx.serialization started being used in Spring Boot's serde as well. Since I had custom settings like snake case in Jackson, it ended up causing a bug.

I'd be happy to see some progress.