The default charset-parameter added to the Content-Type header when using FormHttpMessageWriter or FormHttpMessageConverter with SPR-16613 is causing some trouble if the server does not support specifying a charset.

As the spec for application/x-www-urlencoded did not exist for a long time and the one provided by whatwg - application/x-www-form-urlencoded is still somewhat vague, both clients and servers have implemented various approaches for dealing with this MIME-type.

Earlier HTML specs mention the non-intuitive behavior:

Parameters on the application/x-www-form-urlencoded MIME type are ignored. In particular, this MIME type does not support the charset parameter.

At the moment I'm struggling to integrate a piece of software against such server, while trying to use Spring Securities SpringReactiveOpaqueTokenIntrospector, as there is currently no way to override the default behavior.

Though I'll use an entire custom implementation, I'd like to propose supporting that adding the charset can be turned off. I've already created a patch and tests for the FormHttpMessageWriter and BodyInserters which uses Hints to control omitting the charset and would like to contribute if this proposal is accepted.

Comment From: rstoyanchev

Looking at the code I think both FormHttpMessageConverter and FormHttpMessageWriter we are combining the content type and charset determination into one call that returns a MediaType. We use the charset from that MediaType and also write it as as a header, and as-is, to the output message. We could try and refine that so if the original MediaType does not have a charset or is not provided, and we're using the default UTF-8 encoding, we don't include a charset parameter.

Comment From: wallind

My team is affected by this, and ended up writing a custom implementation to work around it (some parts redacted because IP).

In our case it was the Dun & Bradstreet API POST /v3/token being used in conjunction with the Spring OAuth2 client library (org.springframework.security:spring-security-oauth2-client) that hit the error when the charset=UTF-8 was included.

We originally thought it was something specific to the OAuth2 client library, but it was not. As mentioned above it's the FormHttpMessageWriter that is used for the URL encoded body that introduces the condition which causes us the issue.

Hopefully this custom implementation helps someone workaround the problem in the meantime:

package com.redacted

import org.springframework.http.MediaType
import org.springframework.http.codec.FormHttpMessageWriter

/**
 * Wrapper around [FormHttpMessageWriter] that removes the "else-if" logic that would append the default charset to the
 * Content-Type header if the charset was not specified.
 *
 * Note that this class only needs to override `getMediaType`, not `write`, because [FormHttpMessageWriter.write] calls
 * `getMediaType` to update the `message.getHeaders()` before writing the body. [FormHttpMessageWriter.write] then uses
 * the charset for the actual encoding of the body as well, but that is fine since the only thing we need to
 * prevent is anything but `application/x-www-form-urlencoded` from being added to the Content-Type header.
 *
 * This is needed by some APIs (like Dun & Bradstreet) that do not support parameters like charset being specified in
 * the Content-Type header.
 */
class NoCharsetRequiredInContentTypeFormHttpMessageWriter : FormHttpMessageWriter() {

  /**
   * Override the default implementation of [FormHttpMessageWriter.getMediaType] to not add the default charset to the
   * Content-Type header if the charset was not specified.
   *
   * As of writing this, the default implementation of [FormHttpMessageWriter.getMediaType] looks like this:
   * ```
   *    protected MediaType getMediaType(@Nullable MediaType mediaType) {
   *        if (mediaType == null) {
   *            return DEFAULT_FORM_DATA_MEDIA_TYPE;
   *        }
   *        else if (mediaType.getCharset() == null) {
   *            return new MediaType(mediaType, getDefaultCharset());
   *        }
   *        else {
   *            return mediaType;
   *        }
   *    }
   * ```
   *
   * Our custom implementation here effectively removes the middle "else-if". It also sets the method to public for
   * easier testing.
   *
   * @see FormHttpMessageWriter.getMediaType
   */
  public override fun getMediaType(mediaType: MediaType?): MediaType {
    return mediaType ?: MediaType(MediaType.APPLICATION_FORM_URLENCODED, DEFAULT_CHARSET)
  }
}

then we plug it into the relevant WebClient like so:

WebClient.builder()
      .codecs {
        it.customCodecs().register(
          MultipartHttpMessageWriter(
            /**
             * Since the MultipartHttpMessageWriter only uses the `partWriters` list when the data is multipart
             * (see [MultipartHttpMessageWriter.write]), and the Dun & Bradstreet auth API only uses form data
             * (i.e. `application/x-www-urlencoded`), we can just pass an empty list here.
             */
            listOf(),
            NoCharsetRequiredInContentTypeFormHttpMessageWriter(),
          ),
        )
      }
      .build()

Having the option to disable this behavior would have resolved our issue, preventing the need for the custom implementation.

Comment From: ffray

I like the custom codec solution way better than my code. Didn't notice this simple approach, which comes quite handy!

Comment From: rstoyanchev

This was also reported in #22588 and it led to the creation of the getMediaType method. Considering the lack of clear guidance in the spec, and that some servers don't handle it well, we should make it easier to submit without the charset parameter.

Comment From: rstoyanchev

I've given this a try, not adding a charset parameter if it is the default UTF-8. However, it is a change and it will break tests if not more, so I'm going to defer it until 6.2. The changes are in #31781, and I'm closing this as superseded by it.