RestTemplate does some substitutions on URLs passed to exchange(...).

I just spent an hour trying to understand why my GET request worked with cURL but refused to work with RestTemplate.... It turned out that RestTemplate modified the query parameters in my URL.

Could you please document this behavior somewhere? I was not able to find any documentation about the syntax of templates at https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/client/RestTemplate.html#exchange-org.springframework.http.RequestEntity-java.lang.Class-

Comment From: sbrannen

You are correct that it is not well documented in the Javadoc for RestTemplate, but support for URI templates is discussed in the reference manual.

Comment From: sryze

It looks like I ran into the same issue as the author of this post:

https://stackoverflow.com/questions/60835309/how-to-avoid-double-encoding-of-when-using-spring-resttemplate

When passing a parameter name like param[123] to UriComponentsBuilder.queryParam() the [ and ] characters and encoded twice.

Comment From: rstoyanchev

Can you provide a representative snippet of code, independent of StackOverflow, that shows a more complete example of what leads to double encoding?

Comment From: sryze

@rstoyanchev

Here you go:

package com.example.demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpMethod;
import org.springframework.http.ResponseEntity;
import org.springframework.web.client.RestTemplate;
import org.springframework.web.util.UriComponentsBuilder;

import java.util.Collections;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);

        String url = UriComponentsBuilder.fromHttpUrl("https://postman-echo.com/get")
                .queryParam("params[123]", 123)
                // .build(false)
                .toUriString();
        System.out.println(url);

        RestTemplate restTemplate = new RestTemplate();
        ResponseEntity<String> response = restTemplate.exchange(
                url,
                HttpMethod.GET,
                new HttpEntity<>(Collections.emptyMap()),
                String.class);
        System.out.println(response.getBody());
    }
}

demo.zip

Output:

https://postman-echo.com/get?params%5B123%5D=123
{"args":{"params%5B123%5D":"123"},"headers":{"x-forwarded-proto":"https","x-forwarded-port":"443","host":"postman-echo.com","x-amzn-trace-id":"Root=1-619d33d3-78119a69172eb7214fe55ee5","accept":"text/plain, application/json, application/*+json, */*","content-type":"application/json","user-agent":"Java/1.8.0_301"},"url":"https://postman-echo.com/get?params%255B123%255D=123"}

params%255B123%255D=123 is the result of double-encoding params[123]=123;

If you uncomment the line .build(false), the parameter is encoded once, as expected.

Comment From: poutsma

The underlying problem is that a String is not suitable to express a URL. RestTemplate does not know if, when given a String URL, it should be expanded or not. That is why the RestTemplate offers overloaded variants for each method: one takes a String and object array, one a String and a Map, and one a java.net.URI. The string variants expand variables and are encoded; the URI variant is not modified. This is explained in the reference documentation.

So, the code would have to changed to something like:

        URI url = UriComponentsBuilder.fromHttpUrl("https://postman-echo.com/get")
                .queryParam("params[123]", 123)
                .build();

        RestTemplate restTemplate = new RestTemplate();
        ResponseEntity<String> response = restTemplate.exchange(
                url,
                HttpMethod.GET,
                new HttpEntity<>(Collections.emptyMap()),
                String.class);

This behavior is also documented in the Javadoc of the String variants, both in the RestTemplate::exchange(String, HttpMethod, HttpEntity, Class) (as used in the sample), as well as in the documentation of RequestEntity methods that take a String (as mentioned in the first comment).

Comment From: sryze

Makes sense.

Would it be possible to make RestTemplate encode only unencoded parameters?

Something like:

URI url = new URI(urlString);
for each param in url {
    if (decode(param) != param) {
        // it's encoded, do nothing
    } else {
        // encode it
    }
}

Comment From: poutsma

Would it be possible to make RestTemplate encode only unencoded parameters?

Unfortunately, that is not possible. From Spring Framework's perspective, there is no way to determine whether a given string has been encoded or not. Is that % the start of an encoded sequence, or is it just a percent that happens to be followed by a hex string? You cannot know for sure. And then there are the cases where users actually want to encode an already encoded string, because the web service they use requires it.

URIs really are a lot more complicated than they appear at first glance. See also The great thing about URL encodings is that there are so many to choose from.

Comment From: rstoyanchev

To summarize, when you use UriComponentsBuilder, you can either build a (fully encoded) URI and pass that into the RestTemplate or use UriComponnets#toString to obtain a simple concatenated String, and pass that into the RestTEmplate which always encodes String based URLs.

So this is expected behavior and documented, but if you have ideas for where you would have liked to see something more in the Javadoc or reference doc, please feel free to suggest it.