Johnny Lim opened SPR-14828 and commented
'+' is valid for a space but with the following code:
String httpUrl = "http://localhost:8080/test/print?value=%EA%B0%80+%EB%82%98";
URI uri = UriComponentsBuilder.fromHttpUrl(httpUrl).build(true).toUri();
System.out.println(uri);
I got the following error:
java.lang.IllegalArgumentException: Invalid character '+' for QUERY_PARAM in "%EA%B0%80+%EB%82%98"
at org.springframework.web.util.HierarchicalUriComponents.verifyUriComponent(HierarchicalUriComponents.java:313)
at org.springframework.web.util.HierarchicalUriComponents.verify(HierarchicalUriComponents.java:281)
at org.springframework.web.util.HierarchicalUriComponents.<init>(HierarchicalUriComponents.java:90)
at org.springframework.web.util.UriComponentsBuilder.build(UriComponentsBuilder.java:336)
at learningtest.org.springframework.web.UriComponentBuilderTests.testFromHttpUrlBuildEncoded(UriComponentBuilderTests.java:19)
Affects: 4.3.3
Reference URL: https://github.com/izeye/spring-boot-throwaway-branches/blob/rest-and-logback-access/src/test/java/learningtest/org/springframework/web/UriComponentBuilderTests.java
Issue Links: - #20551 UriComponentsBuilder inconsistent encode/decode query params behavior ("is duplicated by") - #14805 UriComponents.Type.QUERY_PARAM does not match spec ("is duplicated by") - #21259 UriComponentsBuilder does not encode "+" properly - #22006 Not encoding '+' in URLs anymore breaks backwards compatibility with apps running on spring 4 - #20750 Encoding of URI Variables on RestTemplate - #21473 UriUtils query param "safe" encoding mode (%-encode sub-delims)
Referenced from: commits https://github.com/spring-projects/spring-framework/commit/f2e293aadf98cb64ba7ef3044b59f469efd21503
1 votes, 6 watchers
Comment From: spring-projects-issues
Rossen Stoyanchev commented
According to the URI spec RFC 3986, a "+" is allowed in the query (i.e. pchar > sub-delimiters > ";"). This matches our own HierarchicalUriComponents.QUERY except QUERY_PARAM further excludes "=", "&", and "+". The "=", "&" make sense since those are commonly used as query parameter delimiters but I'm not sure where the "+" comes from.
Arjen Poutsma what's your take on this? Is the "+" encoded as an extra precaution against it being decoded and interpreted as a space? I found an old discussion on this here. Or could it be based on the older version of the spec RFC 2396 which does seem to call out "+" as a reserved character in the query?
Comment From: spring-projects-issues
Arjen Poutsma commented
It's been a while since I wrote that code, so I can't remember what was the reason for excluding the '+'. My guess would be the same as yours: it was based on an older version of the RFC.
Comment From: spring-projects-issues
Alexandre Navarro commented
I have exactly the same problem. Interested to fix it in a future version.
Comment From: spring-projects-issues
Kazuki Shimizu commented
Hi,
Probably by this changes, + does't encode to %2B .
Is this behavior working as design?
In this case, I think it should encode to the %2B because there is a case that + is decoded as space by the application server.
What do you think?
@Test
public void test() {
// http://localhost:8080/?q=+
System.out.println(UriComponentsBuilder.fromHttpUrl(
"http://localhost:8080/").queryParam("q", "+").build().encode()
.toUri());
// http://localhost:8080/?q=%20
System.out.println(UriComponentsBuilder.fromHttpUrl(
"http://localhost:8080/").queryParam("q", " ").build().encode()
.toUri());
// http://localhost:8080/?q=%26
System.out.println(UriComponentsBuilder.fromHttpUrl(
"http://localhost:8080/").queryParam("q", "&").build().encode()
.toUri());
// http://localhost:8080/?q=%3D
System.out.println(UriComponentsBuilder.fromHttpUrl(
"http://localhost:8080/").queryParam("q", "=").build().encode()
.toUri());
}
Comment From: spring-projects-issues
Rossen Stoyanchev commented
This is by design indeed. Only reserved characters are reserved but "+" is allowed. Is this through the RestTemplate? If yes take a look at the uriTemplateHandler and the DefaultUriTemplateHandler implementation (DefaultUriBuilderFactory in 5.0). It has a strict encoding property.
Comment From: spring-projects-issues
Kazuki Shimizu commented
Thanks for quick answer!
RestTemplate ?
No, I don't use the RestTemplate in my application.
I was making a query string for pagination link using the UriComponentsBuilder like above example.
There is a case that includes 0x2b(+) *1 in a pagination link because it is a value that input by an end user.
Comment From: spring-projects-issues
Kazuki Shimizu commented
Oh... Sorry, I click the "Add" button by mistake.
I understood that need to change to use an another APIs. Thanks.
Comment From: spring-projects-issues
Dmitry Katsubo commented
Rossen Stoyanchev: I've tried to locate "strict" mode in DefaultUriBuilderFactory, but it has only URI_COMPONENT in enum EncodingMode which is the default. So this one does not encode (escape) + in path and query:
assertEquals("http://localhost:8080/test+/print?value=%25EA%25B0%2580+%25EB%2582%2598", new DefaultUriBuilderFactory().uriString("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98").build("test+").toString());
DefaultUriTemplateHandler encodes only path, but not query:
DefaultUriTemplateHandler defaultUriTemplateHandler = new DefaultUriTemplateHandler();
defaultUriTemplateHandler.setStrictEncoding(true);
assertEquals("http://localhost:8080/test%2B/print?value=%EA%B0%80+%EB%82%98", defaultUriTemplateHandler.expand("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98", "test+").toString());
Now I have difficulty in how to escape query part. Actually in my case I start from unencoded URL string, hence all characters that have special meaning should be encoded. How to do it?
assertEquals("http://localhost:8080/test+/print?value=v1%20v2%2Bv3", UriComponentsBuilder.fromHttpUrl("http://localhost:8080/{path}/print?value={val}").buildAndExpand("test+", "v1 v2+v3").encode().toUri().toString());
Comment From: spring-projects-issues
Rossen Stoyanchev commented
Dmitry Katsubo, the EncodingMode is an enum with 3 values. You need this:
DefaultUriBuilderFactory factory = new DefaultUriBuilderFactory();
factory.setEncodingMode(EncodingMode.VALUES_ONLY);
URI uri = factory.uriString("http://localhost:8080/{path}/print?value=%EA%B0%80+%EB%82%98").build("test+");
Note that strict encoding is only applied to expanded URI variable values (and the same is true for DefaultUriTemplateHandler). This is why your query is not getting encoded. See the Javadoc for details.
Kazuki Shimizu the above DefaultUriBuilderFactory internally uses UriComponentsBuilder and essentially does this:
List<String> vars = UriUtils.encodeUriVariables("+");
System.out.println(UriComponentsBuilder.fromHttpUrl(
"http://localhost:8080/?").queryParam("q", "{value}").build().expand(vars)
.toUri());
Comment From: spring-projects-issues
Dmitry Katsubo commented
OK, here is an example which I expect to succeeded because I set query parameter as unencoded value with intent that + should be passed literally to remote server hence should be encoded:
DefaultUriBuilderFactory factory = new DefaultUriBuilderFactory();
factory.setEncodingMode(EncodingMode.VALUES_ONLY);
UriBuilder uriString = factory.uriString("http://localhost/path");
uriString.queryParam("value", "v1+v2");
assertEquals("http://localhost/path?value=v1%2Bv2", uriString.build().toString());
Comment From: spring-projects-issues
Rossen Stoyanchev commented
Did you miss this part "strict encoding is only applied to expanded URI variable values" ?
Comment From: spring-projects-issues
Dmitry Katsubo commented
Is it possible to further extend API so that one can encode URL query parameters in one go? Namely: * Add method ``` UriComponentsBuilder#getQueryParams() { return queryParams; }
* Make methods `UriUtils.encodeUriVariables()` public.
Then it would be possible to encode query params in one go, without extra substitution step.
Comment From: spring-projects-issues
Rossen Stoyanchev commented
I've made UriUtils.encodeUriVariables() public. As for UriComponentsBuilder#getQueryParams(), I don't think we can add just one accessor, there are no others on the builder. Is there a reason you can't call build() first and then access the queryParams?
Comment From: spring-projects-issues
Dmitry Katsubo commented
build() transforms query params into non-mutable state, see HierarchicalUriComponents constructor, line 96. So at the moment it is only possible to build(), then get query params, encode them, put back via UriComponentsBuilder#replaceQueryParams(MultiValueMap<String, String> params) and then build() again...
Another alternative is to encapsulate UriComponentsBuilder the same way as DefaultUriBuilderFactory$DefaultUriBuilder does and do necessary preparations using UriUtils.encodeUriVariables() in corresponding methods (queryParam(), replaceQueryParam(), ...).
Any better idea?
Comment From: spring-projects-issues
Rossen Stoyanchev commented
I see, or you could build a MultiValueMap of query params, use UriUtils to encode, and pass into UriComponentsBuilder#query. The only catch is we need to enhance UriUtils to support MultiValueMaps.
I think adding another encoding mode to DefaultUriBuilderFactory.EncodingMode would be perfectly okay to do. It's exactly what it's meant for -- provide control over encoding strategies. However I am not sure I understand exactly what the new mode is, i.e. aside from fully encoding query params, how does it work for others?
Comment From: spring-projects-issues
Dmitry Katsubo commented
I tried somehow to apply the behaviour I need using the strategies that I've mentioned above, but honestly I've failed. The problem is that once I replace + with %2B, it (correctly) gets double-encoded as %252B when encode() is called. So there are not so many alternatives: either use queryParam("q", "{value}").build().expand(vars) as advised above or do some massaging of URL after it is built.
At the moment I've decided to switch to javax.ws.rs.core.UriBuilder which provides very similar API with minor effect that last one is more strict and %-encodes (among others) , as %2C and / as %2F as well, but that is perfectly OK as receiver should interpret that correctly.
If somebody is interested in code I came up with, here it goes:
public static class UriBuilderFactory extends UriComponentsBuilder {
public static UriComponentsBuilder fromHttpUrl(String httpUrl) {
return new UriBuilderFactory(UriComponentsBuilder.fromHttpUrl(httpUrl));
}
public UriBuilderFactory(UriComponentsBuilder other) {
super(other);
}
@Override
// queryParams(), replaceQueryParam() and replaceQueryParams() should be overridden and values should be processed in similar way
public UriComponentsBuilder queryParam(String name, Object... values) {
return super.queryParam(name, encode(values));
}
// To be replaced by UriUtils.encodeUriVariables()
private static Object[] encode(Object[] values) {
Object[] encodedValues = new Object[values.length];
for (int i = 0; i < values.length; i++) {
if (values[i] != null) {
encodedValues[i] = values[i].toString().replace("+", "%2B");
}
}
return encodedValues;
}
}
// Looks OK however not calling encode() has other side effects like passing through '=' which should be encoded:
assertEquals("http://host.com/path?key=%2Ba,b%2Bc&d=e/", UriBuilderFactory.fromHttpUrl("http://host.com/{var}").queryParam("key", "%2Ba,b+c&d=e/").buildAndExpand("path").toUriString());
// This implicitly calls encode() and hence '+' gets double-encoded:
assertEquals("http://host.com/path?key=%252Ba,b%252Bc%26d%3De/", UriBuilderFactory.fromHttpUrl("http://host.com/{var}").queryParam("key", "%2Ba,b+c&d=e/").build("path").toString());
Comment From: spring-projects-issues
Rossen Stoyanchev commented
Dmitry Katsubo, Kazuki Shimizu I've created #21577 which should provide a solution.
Comment From: hviranicitco
I know this is an old issue, but we are facing the exact same issue.
We are using Spring webclient directly. So something like
WebClient.builder()
.baseUrl("http://some-base-url")
.build()
.get()
.uri {
val uriBuilder = it.path("/some-path")
uriBuilder.queryParam("search", searchText)
uriBuilder.build()
}
.retrieve()
.awaitBody()
The searchText comes from the user input. Now when the search input contains +, we expect that this would be encoded to %2B and send to client like http://some-base-url/some-path?search=%2B, but instead it gets sent as http://some-base-url/some-path?search=+, which would be interpreted as space rather than plus sign.
So the workaround for us is we do something like:
val factory = DefaultUriBuilderFactory("http://some-base-url")
factory.encodingMode = DefaultUriBuilderFactory.EncodingMode.NONE
WebClient.builder()
.uriBuilderFactory(factory)
.build()
.get()
.uri {
val uriBuilder = it.path("/some-path")
uriBuilder.queryParam("search", urlEncode(searchText))
uriBuilder.build()
}
.retrieve()
.awaitBody()
So disable encoding and do the encoding our side. Here, urlEncode is just wrapper around java.net.URLEncoder.encode. This works as expected.
Please let me know if I should open a new issue. I posted here, since this has correct context.
Comment From: halilbaydar
Having same issue. UriComponentsBuilder doesn't encode '+'. So I take responsibility of encoding it myself using URLEncoder and it generates %2B but this time UriComponentsBuilder encodes % to 25 I don't know why?
original query param => 2024-10-22T16:42:38+03:00 and it doesn't encode it but it encodes 2024-10-07T16%3A42%3A38%2B03%3A00. %2B => %252B without reason.
Issue should be re-opened
Comment From: rstoyanchev
@hviranicitco please review the section on encoding in the documentation.
You can apply full encoding through URI variables:
val uriBuilder = it.path("/some-path")
uriBuilder.queryParam("search", "{search}")
uriBuilder.build(searchText)