Hello,

i am using htmlunit with spring test in order to test all ihm interface from my web application. It works fine with html form (post and get) and with ajax get but i have a problem with ajax post request.

The controller don't received the request. If i replace post by get the junit test case works fine.

this the html view

<!DOCTYPE html>
<html lang="fr" xmlns="http://www.w3.org/1999/xhtml" 
    xmlns:th="http://www.thymeleaf.org">
    <head>
        <title>Spring Html Unit</title>
        <meta charset="utf-8" />
        <meta name="format-detection" content="telephone=no">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <script src="https://code.jquery.com/jquery-3.5.1.min.js" integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin="anonymous"></script>
        <script>
            function postTest() {
                $.ajax({
                    type:"post",
                    data: {
                        'subject' : 'subject test',
                        'message' : 'message test'
                    },
                    url:"[[@{/test/post}]]",
                    success: function(data, textStatus, jqXHR){
                        $("#result").text(data);
                    }
                });
            }       
        </script>
    </head>
    <body>
        <h4 th:text="'Hello to Thymeleaf '+${myParam}">Hello from Thymeleaf</h4>

        <button type="button" onClick="postTest()">Test post</button>
        <div>
            <span>result :</span><span id="result"></span>
        </div>

    </body>
</html>

and the controller

@Controller
public class WelcomeController {

    @GetMapping("/")
    public String init(Model model) {
        model.addAttribute("myParam", "Guillaume");
        return "welcome";
    }

    @PostMapping("/test/post")
    public @ResponseBody String post(@RequestParam String subject, @RequestParam String message, Model model) {
        return subject + " " + message;
    }
}

you can also find the complete code on my github https://github.com/guisimon28/spring-test-htmlunit

Can you help me to find if there is some missing configuration or if its a htmlunig bug or pull request ?

Thanks for All

Guillaume

Comment From: bclozel

Thanks for getting in touch, but it feels like this is a question that would be better suited to StackOverflow. As mentioned in the guidelines for contributing, we prefer to use GitHub issues only for bugs and enhancements. Feel free to update this issue with a link to the re-posted question (so that other people can find it) or add some more details if you feel this is a genuine bug.

The sample you've provided doesn't use Spring Boot - I'd suggest to use Spring Boot instead to dramatically reduce the amount of boilerplate.

Also there seems to be a confusion around your usage of jQuery. Right now it's sending the form as a POST, but with an urlencoded request body. You should either switch to GET or send the body as "multipart/form-data" and use a JavaScript FormData object.

Even better, you could look into the getting started guides for Spring Boot. See https://spring.io/guides/gs/handling-form-submission/

Thanks!

Comment From: guisimon28

Hi bclozel,

i thinked it was a bug because my code works fine in a tomcat server. This code was an example to reproduce the probem i have with my real application. With my real application jquery post works fine when my application is deployed in tomcat but not with MockMVc

it works great on chrome. I put the application on debug in eclipse with a tomcat server. And when i click on Test post button the result label is refresh with the content

That the reason why i think there is a difference between a real browser and the MockMvc b So i am not agree, maybe you can try in a browser and with MockMvc to see the difference.

regards

Comment From: guisimon28

Hello,

after some investigations, we think there is a problem in HtmlUnitRequestBuilder.content() as you can see in this analyse https://github.com/HtmlUnit/htmlunit/issues/223#issuecomment-695139395

mor info on this github to reproduce : https://github.com/twendelmuth/spring-test-htmlunit

can you reopen the ticket in order to correct spring test ?

regards

Guillaume

Comment From: rstoyanchev

after some investigations, we think there is a problem in HtmlUnitRequestBuilder.content()

If I understand correctly, the issue is that we have a POST with "application/x-www-form-urlencoded" data, but the form data is not parsed and used to populate request parameters as Servlet containers do. Wouldn't it make more sense for HtmlUnit to do that so that anything using it (besides Spring's HtmlUnit integration) can also benefit from this improvement?

Comment From: rstoyanchev

Or I guess in HtmlUnit requestParameters actually means URL query parameters rather than what the Servlet API calles "request pameters", in which case I to think it make sense for HtmlUnitRequestBuilder to do that.

Note that we already have similar logic when using MockMvc (without HtmlUnit).

Comment From: rstoyanchev

It works fine with html form (post and get) and with ajax get but i have a problem with ajax post request.

@guisimon28 I'm wondering how does it work with HTML form post? I would expect that to run into the same issue. Is the server side controller in that case different?

Comment From: guisimon28

thanks for analyse my issue rstoyanchev, i can't debug today. But the same test with html form post works fine. All works fine with Htmlunit when i put my application on a server and start htmunit test on a remote server.

regards

Comment From: guisimon28

Still dont have time to debug do you think you can improve htmlunitrequestbuilder like mokckmvc ?

Comment From: fredarene

Hi @rstoyanchev ,

I work with @guisimon28.

When we debug the passage in HtmlUnitRequestBuilder.buildRequestfor both cases, we see that :

  • html FORM POST : the webRequest.requestParameters_ is populated with the form's params; they're then put in MockHttpServletRequest.parameters and used "correctly"

  • AJAX POST : the webRequest.requestBody_ is populated with the form's params and dumped in MockHttpServletRequest.content, so it seems the params are not user afterwards, at least not as parameters of the request

Comment From: rstoyanchev

Thanks @fredarene for the extra findings. It sounds like HtmlUnit is populating WebRequest.requestParameters in one case but not the other. I would think that there is a case to argue it should do the same in both cases.

Any chance you can update the sample project with both scenarios?

Comment From: twendelmuth

Hi,

quoting my answer here from the HTMLUnit side. I've transformed the Spring Boot project to illustrate the issues as well

Okay I finally found some time to look into this. You're indeed correct that the error seems to stem from the MockMvcWebClientBuilder from Spring. To be honest I'm not to familiar with the MockMvcWebClientBuilder. The problem seems to be what we're seeing here: The FormData / RequestBody is not correctly put in to the @RequestParam parameters in the Controller. Therefore the request fails with 400 Bad Request. I'm having some difficulties debugging the Spring test code ... however I think the issue comes from HtmlUnitRequestBuilder.content():

This one takes the request body without any escaping and passes it to the controller:

private void content(MockHttpServletRequest request, Charset charset) { String requestBody = this.webRequest.getRequestBody(); if (requestBody == null) { return; } request.setContent(requestBody.getBytes(charset)); //requestBody = 'subject test message test' //requestBody should be: 'subject=subject+test&message=message+test' since it's form encoded. }

To get to the results I've transformed the project to use SpringBoot since it was easier for me to find out what's actually happening and where the error potentially is. I hope that it's running with gradle ... since I needed to convert it to a maven project because of my gradle issue 👼

The project is here: https://github.com/twendelmuth/spring-test-htmlunit Added tests that should illustrate what's wrong: https://github.com/twendelmuth/spring-test-htmlunit/tree/master/src/test/java/spring/test/htmlunit/controller And more specific: https://github.com/twendelmuth/spring-test-htmlunit/blob/master/src/test/java/spring/test/htmlunit/controller/WelcomeControllerMockMvc.java#L37 - this is what's happening in the end in your test.

I hope this helps :-)

Comment From: rstoyanchev

@twendelmuth I wasn't looking for the example with "AJAX POST" but rather for the example with "html FORM POST" in order to understand why they work differently even though they are the same type of request. In any case I investigated and updated the HtmlUnit issue. I'll wait for the response there.

Comment From: guisimon28

@rstoyanchev

in your post on 24/09/2020, you say "Note that we already have similar logic when using MockMvc (without HtmlUnit)."

can you apply the same logic with HtmlUnit ?

regards

Comment From: rstoyanchev

MockMvc simulates a client and builds the request from the ground up. By contrast HtmlUnit is a client and it is a legitimate question what HtmlUnit should do in this case. From the comment in https://github.com/HtmlUnit/htmlunit/issues/223#issuecomment-706703016 it's clear the main question there is what does a real browser do? Therefore the best way to move this forward is to provide the answers to the questions there first so we can figure out where the fix belongs. I have not gotten to it yet.

Comment From: bclozel

@guisimon28 please take a look at https://github.com/HtmlUnit/htmlunit/issues/223#issuecomment-714391359

Comment From: bclozel

This issue is superseded, since this seems to be an inconsistency in HtmlUnit's behavior (see issue linked in previous comment). I'm closing this issue for now, we can reopen it if work is required in the Spring Framework codebase.