Spring Security Should return www-authenticate even for "X-Requested-With: XMLHttpRequest" requests

Describe the bug Since migrating to Spring Security 6, Calling APIs using simple jQuery/XHR with basic auth results in final 401 errors, despite being logged in through the browser basic auth dialog.

Analyzing the responses, they are missing the mandatory WWW-Authenticate header. Thus the browser will not attempt the (already present) basic auth credentials.

IMHO this is a bug, as that header is mandatory: https://datatracker.ietf.org/doc/html/rfc9110#name-www-authenticate

To Reproduce Implement a simple app with

    public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        return http
                .csrf(AbstractHttpConfigurer::disable)
                .cors(cors -> cors.disable())
                .authorizeHttpRequests(
                        matchers -> matchers
                                .requestMatchers("/some/**", "/some/more/**", "/error/**")
                                .permitAll()
                                .requestMatchers("/api/**", "/app/**")
                                .authenticated()
                                .anyRequest().authenticated())

                .sessionManagement(session -> session.sessionCreationPolicy(SessionCreationPolicy.STATELESS))
                .httpBasic(basic -> basic.realmName("my-realm"))
                .build();
    }
        // a valid AuthenticationProvider, too...

Expected behavior have a (valid) www-authenticate response header in all 401 responses.

Additional Info

The behaviour was first implemented here: https://github.com/spring-projects/spring-security/commit/4ef0460ef66b4417efc7999d17ebd4cd3ebce3a6#diff-0f2a9f7a8a020191e00efb336582d7d71dd46130bc4b5cbc86eba681c498751fR92

Current state: https://github.com/spring-projects/spring-security/blob/6e495b8ba9b3e5ef397fc6852ab0bc7737ec38b9/config/src/main/java/org/springframework/security/config/annotation/web/configurers/HttpBasicConfigurer.java#L106

I guess this line of code just does not explicitly put the www-authenticate header.

Workaround

My workaround is to provide a simple BasicAuthenticationEntryPoint without that special treatment for XMLHttpRequest:

        BasicAuthenticationEntryPoint basicAuthEntryPoint = new BasicAuthenticationEntryPoint();
        basicAuthEntryPoint.setRealmName(BASIC_AUTH_REALM);

and setting it in the SecurityFilterChain

                .httpBasic(basic -> basic.realmName(BASIC_AUTH_REALM).authenticationEntryPoint(basicAuthEntryPoint))

Comment From: MartinEmrich

To be precise: I do not imply it is a regression from Spring Security 5, most probaby my old WebSecurityConfigurerAdapter code did just not trigger that piece of code.

Comment From: sjohnr

@MartinEmrich thanks for reaching out!

Since migrating to Spring Security 6, Calling APIs using simple jQuery/XHR with basic auth results in final 401 errors, despite being logged in through the browser basic auth dialog.

I am unable to see a change in behavior switching between Spring Security 5 and 6. Regarding...

I do not imply it is a regression from Spring Security 5, most probaby my old WebSecurityConfigurerAdapter code did just not trigger that piece of code.

Can you please provide a minimal, reproducible sample that somehow demonstrates the change in behavior between 5 and 6? Otherwise, I'm not quite sure I'm following how this has to do with migrating?

The behaviour was first implemented here: 4ef0460

This behavior seems to go back to 2013 as pointed out by the referenced commit you provided. If necessary, we could ask for clarification from the author but I think it's fairly clear from the commit message that it was an effort to define a clear set of defaults for Javascript clients. It's also clear that the behavior to not provide a WWW-Authenticate header by default when using XHR is intentional. Further, you can customize this behavior easily as demonstrated in your workaround.

IMHO this is a bug, as that header is mandatory: https://datatracker.ietf.org/doc/html/rfc9110#name-www-authenticate

I can see how this default does not align with your preference, and I also understand there could be arguments made for changing the behavior to simply align with the spec in all cases. The defaults provided by Spring Security are quite generic and are intended as "one size fits most" on a best-effort basis. However, Spring Security is designed to be customized when the defaults do not suit preference or requirements. I feel that customizing the behavior is realistic for your use case given the length of time this default has been in place in the framework.

I plan to close this issue with the above discussion points but I will wait for your response to see if I have missed something in the consideration since I was not able to reproduce the change in behavior you described.

Comment From: MartinEmrich

Hello @sjohnr

As stated, I do not claim that the behaviour has changed between Spring Security 5 or 6. So I cannot produce a test case showing such a difference. I just presented my train of thought leading to my discovery. If I mislead you, I apologizes.

I rather argue that the behaviour was violating an RFC all along.

I understand that the intent of this code is to prevent any human-interaction responses to be sent to Javascript-initiated requests, which clearly makes sense (like an overly long HTML error page). And on that point I agree, sending a 401 without a body is very efficient.

Thus I might even suggest that the missing www-authenticate header was not intentional for the original author; it just so happened, as that short-circuit 401 response is processed before the code that produces the (correct, including realm) www-authenticate header.

Doing research (meaning "using Google"), I can find lots of web developers complaining about the www-authenticate header provided by other web servers, but also developers complaining abouth the missing www-authenticate header from servers implementing a similar behaviour as Spring Security.

In my opinion, the RFC is very clear on this, so IMHO the default should be sending the www-authenticate header with every 401 (Though Spring Security might provide a simple switch like a .disableWwwAuthenticateOn401() method) to cater for less-careful frontend developers.

Comment From: sjohnr

IMHO the default should be sending the www-authenticate header with every 401 (Though Spring Security might provide a simple switch like a .disableWwwAuthenticateOn401() method) to cater for less-careful frontend developers.

Thanks for the update @MartinEmrich. Based on your response and the fact that it's been a number of years since this default was applied, it might be time to revisit.

However, we usually wait for upvotes on issues before making a change like this because we want to be going where the community needs us to. It's clear that the spec intends HTTP Basic responses to include a WWW-Authenticate so I would say many upvotes (and the absence of other comments on this issue) would imply developers agree and we might want to look at a change. If other folks have suggestions such as convenience methods to switch the behavior (I don't feel this is necessary personally given how easy it is) they can comment as such. We can leave this issue open and see what happens. Sound good?

Comment From: MartinEmrich

That sounds reasonable, changing a behaviour present for 10+ years might upset people just as much as not fixing it.

But maybe the documentation can include it, so it does not surprise people. I read this section, and then searched for the cause of my issue (with my frontend, which just uses Jquery, explicitly relying on the browser layer basic authentication). It was erratic, depending on browser vendor, OS, cookie/session state, privat browsing mode, etc). Only after digging into the Spring Security source code, and bisecting the requests my browser made, I put the pieces together.

From https://docs.spring.io/spring-security/reference/servlet/authentication/passwords/basic.html

Since the user is not authenticated, ExceptionTranslationFilter initiates Start Authentication. The configured AuthenticationEntryPoint is an instance of BasicAuthenticationEntryPoint, which sends a WWW-Authenticate header. The RequestCache is typically a NullRequestCache that does not save the request since the client is capable of replaying the requests it originally requested.

I would happily provide a few sentences, but TBH even after years I do not feel as a confident Spring Boot developer knowing all the tech terms.

Maybe at that section something like

The default HTTP Basic Auth Provider will suppress both Response body and WWW-Authenticate header in the response, when the request was made with a X-Requested-By: XMLHttpRequest reader. To override, implement your own BasicAuthenticationEntryPoint like this:

As I have a working fix now, I won't nag you further.

Comment From: sjohnr

@MartinEmrich, I have an update on this issue. I spoke to @rwinch about this, and he informed me that in fact the behavior for X-Requested-With: XMLHttpRequest to only send a 401 Unauthorized response is intentional, as I mentioned above. This is due to the fact that sending the WWW-Authenticate response header is the primary way that browsers are informed to display a basic authentication dialog and javascript applications would typically want to display their own login page or dialog.

However, it appears that browsers may behave differently in different scenarios. Since it may not be uniform across all browsers, the default provided by Spring Security is currently a best guess as to the correct response and may need to be customized. We would not look to provide an updated default for this scenario at this point. With that in mind, I would like to close this issue.

Before doing so, I have looked more closely at the scenario you outline which is:

Calling APIs using simple jQuery/XHR with basic auth results in final 401 errors, despite being logged in through the browser basic auth dialog.

For at least two browsers on my machine, I am not able to reproduce this behavior. The browser will always send the basic auth credentials once I've entered them. The basic auth dialog is still displayed when I'm not logged in and I request an authenticated endpoint directly in the browser. So I don't see an issue like you have reported. Here is a simple application that I'm using:

@SpringBootApplication
public class Application {

    /**
     * Main entrypoint for the application.
     */
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }

}

@Configuration
@EnableWebSecurity
public class SecurityConfiguration {

    @Bean
    public SecurityFilterChain securityFilterChain(HttpSecurity http) throws Exception {
        http
            .authorizeHttpRequests((authorize) -> authorize
                .requestMatchers("/", "/index.html", "/favicon.ico").permitAll()
                .anyRequest().authenticated()
            )
            .httpBasic(Customizer.withDefaults());

        return http.build();
    }

    @Bean
    public UserDetailsService userDetailsService() {
        UserDetails userDetails = User.builder()
            .username("user")
            .password("{noop}password")
            .roles("USER")
            .build();

        return new InMemoryUserDetailsManager(userDetails);
    }

}

@Controller
public class HomeController {
    @GetMapping("/")
    public String home() {
        return "redirect:/index.html";
    }
}

@RestController
public class HelloController {

    @GetMapping(value = "/hello", produces = MediaType.APPLICATION_JSON_VALUE)
    public Map<String, String> hello() {
        return Map.of("message", "Hello, world!");
    }

}

<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Home</title>
</head>
<body>

<script>
    function onLoad() {
        console.log(this.responseText);
    }

    const req = new XMLHttpRequest();
    req.addEventListener("load", onLoad);
    req.open("GET", "http://localhost:8080/hello");
    req.setRequestHeader("X-Requested-With", "XMLHttpRequest");
    req.send();
</script>
</body>
</html>

I'm going to close this issue, but if you can help me update the above sample to reproduce the issue, we can reopen if necessary.

Regarding documentation:

Maybe at that section something like

The default HTTP Basic Auth Provider will suppress both Response body and WWW-Authenticate header in the response, when the request was made with a X-Requested-By: XMLHttpRequest reader. To override, implement your own BasicAuthenticationEntryPoint like this:

If you feel this would have been helpful for you, I think that would be a nice note to add. Would you like to open a PR to add this and an example? (feel free to respond even though the issue is closed)

Comment From: MartinEmrich

No objections here.

Indeed there were several strange things going on. One was that I bookmarked literally http://admin:admin@localhost:8080/, and instead of translating the basic auth credentials to the Authorization: basic xxxx header, Chrome sent that URI unmodified. It did not show admin/admin in the adress bar (obviously), and not even with copy/paste. Thus the "regular" basic auth credentials were empty, as nothing triggered the input dialog during loading any static content. Then the XHR requests did get the "first" 401.

After using a bookmark without credentials, the first static content requests triggered the dialog, and from then on the credentials were also sent with all XHR requests.

If I can understand how spring documentation works, I might cook up a PR later.