Summary

The Integration tests I'm trying to run to check if my HttpSecurity access definitions (https://github.com/jonasbark/spring-mvc-unit-test-problem/blob/master/src/main/kotlin/pass/users/api/spec/UsersAPIKeycloakSecurity.kt) work as they should do not complete when every unit test is run (test gradle task). However, they run just fine when - I run them on their own (single test) - I add @DirtiesContext(classMode = ClassMode.BEFORE_EACH_TEST_METHOD) annotation on TenantTenantAccessTest

The request from the failing integration tests are supposed to throw a 403 error because they do not have correct authorities. However they complete with HTTP 200. This only seems to happen for PUT / DELETE requests.

I suspect some kind of caching / optimization?

I tried several workarounds, such as delay every integration test, clear SecurityContext manually and such. What I also noticed is that the Authentication object of the SecurityContextHolder contains the correct authorities at any time in the unit tests - also when the test itself fails.

Actual Behavior

Integration Tests are reproducibly flaky - they succeed when run on their own.

Expected Behavior

All Integration Tests succeed in the first place

Configuration

  • Spring Boot with MVC Servlet environment
  • functional router definitions
  • MockMvc for integration tests

Version

id("org.springframework.boot") version "2.2.6.RELEASE"
id("io.spring.dependency-management") version "1.0.8.RELEASE"
testImplementation("org.springframework.boot:spring-boot-starter-test:2.2.6.RELEASE")
testImplementation("org.springframework.security:spring-security-test:5.2.3.RELEASE")

https://github.com/jonasbark/spring-mvc-unit-test-problem/blob/master/build.gradle.kts

I also tried the latest Spring Boot milestone 2.3.0.M4

Sample

Sample repository: https://github.com/jonasbark/spring-mvc-unit-test-problem Reproducible by running ./gradlew test.

Comment From: rwinch

The reason it is flaky is that the test changes the inputs. Every test method a new testDbEntity is being created and the id is being generated. So if you run only a single test, the id is 1. For each test it increments.

That means that if more than 9 tests have ran, at pass client admin forbidden on remove 5e79d316c1584

    @Test
    @WithMockUser(authorities = ["pass.user", "pass.client.admin"])
    fun `pass client admin forbidden on remove 5e79d316c1584`() {
        val tenantId: Long = testDbEntity.id!!
        mockMvc.delete("/tenants/{tenantId}/", tenantId) {
            accept(MediaType.APPLICATION_JSON)
            contentType = MediaType.APPLICATION_JSON

        }.andExpect {
            status { isForbidden }
        }
    }

there are two digits being passed into the URL. The security expression "/tenants/{tenantId:[\\d+]}/" that is configured matches with a single digit id, but not multiple.

Comment From: jonasbark

Oh No, so many hours wasted on something that simple in the end. Sorry for bugging you and thanks a ton for investigating, @rwinch!

Comment From: rwinch

Just glad you have an answer now :smile: