JUnit5 introduced the support for parallel test execution as described here.

When running parallel tests with the @SpringBootTest annotation there is a bug that is causing the caching of the application context, even when using the annotation @DirtiesContext to force context re-creation. Affected spring-boot versions (the one I tried): 2.5.2 and 2.6.2

I replicated this behavior in the attached Intellij IDEA project.

junit-parallel-integration-test-bug.zip

In the project there are 4 tests that are logging when DummyBean is created/destroyed to keep track of when the application context is re-created. Steps to reproduce the issue:

  1. Launch all the tests in the project. You will see in standard output 4 times the string Destroying DummyBean

  2. Enable concurrent execution by de-commenting the lines in junit-platform.properties (see image) Spring @DirtiesContext does not work when running Spring integration tests in parallel

  3. Launch all the tests again. The tests run in parallel but you will get only one Destroying DummyBean in standard output

  4. Bonus: if you execute only top-level classes in parallel (i.e. set junit.jupiter.execution.parallel.mode.default = same_thread) you will get now 2 logs in standard output --> another clear sign that there is something fishy with parallel execution

Is this known or expected? We were trying to execute integration tests in parallel in order to speed up the performance of test execution that in our case are becoming quite troublesome...

Thanks in advance for any feedback.

Comment From: snicoll

If you need to flag a context as dirty (while it being cached), that sounds incompatible with running your tests in parallel as when one thread takes that decision, another thread might be using the same cached context in another test. That said I haven't found any concrete section of the reference doc about this so I'll move this issue to framework so that it can be taken into consideration, or revised.

Comment From: thedarkdestructor

Hi @snicoll. Were you able to replicate the scenario looking at the attached sources?

I don't need context caching: all the contrary. I want that integration tests run in parallel, each using its own context to achieve full isolation. It is possible to achieve this behavior with vanilla Junit: you just need to "manually setup" your "test context" for each test and then run the test in parallel.

I would like to do the same in spring-boot: each test should create its own context and run in parallel. I expected to obtain this by combining @SpringBootTest and @DirtiesContext(classMode = DirtiesContext.ClassMode.AFTER_EACH_TEST_METHOD). This is not the case, context are re-used as I wrote in the issue.

Let me know if my explanation is clearer now or if you need additional info.

Thanks in advance.

Comment From: snicoll

I don't need context caching: all the contrary.

That's not how things are working, and the reason why this issue has been moved to the Spring Framework issue tracker.

I would like to do the same in spring-boot: each test should create its own context and run in parallel

Unfortunately, that isn't the same thing at all. @SpringBootTest or any test that uses SpringExtension is an "integration test" and benefit from the context caching feature.

Let me know if my explanation is clearer now or if you need additional info.

I understood you from the get-go. If you want to trick the cache, you could add a fake property with a different value for each test. This would turn each test into a different key and not use the context cache at all. There might be other options and, perhaps, things that need clarification in the reference guide.

Comment From: sbrannen

When running parallel tests with the @SpringBootTest annotation there is a bug that is causing the caching of the application context, even when using the annotation @DirtiesContext to force context re-creation. Affected spring-boot versions (the one I tried): 2.5.2 and 2.6.2

This is not a bug. On the contrary, as documented in the Parallel Test Execution section of the reference manual, using @DirtiesContext with parallel test execution is not supported.

As mentioned in https://github.com/spring-projects/spring-framework/issues/27858#issuecomment-1001999366, "If you want to trick the cache, you could add a fake property with a different value for each test." That would result in a different ApplicationContext for each test class, but it would also result in an increase in the number of application contexts managed by the cache. If you want to reduce the size of the cache, you can do that via the spring.test.context.cache.maxSize (as documented in the reference manual).

I don't need context caching: all the contrary. I want that integration tests run in parallel, each using its own context to achieve full isolation. It is possible to achieve this behavior with vanilla JUnit: you just need to "manually setup" your "test context" for each test and then run the test in parallel.

There is currently no way to completely disable context caching in the TestContext framework (see #18295). So, if you need each test class to have its own ApplicationContext, the only way to achieve that at the moment with the TestContext framework is to ensure that each test class declares context configuration that results in a unique key for the cache (as suggested by @snicoll in https://github.com/spring-projects/spring-framework/issues/27858#issuecomment-1001999366).

In light of that, I am closing this issue.

Comment From: rajjaiswalsaumya

How do we do the same if we want to test a SLice Test of a controller that depends on Configuration Property and the Service layer uses PostConstruct to massage the Configuration Property? I need context to refresh on each Test method level.

Comment From: sbrannen

Hi @rajjaiswalsaumya,

How do we do the same if we want to test a SLice Test of a controller that depends on Configuration Property and the Service layer uses PostConstruct to massage the Configuration Property? I need context to refresh on each Test method level.

It sounds like this is a question that would be better suited to Stack Overflow. As mentioned in the guidelines for contributing, we prefer to use the issue tracker only for bugs and enhancements. Feel free to update this issue with a link to the re-posted question (so that other people can find it) or add some more details if you feel this is a genuine bug.