Hi!
I would like to start a discussion about the mechanism of caching application contexts that are using @MockBeans across spring boot tests.
I was following this guide about testing the web layer: https://spring.io/guides/gs/testing-web/ and I was surprised that application context was build independently for every test class. I was hoping it will be cached to decrease test time, but I was wrong.
I've created small sample app to show this. Here is pull request: https://github.com/spring-projects/spring-boot-issues/pull/56
There's one @Service (FooBarService) used by controller (FooBarApplication) in two methods. And two tests that have some expectations about the web layer (as indicated by the @WebMvcTest annotation). Those two tests work on the same context - the only difference is distinct mocks. But if you do mvn clean test you'll see the spring banner twice, which means that context for the web layer is created two times.
I've done some research and debugging and found that this is caused by including the @MockBean and @SpyBean definitions in the key for context cache (through the MockitoContextCustomizer that is a part of MergedContextConfiguration).
Well... After rethinking all of this I can understand that from technical point of view those contexts are not the same, because mock@1 and mock@2 are not equal. But still I feel that my assumption that context should be reused is what users could expect. Especially if we realize that @WebMvcTest is used to reduce the test time by not starting tomcat. Having this goal in mind it's hard to accept the extra time of repeated context building.
I'm not sure if such a change is a bug fixing or enhancement. I feel that this behaviour is bug but you can discuss it.
For now my workaround (or maybe it's official way to do this?) is to create an abstract test class with all @MockBeans definitions, something like:
@RunWith(SpringRunner.class)
@WebMvcTest
public abstract class AbstractTest {
protected @MockBean FooBarService service;
}
public class FooTest extends AbstractTest {...}
But maybe it's better to consider an approach in which context is build once and mocks are replaced in beforeTestClass and afterTestClass methods of TestExecutionListener? And to avoid refreshing dependencies in every class referencing those mocked beans maybe it's ok to generate some beanToMockProxy object, that will be autowired in controllers and will hold a reference to mockito mock (created per test class), proxing all the methods to that mock? I don't know... I'm just thinking loud.
Cheers, Alek
Comment From: philwebb
Thanks for the detailed analysis. This is indeed a bug and it was introduced when commit a985a5c861587c296f19db6963f6dd9770e5f22f changed MockDefinition to use a ResolvableType.
The hashCode and equals method of MockDefinition should match if two different definitions are defined, but result in the same mock. The problem is that ResolvableType is actually using the source field for its hashCode method.
Comment From: philwebb
@jhoeller do you think this is a Spring Framework bug? Looking at ResolvableType, I wonder if hashCode() and equals() should use this.typeProvider.getType() rather than getSource()? It feels odd that two ResolvableTypes that ultimately resolve to the same thing should not be considered equal.
Comment From: philwebb
I've raised https://jira.spring.io/browse/SPR-14826
Comment From: wilkinsona
This is fixed in the latest Spring Framework 4.3.4 snapshots. Overriding spring.version in the project that reproduced the problem results in this output:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running example.BarTest
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.4.1.RELEASE)
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.47 sec - in example.BarTest
Running example.FooTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in example.FooTest
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
Note that the banner only appears once, i.e. the context has been reused.
Comment From: alebar
Thank you all very much for taking care of that :)
I know that the issue is closed but I'd like to follow up on the topic of context caching. You've fixed the mechanism of calculating cache's keys based on mock definitions. But I'm not convinced for the general idea of including mocks in that key :(
Please consider one more example that I've provided in issues repo. The PR is here: https://github.com/spring-projects/spring-boot-issues/pull/59
In that example I have two tests, that are testing separate features and are independent. And they are using the same WebMvc context - the only difference is in mocks used in both. Despite that, spring is creating two contexts (which may take long time) instead of one (run mvn test to see banner twice).
My feeling is that developer's decision of what to mock and when shouldn't have impact on context creation. I'd expect in that case one application context, shared between tests and with @MockBeans created and injected into context only for one test class.
What do you think?
Comment From: wilkinsona
At the time when we're getting the context cache key to decide whether a context can be reused or we need to create a new one, we have no knowledge at all of how the context's beans will be wired together. We also have no knowledge of the functionality that your test is going to exercise. This means that it's impossible for us to know that your FooTest can happily run with a mocked BarService and that your BarTest can happily run with a mocked FooService.
Comment From: ninj
Might be not quite be what @alebar intended, but would using a @ContextHierarchy help any?
http://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/test/context/ContextHierarchy.html
Comment From: wilkinsona
@mohamedj Can you please open a new issue with a small sample that reproduces the problem? The fix for this particular issue was in Spring Framework and, as far as I know, it is still in place. You may well be encountering a different problem with the same symptoms.
Comment From: rameezrajas
This problem seems to be still not resolved. We are using spring boot 1.4.2 and spring framework 4.3.4.
Comment From: philwebb
@rameezrajas We believe the original issue as reported is fixed. If you're still seeing problems can you please open a new issue and attach a sample application that replicates it. Cheers.
Comment From: opavlov24
I have the same problem as @mohamedj . If we use @MockBean and @SpringBootTest together, a context is reloaded for each test class. Without @MockBean everything is OK. We're using Spring Boot 1.5.3 .
Comment From: snicoll
@opavlov24 please see the comment just above yours?
Comment From: eutiagocosta
I also have the same problem, im using springBootVersion = '1.5.3.RELEASE', And spring-boot-test 1.5.3, Awaiting correction.
Comment From: philwebb
@eutiagocosta Please see the comment above, we believe the original issue as reported is fixed. If you're still seeing problems can you please open a new issue and attach a sample application that replicates it?
Comment From: eutiagocosta
Any test that uses org.springframework.boot.test.mock.mockito.MockBean reloads the context, for example for example
@MockBean
private RestTemplate template;
and this prevents me from testing events, when it reloads the context, spring loses itself in counting parallel processes.
Comment From: philwebb
@eutiagocosta Please open a new issue and provide a full sample application. It's expected that if you have one test that uses a mock, and one that does not, you will indeed get a different context.
Comment From: matheusalagia
@opavlov24 and @eutiagocosta take a look at this issue: https://github.com/spring-projects/spring-boot/issues/9282
Comment From: ghost
I think this is still an issue with 4.3.9 and a context using BeanRegistryPostProcessor with a bean definition using a generic resolvable type (ResolvableType.fromClassWithGenerics). Sounds like a similar source of cache busting?
Comment From: philwebb
@eighty4 Possibly. If you're able to replicate the problem in a sample, and you think it's a bug, could you please raise a new issue.
Comment From: surapuramakhil
@philwebb In reality for any project there would be at least of 40 integration tests (bare minimum) and its may not be possible for having integration tests without mocking. Or maintaining same mocks for all tests. Reloading context for 40 times Will easily make test time 1hr that very bad for testing For engineering perspective, It does not mandatory required for rebuilding context on configuration change. If proxies are maintained - this can be handled very easily. Like proxy holder contains service holder and that pointing address can be changed on mocking. We don't have to recreate all beans or change objects of each bean where mocked DI is there. And Unit FW will take care of copy cache and modify that proxy bean so scope can only to be on that test I suggest or feel like this should have to be new feature of spring boot test - which will be required for many people who are using spring.
Comment From: romashatomar
@philwebb , was the issue with SpringBoot and MockBean resolved , eventually as a part of some other ticket ? I am facing the same issue with a test class having MockBean and the other class not containing any MockBeans. Both are loading the application context (integration tests) and they run individually and are successful but as a test suite, they fail. I am using 2.2.8 Spring Boot and spring version 5.2.7. Thank you
Comment From: wilkinsona
@romashatomar When one test class mocks a bean and another doesn't, loading two separate contexts is expected as the contexts have different contents. They shouldn't fail. If you believe you've found a bug, please open a new issue providing a minimal sample that reproduces the problem.
Comment From: surapuramakhil
@wilkinsona it's defiantly not a but bug but its having a poor design which is heavily impacting test suite execution time.
to brief you with an example - its decent of having spring context boot of 90 - 120 sec. but when you have 30 ( very minimal ) but a decent complex module / software would be having more than thousands of test cases. and its never make sense of each test case should have same mocking configuration. if you have 30 test cases then execution time will be around 45 to 60 minutes. that extremely bad compared to only 30 test classes.
although if redesign happens at code for springboot testing it can be drastically getdown to much more lower which will be considered as resonable amount of time for running those test cases.
let me explain you theoretically
We don't have to rebuild entire spring context for executing of that test class. we have only rebuild the beans which are DI beans/Auto wired beans of that test class. this has to happen in copied spring context instead of original which acts as cache for rest
Comment From: brolinuk
@surapuramakhi
We don't have to rebuild entire spring context for executing of that test class. we have only rebuild the beans which are DI beans/Auto wired beans of that test class. this has to happen in copied spring context instead of original which acts as cache for rest
Just to clarify whether this is what @MockBean provides now with the latest spring or it is still something nice to have?
Comment From: surapuramakhil
@brolinuk Spring context will be rebuild even if you have different mock in beans.
Like , for example we have 2 test classes and 26 beans(a ... z) testClassA -> mocks beanA testClassB -> mocks beanB
in above case spring context will be initialized twice