Spring Offer a way for MockitoTestExecutionListener to enable strict stubbing

Running tests with SpringRunner does not report unused stubs like MockitoJUnitRunner does.

In MockitoTestExecutionListener.initMocks it should use

Mockito.mockitoSession()
  .initMocks(this)
  .strictness(Strictness.STRICT_STUBS)
  .startMocking();

to enable strict mocking.

Comment From: bclozel

This feature is still being debated by the community in mockito/mockito#769, and I'm not sure we're in a position to force that choice right now on the Spring Boot community. Looking at the comments in that issue, it seems that some developers had issues while applying this to their projects; arguable, some of those might be linked to their setup or even bad practices.

If we do consider that change, we should do that in a proper milestone and document that change properly. I'm marking this issue for team attention and we'll discuss this.

In the meantime @jschmied, don't hesitate to point us to official Mockito documentation/best practices about this feature.

Comment From: jschmied

The discussion you are refering ist ourdated. Since Mockito 2.+, strict stubbing is used by default when initializing our mocks using either of:

MockitoJUnitRunner
MockitoJUnit.rule()

Spring ist using Mockito.initMocks() instead of Mockito.mockitoSession().initMocks(this).strictness(Strictness.STRICT_STUBS).startMocking(); what is causing this problem.

See: https://www.baeldung.com/mockito-unnecessary-stubbing-exception

So by default (We are at Mockito 2.23 in Spring Boot 2.1) all plain JUnit test use strict stubbing by default but all Sprint tests do not.

This behaviour is inconstent and anoying.

Of course, enabling it would possibly force users to clean up their spring tests. All plain JUnit test had to be cleaned up with switch to spring boot 2.x already.

It would be nice to be able it at least by some property or annotation (or in my oppinon, better switch it on and give user the possibility to disable it easily)

Comment From: wilkinsona

As far as I can tell, the issue to which @bclozel linked remains accurate. MockitoJUnit.rule() calls JUnitRule(Plugins.getMockitoLogger(), Strictness.WARN) in Mockito 2.28 and continues to do so in Mockito 3.1.

Comment From: jschmied

I have seen nobody using MockitoJUnit.rule(), anybody uses MockitoJUnitRunner.

Fact is: if you have two JUnit test, one with

@RunWith(MockitoJUnitRunner.class)

one with

@RunWith(SpringRunner.class)
@SpringBootTest()

the first one will fail when there are unnecessary stubs (default is STRICT), the second one will pass.

You can test easily.

Comment From: wilkinsona

On further inspection, as far as I can tell, Mockito does not use STRICT_STUBS by default in any situation. As with the rule, the runner's default behaviour is to report problems to the console. This is the equivalent of Strictness.WARN. Strictness.STRICT_STUBS is "planned as default for Mockito v4".

I think we could safely switch to Strictness.WARN. It will result in some additional console output containing hints on how to improve the tests' usage of Mockito but, unlike STRICT_STUBS, it will not cause test failures. I think there's even an argument to be made to do this in a Spring Boot maintenance release so that we're aligned with Mockito's default behaviour. We would then switch to Strictness.STRICT_STUBS by default when we upgrade to a version of Mockito that does so.

Comment From: jschmied

This would be nice. I usually run JUnit from eclipse first and unused stubs make the tests failing in the IDE. I did not test this as part of maven build on Jenkins. So you might be right.

Comment From: wilkinsona

@jschmied Could you share some example tests with us please? It would be useful to see exactly what you are seeing and to confirm that Mockito's behaviour matches its documentation.

Comment From: jschmied

Sure. This one fails with UnnecessaryStubbingException:

@RunWith(MockitoJUnitRunner.class)
public class TestJUnit {
    @Test
    public void test1() {
        List<Object> list = mock(List.class);
        when(list.get(anyInt())).thenReturn(new Object());
    }
}

This one passes:

@RunWith(SpringRunner.class)
@SpringBootTest
public class TestSpring {

    @Test
    public void test1() {
        List<Object> list = mock(List.class);
        when(list.get(anyInt())).thenReturn(new Object());
    }

}

Only the annotations on top differ

Comment From: wilkinsona

Thank you, @jschmied. I can see now that I was wrong before. Sorry. Mockito does fail by default when using MockitoJUnitRunner. It does not fail by default when using MockitoRule but does output some hints to the console.

The concept of strictness differs between the runner and the rule. There are three variants of the runner:

Silent
Strict
StrictStubs

There are three variants of Strictness used by the rule:

LENIENT
WARN
STRICT_STUBS

The Silent runner is equivalent to the LENIENT rule. The StrictStubs runner is equivalent to the STRICT_STUBS rule. The Strict runner and the WARN rule are not equivalent. The former fails when there is unnecessary stubbing while the latter only outputs hints to the console.

An equivalent of the Strict runner doesn't appear to be available to us as there's no equivalent Strictness. STRICT_STUBS would be too much as it's stricter than Mockito is by default even when using the runner. Using WARN by default in Boot feels reasonable to me. It matches the default of the rule and is no more strict than the default of the runner.

Comment From: philwebb

I think there's currently too much risk in changing our defaults at this point. The MockitoTestExecutionListener.initMocks method currently just delegates to MockitoAnnotations.initMocks (from the org.mockito package).

As soon as the Mockto team decide that MockitoAnnotations.initMocks should switch to a stricter form, we'll pick it up by virtue of an upgrade.

The only problem that we see with our current approach is that we don't offer a good way to change the defaults for an individual test. I think we should investigate offering a way to switch the strictness for an individual test.

Comment From: JoaaoVerona

+1. It would be nice if we could change the strictness used in a test. Even better if you could support @MockitoSettings(strictness = STRICT_STUBS).

Comment From: mminot-yseop

Indeed, it took me a while to understand why my unit and integration tests were behaving differently, and noticing that @MockitoSettings(strictness = STRICT_STUBS) didn’t help with @SpringBootTest was quite a letdown. For now I’ll have to resort to something like…

@AfterEach
void afterEach() {
    // Only stubbed (or otherwise expected) calls allowed.
    ignoreStubs(/* explicit list of mockbeans here */);
    verifyNoMoreInteractions(/* the same list */);
}

This is not really ideal (and does not provide the usual nice helpful errors for unnecessary stubbings).

Comment From: iekdosha

A workaround to manually set the strictness would help.

Comment From: enepomnyaschih

@mminot-yseop A couple of remarks:

Probably reset(/* the same list */) is necessary, or the test should be annotated with @DirtiesContext. Otherwise mocks get preserved between tests (or maybe I did something wrong when tested it).
This workaround doesn't verify unnecessary stubbings.

Due to these issues, I switched back to vanilla Mockito and its @Mock and @InjectMocks annotations.

Comment From: mminot-yseop

Probably reset(/* the same list */) is necessary, or the test should be annotated with @DirtiesContext. Otherwise mocks get preserved between tests (or maybe I did something wrong when tested it).

Hum I was using this in conjunction with @Mock and co., and it seemed fine. I virtually never had to use reset in my life.

This workaround doesn't verify unnecessary stubbings.

I said so in my message, you know 😁 ↓

does not provide the usual nice helpful errors for unnecessary stubbings

Yeah probably not perfect at all. I kinda gave up on this since that first message and bit the bullet of “no strictness”. 😢

Comment From: robwbruder

A workaround to enable "strict stubs" in @SpringBootTest tests can be found here: https://stackoverflow.com/a/75388020/20670189

Comment From: vpavic

Since Spring Boot usually aligns with the defaults of technologies it integrates (and in some cases even maintains the tests that verify the alignment), what's the reason not to do the same here?

Having substantially different behavior of mocks between tests set up with Mockito's own facilities (for example, @ExtendWith(MockitoExtension.class)) and ones set up by Spring Boot is quite annoying.

Comment From: wilkinsona

There are three different ways to do things with Mockito. In addition to @ExtendWith(MockitoExtension.class), there's also a JUnit runner and MockitoAnnotations.initMocks. Each has different default behavior. Boot aligns with MockitoAnnotations.initMocks as that's what it calls under the covers.

Comment From: mminot-yseop

There are three different ways to do things with Mockito. In addition to @ExtendWith(MockitoExtension.class), there's also a JUnit runner and MockitoAnnotations.initMocks. Each has different default behavior. Boot aligns with MockitoAnnotations.initMocks as that's what it calls under the covers.

That’s an interesting point. However, this issue, I think, is not exactly about the default behavior but mostly about the lack of simple, built-in way to switch to the “strict” mode in a Boot test, and therefore remains valid IMHO.

Sidenote: I kinda fail to see in which contexts it can be interesting not to be strict. 🤔 (But I’m no expert of course, just your average user.) The most extreme cases I saw so far were tests with at most one or two annoying mocks that are not-so-relevant to the things being tested (while still being relied upon here and there), but in such cases there’s still lenient().when(…), which I generally use (sparingly) along with code comments justifying its presence.

Comment From: wilkinsona

@mminot-yseop Apologies for the confusion. I was responding to @vpavic. We agree that this issue remains valid. It would have been closed if we thought otherwise.

Comment From: mminot-yseop

@mminot-yseop Apologies for the confusion. I was responding to @vpavic. We agree that this issue remains valid. It would have been closed if we thought otherwise.

Sure 😄 No real confusion, I guess I was just being talkative. Now that I think about it:

there's also a JUnit runner

Isn’t that the JUnit 4 way of doing things, though? I got the impression that nearly all modern-ish projects rely on v5, and therefore use @ExtendWith(MockitoExtension.class). As for manual calls to initMocks, there seem to be like 3–4 occurrences of that in my company’s codebase so far (including one in a terribly old and dead project, and some really dubious uses), But perhaps it’s way more common in some other, slightly esoteric contexts and projects, I dunno. (Ran ag initMocks --java ~/workspace/)

Comment From: vpavic

@wilkinsona Sorry, but I disagree with your reasoning here. Mockito clearly documents (section 40. Improved productivity and cleaner tests with "stricter" Mockito) that:

Starting with version 2.1 Mockito has been getting new features that nudge the framework towards "strictness".

From end user perspective, it shouldn't matter which way of configuring Mockito does Spring Boot use internally, but rather what Mockito's user-facing configuration facilities do.

These days, JUnit Platform is the dominant unit testing framework and @ExtendWith(MockitoExtension.class) is the preferred method of setting up Mockito there, which results in strict mocks. As this is what most users will end up with when using Mockito in tests that don't involve Spring Boot's testing support, IMO that's what Spring Boot should align with as well.

Comment From: wilkinsona

We're at risk of going round in circles here.

but rather what Mockito's user-facing configuration facilities do.

MockitoAnnotations.openMocks (the non-deprecated replacement for initMocks) is also a user-facing configuration facility. It's documented on the page to which you have already linked.

As this is what most users will end up with when using Mockito in tests that don't involve Spring Boot's testing support, IMO that's what Spring Boot should align with as well.

This ignores the risk that @philwebb has already described above. Minimising that risk requires a setting that allows the strictness to be configured. That's what this issue is tracking.

Comment From: vpavic

Both this issue and the comment stating risks are 3+ years old now. In the meanwhile there has been no evidence that Mockito will stop favoring the strictness by default, quite the opposite. I think that should warrant re-evaluation of the goals of this ticket, which IMO should be to flip the default behavior and offer a way to disable strict stubbing.

I'm aware that MockitoAnnotations.openMocks is documented (and also preferred by the team in Mockito-based tests within this project) but since JUnit Platform has become dominant in organizations I've worked with I literally haven't seen anyone prefer this style over using MockitoExtension. The initMocks approach was popular with JUnit 4 due to limitations of @RunWith however JUnit Platform's @ExtendWith being both repeatable and accepting an array of extensions (in contrast to @RunWith accepting a single runner) it's simply too low-level and verbose to be attractive to the developers these days.

You don't have to take my word on this, since a very simple GitHub code search can easily show which approach is the more popular one: - https://cs.github.com/?scopeName=All+repos&scope=&q=%22org.junit.jupiter.api.Test%22+AND+%22openMocks%22 - https://cs.github.com/?scopeName=All+repos&scope=&q=%22org.junit.jupiter.api.Test%22+AND+%22MockitoExtension%22

That's an order of magnitude difference and I'd argue that the figures are even tilted to benefit the openMocks approach here, since some of the code was definitely migrated from JUnit 4 to JUnit Platform in simplest possible fashion and is not utilizing @ExtendWith(MockitoExtension.class) for that reason.

Comment From: snicoll

With @MockBean and friends being moved over framework, I am going to move this issue as well.

@simonbasle can you please review the history of this issue? Perhaps a chance for us to consider this in some form for Spring Framework 6.2 still.

Comment From: simonbasle

I think we can use 6.2 as an opportunity to change Spring Framework's newly introduced @MockitoBean default to the strict style, with an additional class-level annotation that can be used to switch to a lenient style.

I was not able to find a way to account for the @MockitoSettings annotation since it is meta-annotated with @ExtendWith(MockitoExtension.class), but the usage pattern would be pretty similar:

@SpringJUnitConfig(Config.class)
@MockitoBeanSettings(Strictness.LENIENT)
public class MyIntegrationTests {
    @Test
    public void unusedStubbingNotReported() {
        var list = Mockito.mock(List.class);
        Mockito.when(list.get(Mockito.anyInt())).thenReturn(new Object());
    }
}

Comment From: sbrannen

I was not able to find a way to account for the @MockitoSettings annotation since it is meta-annotated with @ExtendWith(MockitoExtension.class),

It's indeed unfortunate that @MockitoSettings auto-registers the MockitoExtension, but then again... @MockitoSettings resides in the mockito-junit-jupiter module. So it's somewhat expected.

but the usage pattern would be pretty similar:

Yes, your proposal looks good. 👍

Comment From: simonbasle

Unfortunately, we had to backtrack on this in 6.2.0-RC2 due to various blocking issues with MockitoSession and MockitoExtension integration in a SpringExtension-enabled test suite. We'll reach out to the Mockito team to explore possible improvements in the Spring Framework 7 timeframe.

Comment From: vpavic

Is there an issue that tracks the re-introduction of strict stubbing as the default behavior?

Comment From: sbrannen

Is there an issue that tracks the re-introduction of strict stubbing as the default behavior?

There wasn't one, but there is now. 😉

34419

Spring Offer a way for MockitoTestExecutionListener to enable strict stubbing

34419