Issue

  • The unit test test002in org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListenerTests fails when executed in a standalone manner.
org.opentest4j.AssertionFailedError: 
expected: "none"
 but was: null
    at java.base@17.0.9/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at java.base@17.0.9/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
    at java.base@17.0.9/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.base@17.0.9/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
    at app//org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListenerTests.test002(ResetMocksTestExecutionListenerTests.java:61)
    at java.base@17.0.9/java.lang.reflect.Method.invoke(Method.java:568)

Reason

  • The unit tests in org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListenerTests will pass when executed for the whole test class because test class is marked with @TestMethodOrder.
  • So,test001 correctly sets the return value for getMock("none").greeting() at https://github.com/spring-projects/spring-boot/blob/6065f219b34104e3e4f50999498b81d2b8ef02a9/spring-boot-project/spring-boot-test/src/test/java/org/springframework/boot/test/mock/mockito/ResetMocksTestExecutionListenerTests.java#L52
  • However, when the unit test test002 is executed on its own, the return value is not set, and thus results in the AssertionFailedError https://github.com/spring-projects/spring-boot/blob/6065f219b34104e3e4f50999498b81d2b8ef02a9/spring-boot-project/spring-boot-test/src/test/java/org/springframework/boot/test/mock/mockito/ResetMocksTestExecutionListenerTests.java#L61

Steps to Reproduce

  • Running the unit test test002 using the following command
./gradlew :spring-boot-project:spring-boot-test:test --tests=org.springframework.boot.test.mock.mockito.ResetMocksTestExecutionListenerTests.test002

Proposed Fix

  • Since the mock is not set when test002 is executed on its own, getMock("none").greeting() should be expected to return null
  • Changing the assertion to expect either null or none will solve this issue. But I would like to confirm whether this is an expected behavior, since the test class is annotated with @TestMethodOrder
  • I am happy to discuss further about this issue, and I can create a PR with the proposed fix

Version

  • Spring Version: 3.1.5

Comment From: wilkinsona

Thanks but these tests are intentionally order dependent and use @TestMethodOrder(MethodOrderer.MethodName.class) to ensure that the order is consistent.