Overview

Now that we have AOT testing support in place, we should introduce a way to run end-to-end tests for all Spring TestContext Framework integration tests in the spring-test module.

This will help us to discover issues with our ongoing AOT efforts.

Related Issues

  • 30834

  • 31472

  • 31733

  • 31442

Comment From: sbrannen

Now that we have AOT testing support in place, we should introduce a way to run end-to-end tests for all Spring TestContext Framework integration tests in the spring-test module.

For the sake of simplicity, we have not introduced a dedicated build for this purpose.

Instead, we currently rely on manual runs of AotIntegrationTests#endToEndTestsForEntireSpringTestModule() within the spring-test module.

Several tests are now annotated with the newly introduced @DisabledInAotMode annotation. Note, however, that @DisabledInAotMode has only been applied for use cases that will never be supported in AOT mode (or are unlikely to be supported in AOT mode, such as @ContextHierarchy and @EJB).

Current results

Test run finished after 5887 ms
[       399 containers found      ]
[        71 containers skipped    ]
[       328 containers started    ]
[         0 containers aborted    ]
[       326 containers successful ]
[         2 containers failed     ]
[       745 tests found           ]
[       123 tests skipped         ]
[       621 tests started         ]
[         1 tests aborted         ]
[       590 tests successful      ]
[        30 tests failed          ]

Failing Test Classes:
org.springframework.test.context.groovy.AbsolutePathGroovySpringContextTests
org.springframework.test.context.groovy.GroovySpringContextTests
org.springframework.test.context.groovy.RelativePathGroovySpringContextTests
org.springframework.test.context.junit.jupiter.orm.JpaEntityListenerTests
org.springframework.test.context.junit4.AbsolutePathSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.ClassPathResourceSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.ConcreteTransactionalJUnit4SpringContextTests
org.springframework.test.context.junit4.InheritedConfigSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.MultipleResourcesSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.ParameterizedDependencyInjectionTests
org.springframework.test.context.junit4.RelativePathSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.SpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.annotation.AnnotationConfigSpringJUnit4ClassRunnerAppCtxTests
org.springframework.test.context.junit4.spr9799.Spr9799XmlConfigTests
org.springframework.test.context.testng.ConcreteTransactionalTestNGSpringContextTests

Failure Categories

Uses @Resource in test class

  • To be addressed by #31733

  • org.springframework.test.context.groovy.AbsolutePathGroovySpringContextTests

  • org.springframework.test.context.groovy.GroovySpringContextTests
  • org.springframework.test.context.groovy.RelativePathGroovySpringContextTests
  • org.springframework.test.context.junit4.AbsolutePathSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.ClassPathResourceSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.ConcreteTransactionalJUnit4SpringContextTests
  • org.springframework.test.context.junit4.InheritedConfigSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.MultipleResourcesSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.RelativePathSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.SpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.junit4.annotation.AnnotationConfigSpringJUnit4ClassRunnerAppCtxTests
  • org.springframework.test.context.testng.ConcreteTransactionalTestNGSpringContextTests

Uses @PersistenceContext in test class

  • To be addressed by #31442

  • org.springframework.test.context.junit.jupiter.orm.JpaEntityListenerTests

Uses <mvc:annotation-driven />

  • To be addressed by #31472 and #31495

  • org.springframework.test.context.junit4.spr9799.Spr9799XmlConfigTests

Unknown Reason

The following test class passes in AOT mode when run by itself but fails when run with all integration tests in the spring-test module.

  • org.springframework.test.context.junit4.ParameterizedDependencyInjectionTests

Comment From: snicoll

Note, however, that @DisabledInAotMode has only been applied for use cases that will never be supported in AOT mode (or are unlikely to be supported in AOT mode, such as @ContextHierarchy and @EJB).

If that's the case, then we need to revisit https://github.com/spring-projects/spring-framework/issues/29030#issue-1352410452

Comment From: sbrannen

Now that #31733 has been resolved, we only have 2 remaining test classes that fail in AOT mode.

Test run finished after 6534 ms
[       399 containers found      ]
[        71 containers skipped    ]
[       328 containers started    ]
[         0 containers aborted    ]
[       327 containers successful ]
[         1 containers failed     ]
[       745 tests found           ]
[       123 tests skipped         ]
[       622 tests started         ]
[         1 tests aborted         ]
[       615 tests successful      ]
[         6 tests failed          ]

Failing Test Classes:
org.springframework.test.context.junit.jupiter.orm.JpaEntityListenerTests
org.springframework.test.context.junit4.ParameterizedDependencyInjectionTests
  • JpaEntityListenerTests still fails because #31442 has not yet been addressed.
  • ParameterizedDependencyInjectionTests still passes in AOT mode when run by itself but fails when run with all integration tests in the spring-test module. 🤷‍♂️

Comment From: sbrannen

Update

Test run finished after 6250 ms
[       425 containers found      ]
[        71 containers skipped    ]
[       353 containers started    ]
[         0 containers aborted    ]
[       346 containers successful ]
[         7 containers failed     ]
[       841 tests found           ]
[       123 tests skipped         ]
[       698 tests started         ]
[         1 tests aborted         ]
[       684 tests successful      ]
[        13 tests failed          ]

Failing Test Classes:
org.springframework.test.context.bean.override.convention.TestBeanByTypeIntegrationTests
org.springframework.test.context.bean.override.convention.TestBeanInheritanceIntegrationTests$ConcreteTestBeanIntegrationTests
org.springframework.test.context.bean.override.convention.TestBeanIntegrationTests
org.springframework.test.context.bean.override.convention.TestBeanIntegrationTests$TestBeanNested
org.springframework.test.context.bean.override.convention.TestBeanIntegrationTests$TestBeanNested2
org.springframework.test.context.bean.override.mockito.MockitoBeanIntegrationTests
org.springframework.test.context.bean.override.mockito.MockitoSpyBeanIntegrationTests
org.springframework.test.context.bean.override.mockito.MockitoSpyBeanIntegrationTests$MockitoBeanNested
org.springframework.test.context.junit.jupiter.orm.JpaEntityListenerTests
org.springframework.test.context.junit4.ParameterizedDependencyInjectionTests
  • Bean Override tests are currently not supported in AOT mode.
  • JpaEntityListenerTests still fails because #31442 has not yet been addressed.
  • ParameterizedDependencyInjectionTests still passes in AOT mode when run by itself but fails when run with all integration tests in the spring-test module. 🤷‍♂️

Comment From: sbrannen

Update

After recent revisions to our bean override tests, the results are now as follows.

Test run finished after 5842 ms
[       427 containers found      ]
[        71 containers skipped    ]
[       355 containers started    ]
[         0 containers aborted    ]
[       350 containers successful ]
[         5 containers failed     ]
[       838 tests found           ]
[       123 tests skipped         ]
[       700 tests started         ]
[         1 tests aborted         ]
[       693 tests successful      ]
[         6 tests failed          ]

Failing Test Classes:
org.springframework.test.context.bean.override.convention.TestBeanByTypeIntegrationTests
org.springframework.test.context.bean.override.convention.TestBeanInheritanceIntegrationTests$ConcreteTestBeanIntegrationTests
org.springframework.test.context.bean.override.mockito.MockitoBeanByTypeIntegrationTests
org.springframework.test.context.bean.override.mockito.MockitoBeanIntegrationTests
org.springframework.test.context.junit.jupiter.orm.JpaEntityListenerTests
org.springframework.test.context.junit4.ParameterizedDependencyInjectionTests
  • AOT issues with Bean Override tests should be addressed in #32925.
  • JpaEntityListenerTests still fails because #31442 has not yet been addressed.
  • ParameterizedDependencyInjectionTests still passes in AOT mode when run by itself but fails when run with all integration tests in the spring-test module. 🤷‍♂️