Overview

The Spring Framework test suite contains numerous tests that are assigned to the PERFORMANCE CI build, and many of these tests have names along the lines of "ensure that XYZ is fast enough".

When these tests were originally written, they were feasible when executed on an average developer's workstation; however, over time these tests have proven to be flaky when executed on the CI server with various load levels for the various CI agents.

We are opening this issue in order to reassess whether such "performance" tests make sense in the current Spring Framework build.

Candidates

  • [ ] org.springframework.aop.framework.IntroductionBenchmarkTests

Deliverables

  • [x] Introduce a JMH benchmark infrastructure in the build
  • [ ] Search for test methods and test classes that are annotated with @EnabledForTestGroups(PERFORMANCE) (or @EnabledForTestGroups(TestGroup.PERFORMANCE)) or that contain "FastEnough" in their titles and determine if the tests should be deleted or modified to make them more robust.
  • [ ] Delete or modify each such test method or test class.

Comment From: bclozel

Thanks for creating this issue. We’ll be moving to concourse CI for this milestone and we won’t be having a « performance » build in that new pipeline.

We could however consider transforming some performances tests into proper JMH benchmarks in a dedicated module.

Comment From: rstoyanchev

+1 for jmh benchmakrs and that would also provide structure for creating more in order to properly track the impact of changes over time. Wouldn't it make sense to have these in the respective modules? Looking at this plugin it could be src/jmh/java under spring-context or spring-web, etc.

Comment From: bclozel

I've introduced the basic JMH infrastructure in 67547e6 as well as a new Wiki page on Micro-Benchmarks (feel free to edit and chime in!).

As an example, I've used this new infrastructure in #24769.

Comment From: sbrannen

I've added a Candidates section to this issue's description as a way to track classes we should be looking at. The list is currently a work in progress.