Spring Throwing SkipException should not fail the test with TestNG

Consider below example: Example uses following dependencies: spring-test:5.3.3 spring-context:5.3.3 spring-core:5.3.3 testng:7.3.0

package com;

import com.SkipTest.SkipTestConfig;
import com.SkipTest.SkipTestListener;
import org.springframework.context.annotation.Configuration;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.testng.AbstractTestNGSpringContextTests;
import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@ContextConfiguration(classes = SkipTestConfig.class)
@Listeners(SkipTestListener.class)
public class SkipTest extends AbstractTestNGSpringContextTests {

    @Test
    public void shouldNotBeExecuted() {
        System.out.println("EXECUTED");
    }

    @Configuration
    public static class SkipTestConfig {

    }

    public static class SkipTestListener implements IInvokedMethodListener {
        @Override
        public void beforeInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
            throw new SkipException("skip");
        }

        @Override
        public void afterInvocation(IInvokedMethod iInvokedMethod, ITestResult iTestResult) {
            System.out.println(iTestResult.getStatus());
        }
    }
}

When extends AbstractTestNGSpringContextTests is added then test fails because throwing SkipException in listener method is treated as any other exception. When extends AbstractTestNGSpringContextTests is removed, then shouldNotBeExecuted is skipped as expected.

Use case: My system level tests use TestNG listener mechanism to perform several actions, one of which is deciding if given system test can be executed. If test should not be executed then we mark it as skipped and throw SkipException. This worked flawlessly for TestNG versions 6.+ for version 7.+ it does not, and the issue seems to be in AbstractTestNGSpringContextTests.

Comment From: sbrannen

Thanks for raising the issue.

It took me a while to track it down, but it turns out that this has nothing to do with Spring.

For example, the following test class passes with TestNG 6.11 (with the test method properly skipped) and starts fails with TestNG 6.13.1 or higher.

package com.example.testng_skip;

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

import com.example.testng_skip.TestNGSkippedTests.MySkipTestListener;

@Listeners(MySkipTestListener.class)
public class TestNGSkippedTests {

    @BeforeMethod
    void beforeMethod() {
        System.err.println(">>>> before ");
    }

    @Test
    void shouldNotBeExecuted() {
        System.err.println(">>>> test ");
    }

    public static class MySkipTestListener implements IInvokedMethodListener {

        @Override
        public void beforeInvocation(IInvokedMethod invokedMethod, ITestResult testResult) {
            throw new SkipException("skip");
        }

        @Override
        public void afterInvocation(IInvokedMethod method, ITestResult testResult) {
        }
    }
}

If you comment out the @BeforeMethod method, the test method will be skipped as expected.

So, there is a regression in TestNG itself.

Comment From: sbrannen

So, there is a regression in TestNG itself.

In light of that, I am closing this issue.

I believe this is effectively a duplicate of https://github.com/cbeust/testng/issues/1632.

Comment From: sbrannen

@klubi, for future reference, whenever you are aware of a related issue open against TestNG (i.e., https://github.com/cbeust/testng/issues/1632), it would be very helpful to us if you could mention that issue in advance so that we may better inform ourselves of possible causes before spending a lot of time investigating on our own.

Thanks

Comment From: klubi

Yeah... @sbrannen I should have :( sorry, will do better next time.