Posted By
If you have ever asked how to rerun failed TestNG tests in Jenkins without triggering the entire suite again, you are not alone. Flaky tests are one of the most frustrating challenges in large-scale CI/CD pipelines and knowing how to handle failed test cases in CI/CD efficiently can save hours of compute time per build.
This blog walks you through a reliable, production-grade approach to rerunning only what broke, using a custom TestNG listener, Maven Surefire, and a clean Jenkins shell setup.
Why TestNG failed test rerun in Jenkins often go wrong?
Most teams assume TestNG will automatically generate a testng-failed.xml after a failed run. It does, sometimes. The problem is that this file is not guaranteed to appear in every failure scenario, and when it does, it may not accurately capture every failed method. Teams looking to reduce flaky tests in Jenkins often discover this the hard way after assuming the rerun ran, only to find the results were incomplete or misleading.
This creates a dangerous blind spot. Jenkins may show a rerun step in the console log, but the rerun either never happened or ran an incomplete set of tests. There is no easy way to verify which tests were retried or whether the results are trustworthy.
What makes the default testng-failed.xml unreliable
TestNG generates its own testng-failed.xml based on internal state, but parallel test execution, listener conflicts, and certain exception types can cause methods to be skipped or misclassified. The file ends up incomplete or missing entirely, and the build result becomes ambiguous.
The result is a CI pipeline you cannot trust
When engineers cannot verify whether a rerun happened or which tests it covered, they stop trusting the pipeline. That distrust leads to full reruns, slower feedback loops, and wasted compute time across every build.
Can Jenkins rerun failed tests automatically with full reliability?
The answer is yes, but not with the default TestNG behavior. The solution is to stop relying on TestNG's native file and generate your own testng-failed.xml using a custom listener. This gives you full control over what gets captured, where the file is written, and how Jenkins uses it to automate test retries in CI/CD.
The high-level flow works in four steps. The full suite runs first, with a listener capturing every failed method. Once the run finishes, the listener writes a precise testng-failed.xml only if failures exist. Jenkins then checks for that file and reruns only the tests that depend on it. The build is marked successful if the rerun passes, and failed only if the rerun also fails.
This approach is worth adopting because it delivers on four key factors in large pipelines.
- Deterministic rerun with no guesswork on what gets retried
- No dependency on TestNG's flaky auto-generation behavior
- Jenkins-friendly design that works without plugin changes
- Scales reliably to hundreds of tests running in parallel
Tested environment
This approach was validated on the following stack. Newer versions should work, but minor behavior differences are possible.
- Java 17
- TestNG 7.10.0
- Maven Surefire Plugin 3.0.0
- Jenkins LTS
Setting up a reliable rerun strategy is one part of a healthy pipeline. If your team is looking to build or mature the broader CI/CD infrastructure, Opcito's CI/CD services cover the full spectrum from pipeline design to deployment automation.
Setting up the required dependencies to rerun failed test cases in TestNG
Add these to your pom.xml before anything else. Getting the dependency versions right is the first step toward a stable testng rerun failed tests maven setup.
TestNG dependency
<dependency>
<groupId>org.testng</groupId>
<artifactId>testng</artifactId>
<version>7.10.2</version>
<scope>test</scope>
</dependency>
Maven Surefire plugin
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
</configuration>
</plugin>
Setting testFailureIgnore to true is important. It tells Jenkins to continue to the rerun step even if the initial run has failures, rather than stopping the build immediately.
Building the custom failed suite generator
This is the core of the entire approach. The listener captures failed test methods during the run and writes a precise testng-failed.xml when the suite finishes.
import com.malrapi.qa.logger.log.Log;
import org.testng.*;
import org.testng.xml.*;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
public class FailedSuiteGenerator extends TestListenerAdapter {
// className -> failed methods
private static final Map<String, Set<String>> FAILED_TESTS =
new ConcurrentHashMap<>();
@Override
public void onTestFailure(ITestResult result) {
FAILED_TESTS
.computeIfAbsent(
result.getTestClass().getName(),
k -> ConcurrentHashMap.newKeySet()
)
.add(result.getMethod().getMethodName());
}
@Override
public void onFinish(ITestContext context) {
generateFailedSuiteIfNeeded();
}
private synchronized void generateFailedSuiteIfNeeded() {
if (FAILED_TESTS.isEmpty()) {
Log.info("✔ No test failures detected. Skipping failed suite generation.");
return;
}
try {
XmlSuite suite = new XmlSuite();
suite.setName("Failed Tests Suite");
XmlTest test = new XmlTest(suite);
test.setName("Failed Tests");
List<XmlClass> classes = new ArrayList<>();
for (Map.Entry<String, Set<String>> entry : FAILED_TESTS.entrySet()) {
XmlClass xmlClass = new XmlClass(entry.getKey());
List<XmlInclude> includes = new ArrayList<>();
for (String method : entry.getValue()) {
includes.add(new XmlInclude(method));
}
xmlClass.setIncludedMethods(includes);
classes.add(xmlClass);
}
test.setXmlClasses(classes);
File outDir = new File("target/surefire-reports");
outDir.mkdirs();
Path failedXml =
Path.of(outDir.getAbsolutePath(), "testng-failed.xml");
Files.writeString(failedXml, suite.toXml());
Log.info("✔ Custom testng-failed.xml generated at: " + failedXml);
} catch (Exception e) {
Log.error("❌ Failed to generate testng-failed.xml", e);
}
}
}
Registering the listener
You have two options depending on your project setup.
Option 1: Register via testng.xml
<listeners>
<listener class-name="com.yourpackage.FailedSuiteGenerator"/>
</listeners>
Option 2: Register via annotation on the base test class
@Listeners(FailedSuiteGenerator.class)
public class BaseTest {
}
Both approaches work. The annotation method is useful when you want the listener active across all tests that extend BaseTest without modifying the XML.
How to run TestNG tests on Jenkins with a two-step rerun setup
This is where everything comes together. The Jenkins freestyle job uses two shell steps, each with a clear and distinct responsibility.
Step 1: Run the full test suite
mvn clean test || true
The || true at the end prevents Jenkins from stopping the job if tests fail. This step works for any TestNG-based framework, including the following.
- Plain Java + TestNG
- Selenium + TestNG
- REST Assured + TestNG
- Spring Boot + TestNG
- Any custom TestNG framework
Passing optional parameters
If your project needs environment flags or a specific suite file, add them here without affecting the rerun logic.\
mvn clean test \ -Denv=stage \ -DsuiteXmlFile=testng.xml \ -DanyCustomFlag=true \ || true
These flags are project-specific and are separate from how the rerun works.
Step 2: Trigger the Jenkins pipeline test retry for failed tests only
echo "Checking for failed TestNG suite..." if [ -f target/surefire-reports/testng-failed.xml ]; then echo "Re-running failed tests using testng-failed.xml" mvn test -DsuiteXmlFile=target/surefire-reports/testng-failed.xml else echo "No failed tests found. Skipping re-run." fi
This step checks whether the custom testng-failed.xml was generated. If it exists, only those tests run. If it does not exist, the step exits cleanly. There is no framework coupling here, and no assumptions about the kinds of tests in the suite.
Why this design is reusable across frameworks
Each concern is handled in the right layer with no cross-contamination between framework config and rerun logic. The table below shows exactly where responsibility sits.
| Concern | Handled where |
|---|---|
| Test failures | TestNG listener |
| Failed suite creation | Java code |
| Re-run decision | Jenkins shell |
| Framework config | Optional Maven flags |
This separation is what makes the design reusable. One Jenkins job, one rerun strategy, and multiple frameworks supported without any special casing per project.
Understanding the TestNG results and build result logic in Jenkins
The build result in this setup is not left to Jenkins defaults. It is explicitly determined by whether the rerun passes or fails, giving you a trustworthy, predictable signal in your TestNG report in Jenkins at the end of every pipeline run.
| Scenario | Build Result |
|---|---|
| Initial run fails, rerun passes | Success |
| Initial run fails, rerun also fails | Failure |
| Initial run passes, no failed XML generated | Failure (by design) |
The third row is intentional. The current requirement is that a build is marked successful only when a rerun explicitly passes. This behavior can be adjusted later if your team's requirements change.
How to confirm the rerun happened without reading logs
One of the biggest pain points with rerun setups is not knowing whether the rerun executed at all. Here are three ways to verify it without digging through logs.
- Check for the generation log line: Once the listener finishes writing the file, it logs the output path. Look for this line in the Jenkins console:
✔ Custom testng-failed.xml generated at: target/surefire-reports/testng-failed.xml - Check for the rerun marker in the Jenkins console: The shell script in step 2 prints a confirmation before triggering the rerun. If the rerun step executed, you would see this line:
Re-running failed tests using testng-failed.xml - Separate the rerun results into their own directory: For a cleaner Jenkins view, write rerun results to a dedicated folder rather than mixing them with the initial run output. Point your configuration to
target/surefire-reports-rerun/and the rerun results will appear as a distinct artifact in the Jenkins UI, separate from the initial run output.
Build smarter pipelines, not longer ones
Reruns should be intentional, visible, and predictable. Relying on TestNG's auto-generated file introduces ambiguity that compounds over time, especially in large parallel suites. By writing your own testng-failed.xml through a listener, you take full ownership of which failed test cases rerun in TestNG and why the build passed or failed.
The approach covered here has been used by the engineering team at Opcito to bring clarity to flaky CI pipelines at scale. Teams that want to go further with structured automation practices can explore Opcito's test automation services for end-to-end QA engineering support. Contact Opcito's experts who are ready to help you build pipelines that are reliable, transparent, and built to scale.
Related Blogs
