Retry Listeners for Flaky Mobile Tests

Network timeouts, emulator hiccups, and animation timing cause intermittent test failures that are not real bugs. IRetryAnalyzer gives tests a second chance without manual re-runs. Used correctly, it reduces false negatives. Used carelessly, it hides real bugs.

Implementing IRetryAnalyzer

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
 
public class RetryAnalyzer implements IRetryAnalyzer {
 
    private int attempt = 0;
    private static final int MAX_RETRIES = 2;
 
    @Override
    public boolean retry(ITestResult result) {
        if (attempt < MAX_RETRIES) {
            attempt++;
            return true;  // retry this test
        }
        return false; // give up after MAX_RETRIES
    }
}

retry() is called when a test fails. Return true to retry, false to stop. The attempt counter is per-instance — TestNG creates a new RetryAnalyzer per test method invocation, so the counter resets between tests.

Attaching the analyzer to tests

Per test:

@Test(retryAnalyzer = RetryAnalyzer.class)
public void testNetworkSensitiveFlow() { ... }

Via annotation transformer (global):

Applying retryAnalyzer to every @Test annotation manually doesn't scale. Use a IAnnotationTransformer listener to inject it globally:

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;
import java.lang.reflect.Constructor;
import java.lang.reflect.Method;
 
public class RetryTransformer implements IAnnotationTransformer {
 
    @Override
    public void transform(ITestAnnotation annotation,
                          Class testClass,
                          Constructor testConstructor,
                          Method testMethod) {
        annotation.setRetryAnalyzer(RetryAnalyzer.class);
    }
}

<suite name="Mobile Suite" parallel="tests" thread-count="2">
  <listeners>
    <listener class-name="com.example.listeners.RetryTransformer"/>
  </listeners>
  <!-- ... -->
</suite>

Now every test in the suite gets automatic retry without modifying test classes.

Excluding stable tests from retry

Some tests should never retry — a test that creates an order and verifies it shouldn't run twice and create two orders. Use a custom annotation:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface NoRetry {}

Check for it in the transformer:

@Override
public void transform(ITestAnnotation annotation,
                      Class testClass,
                      Constructor testConstructor,
                      Method testMethod) {
    if (testMethod.isAnnotationPresent(NoRetry.class)) {
        return; // don't set retry analyzer
    }
    annotation.setRetryAnalyzer(RetryAnalyzer.class);
}

@Test
@NoRetry
public void testPlaceOrder() {
    // This test creates a real order — must not retry
}

Logging retry attempts

Knowing which tests retried (and how many times) is essential for identifying flaky tests. Log in retry():

public class RetryAnalyzer implements IRetryAnalyzer {
    private int attempt = 0;
    private static final int MAX_RETRIES = 2;
    private static final Logger log = LoggerFactory.getLogger(RetryAnalyzer.class);
 
    @Override
    public boolean retry(ITestResult result) {
        if (attempt < MAX_RETRIES) {
            attempt++;
            log.warn("Retrying test '{}' (attempt {}/{}): {}",
                result.getName(),
                attempt,
                MAX_RETRIES,
                result.getThrowable().getMessage()
            );
            return true;
        }
        log.error("Test '{}' failed after {} retries", result.getName(), MAX_RETRIES);
        return false;
    }
}

TestNG result listener for retry tracking

Track retry statistics with a ITestListener:

public class RetryReportListener implements ITestListener {
 
    private final Map<String, Integer> retryCount = new ConcurrentHashMap<>();
 
    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
        // This callback fires on retry attempts (failed but retrying)
        String testName = result.getName();
        retryCount.merge(testName, 1, Integer::sum);
    }
 
    @Override
    public void onFinish(ITestContext context) {
        retryCount.forEach((test, count) ->
            System.out.printf("Test '%s' was retried %d time(s)%n", test, count)
        );
    }
}

After enough runs, the tests with the highest retry counts are your flake candidates — investigate and fix rather than letting retry hide them indefinitely.

Resetting app state before retry

When a test fails mid-flow, the app may be in a broken state — mid-checkout, error dialog showing, keyboard open. The next retry starts from this state and will likely fail again for a different reason.

Fix this by resetting app state in the retry analyzer:

@Override
public boolean retry(ITestResult result) {
    if (attempt < MAX_RETRIES) {
        attempt++;
        // Reset app to clean state
        AppiumDriver driver = DriverManager.getDriver();
        if (driver != null) {
            try {
                driver.terminateApp(getAppPackage());
                driver.activateApp(getAppPackage());
            } catch (Exception e) {
                // If terminate fails, the session may be corrupt — don't retry
                return false;
            }
        }
        return true;
    }
    return false;
}

terminateApp + activateApp is faster than quitting and creating a new session, while still clearing any mid-flow state.