CommandsIntermediate7-9 min reference

TestNG

A practical reference for TestNG — the JVM test framework with first-class data providers, groups, parallel execution, and listener hooks. Examples assume Java; the same APIs map directly to Kotlin.

Annotations

Annotation	Runs
`@Test`	The test method itself
`@BeforeSuite` / `@AfterSuite`	Once per suite
`@BeforeTest` / `@AfterTest`	Once per `<test>` in `testng.xml`
`@BeforeClass` / `@AfterClass`	Once per test class
`@BeforeMethod` / `@AfterMethod`	Before / after every `@Test` method
`@BeforeGroups("auth")` / `@AfterGroups("auth")`	Around all tests in a group

Execution order

@BeforeSuite
  @BeforeTest
    @BeforeClass
      @BeforeGroups("g")
        @BeforeMethod  →  @Test  →  @AfterMethod
        @BeforeMethod  →  @Test  →  @AfterMethod
      @AfterGroups("g")
    @AfterClass
  @AfterTest
@AfterSuite

`@Test` attributes

@Test(
  description       = "Login with admin credentials",
  enabled           = true,
  priority          = 1,                          // lower runs first
  timeOut           = 5_000,                      // milliseconds
  expectedExceptions = ValidationException.class,
  dependsOnMethods  = {"setUpDatabase"},
  dependsOnGroups   = {"smoke"},
  invocationCount   = 3,                          // run 3 times
  invocationTimeOut = 30_000,
  groups            = {"login", "smoke"},
  alwaysRun         = false                       // run even if a dep fails
)
public void testAdminLogin() { /* ... */ }

Assertions

import static org.testng.Assert.*;
 
assertEquals(actual, expected);
assertEquals(actual, expected, "user id mismatch");
assertNotEquals(actual, unexpected);
assertTrue(condition);
assertFalse(condition);
assertNull(object);
assertNotNull(object);
assertSame(a, b);                       // same reference
assertNotSame(a, b);
 
assertEquals(arr1, arr2);               // arrays + collections — content compare
assertEqualsNoOrder(arr1, arr2);
 
// Exceptions
expectThrows(IllegalArgumentException.class, () -> svc.create(null));

Soft assertions — collect, then report

assertX(...) aborts the test on the first failure. SoftAssert collects them and only fails when you call assertAll().

import org.testng.asserts.SoftAssert;
 
@Test
public void dashboardSummary() {
  SoftAssert s = new SoftAssert();
 
  s.assertEquals(driver.getTitle(), "Dashboard");
  s.assertTrue(banner.isDisplayed(), "banner missing");
  s.assertEquals(userCount, 3, "user count wrong");
  s.assertEquals(currency, "USD");
 
  s.assertAll();   // ← required: throws if any of the above failed
}

Data Providers

Data-driven tests in two parts: the provider supplies rows; @Test consumes them.

@DataProvider(name = "credentials")
public Object[][] credentials() {
  return new Object[][] {
    {"admin@test.com",   "Admin123",  true},
    {"viewer@test.com",  "View123",   true},
    {"invalid@test.com", "wrong",     false},
    {"",                 "",          false},
  };
}
 
@Test(dataProvider = "credentials")
public void login(String email, String password, boolean expected) {
  boolean ok = loginPage.login(email, password);
  assertEquals(ok, expected);
}

Iterator-based provider — lazy, large datasets

@DataProvider(name = "userRows")
public Iterator<Object[]> userRows() {
  return Files.lines(Path.of("src/test/resources/users.csv"))
              .skip(1)                                        // header
              .map(line -> line.split(","))
              .map(parts -> new Object[]{ parts[0], parts[1], Boolean.parseBoolean(parts[2]) })
              .iterator();
}

External data — CSV / JSON / DB

Read inside the provider; return rows. Keep CSV / JSON / DB code out of @Test methods so the test reads cleanly.

@DataProvider
public Object[][] usersFromJson() throws IOException {
  ObjectMapper m = new ObjectMapper();
  List<User> users = m.readValue(new File("users.json"), new TypeReference<>(){});
  return users.stream()
              .map(u -> new Object[]{ u.email, u.role })
              .toArray(Object[][]::new);
}

Provider in another class

@Test(dataProvider = "credentials", dataProviderClass = LoginData.class)
public void login(String email, String password, boolean expected) { /* ... */ }

Parallel data providers

@DataProvider(name = "credentials", parallel = true)
public Object[][] credentials() { /* ... */ }

Each row runs on its own thread. Combine with thread-safe WebDriver setup.

Groups

Tag tests so you can pick subsets — smoke, regression, slow, etc.

@Test(groups = {"smoke", "login"})
public void successfulLogin() { /* ... */ }
 
@Test(groups = {"regression", "login"})
public void invalidLogin() { /* ... */ }

Run a group

In testng.xml:

<suite name="Suite">
  <test name="Smoke">
    <groups>
      <run>
        <include name="smoke"/>
        <exclude name="slow"/>
      </run>
    </groups>
    <packages>
      <package name="com.qa.tests"/>
    </packages>
  </test>
</suite>

MetaGroups — groups of groups

<groups>
  <define name="all-fast">
    <include name="smoke"/>
    <include name="login"/>
  </define>
  <run>
    <include name="all-fast"/>
  </run>
</groups>

Maven Surefire group filter

mvn test -Dgroups=smoke
mvn test -Dgroups=smoke -DexcludedGroups=slow

Test Dependencies

Hard dependency — skip on failure

@Test
public void login() { /* ... */ }
 
@Test(dependsOnMethods = {"login"})
public void openDashboard() { /* skipped if login fails */ }

Group dependency

@Test(groups = "setup")
public void seedDatabase() { /* ... */ }
 
@Test(dependsOnGroups = {"setup"})
public void runQueries() { /* ... */ }

Always-run

@Test(dependsOnMethods = {"login"}, alwaysRun = true)
public void cleanup() {
  // runs even if login fails
}

Parallel Execution

Configure parallelism in testng.xml:

<suite name="Suite" parallel="methods" thread-count="4">
  <test name="RegressionTests">
    <classes>
      <class name="com.qa.tests.LoginTest"/>
      <class name="com.qa.tests.DashboardTest"/>
    </classes>
  </test>
</suite>

`parallel=`	What runs in parallel
`methods`	Each `@Test` method gets its own thread
`tests`	Each `<test>` block in the XML
`classes`	Each test class
`instances`	Each `@Factory`-generated instance
`false`	Sequential (default)

Thread-pool per test method

@Test(threadPoolSize = 3, invocationCount = 10, timeOut = 10_000)
public void hammerEndpoint() { /* ... */ }

Runs the test 10 times across 3 threads in parallel.

Thread-safe driver setup

When tests run in parallel, a single static WebDriver will collide. Use ThreadLocal:

public class DriverFactory {
  private static final ThreadLocal<WebDriver> driver = new ThreadLocal<>();
 
  public static void set(WebDriver d) { driver.set(d); }
  public static WebDriver get()       { return driver.get(); }
  public static void remove()         { driver.remove(); }
}
 
public class BaseTest {
  @BeforeMethod
  public void setUp() {
    DriverFactory.set(new ChromeDriver());
  }
 
  @AfterMethod(alwaysRun = true)
  public void tearDown() {
    DriverFactory.get().quit();
    DriverFactory.remove();
  }
}

Listeners

Listeners hook into the run lifecycle for screenshots-on-failure, retries, custom reports, etc.

`ITestListener`

public class ScreenshotOnFailure implements ITestListener {
  @Override
  public void onTestFailure(ITestResult result) {
    WebDriver driver = DriverFactory.get();
    File png = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
    try {
      Files.copy(png.toPath(),
                 Path.of("target/screenshots/" + result.getName() + ".png"),
                 StandardCopyOption.REPLACE_EXISTING);
    } catch (IOException ignored) {}
  }
}

Retry analyzer

public class RetryAnalyzer implements IRetryAnalyzer {
  private int retries = 0;
  private static final int MAX = 2;
 
  @Override
  public boolean retry(ITestResult result) {
    if (retries < MAX) {
      retries++;
      return true;
    }
    return false;
  }
}
 
@Test(retryAnalyzer = RetryAnalyzer.class)
public void flakyTest() { /* ... */ }

Apply globally via IAnnotationTransformer:

public class RetryTransformer implements IAnnotationTransformer {
  @Override
  public void transform(ITestAnnotation annotation, Class testClass,
                        Constructor testConstructor, Method testMethod) {
    annotation.setRetryAnalyzer(RetryAnalyzer.class);
  }
}

Registering listeners

Per-class:

@Listeners({ ScreenshotOnFailure.class, RetryTransformer.class })
public class LoginTest { /* ... */ }

Globally in testng.xml:

<listeners>
  <listener class-name="com.qa.listeners.ScreenshotOnFailure"/>
  <listener class-name="com.qa.listeners.RetryTransformer"/>
</listeners>

Common listener hooks

Interface	Methods
`ISuiteListener`	`onStart(suite)`, `onFinish(suite)`
`ITestListener`	`onTestStart`, `onTestSuccess`, `onTestFailure`, `onTestSkipped`, `onTestFailedButWithinSuccessPercentage`
`IInvokedMethodListener`	`beforeInvocation`, `afterInvocation`
`IReporter`	`generateReport(...)` — write your own report from results
`IRetryAnalyzer`	`retry(result)` — return true to re-run
`IAnnotationTransformer`	Modify annotations at runtime

testng.xml Configuration

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
 
<suite name="Regression" parallel="methods" thread-count="4">
 
  <parameter name="environment" value="staging"/>
 
  <listeners>
    <listener class-name="com.qa.listeners.ScreenshotOnFailure"/>
  </listeners>
 
  <test name="Chrome">
    <parameter name="browser" value="chrome"/>
    <groups>
      <run>
        <include name="smoke"/>
        <exclude name="wip"/>
      </run>
    </groups>
    <packages>
      <package name="com.qa.tests.web"/>
    </packages>
  </test>
 
  <test name="Firefox">
    <parameter name="browser" value="firefox"/>
    <classes>
      <class name="com.qa.tests.web.LoginTest"/>
      <class name="com.qa.tests.web.CheckoutTest">
        <methods>
          <include name="successfulCheckout"/>
        </methods>
      </class>
    </classes>
  </test>
 
</suite>

Receiving parameters

@Parameters({ "browser", "environment" })
@BeforeMethod
public void setUp(String browser, String environment) {
  driver = DriverFactory.create(browser);
  driver.get(EnvConfig.urlFor(environment));
}
 
// Optional with default
@Parameters("browser")
@BeforeMethod
public void setUp(@Optional("chrome") String browser) { /* ... */ }

TestNG with Selenium Pattern

Putting the pieces together — a parallel-safe Selenium suite.

public abstract class BaseTest {
  protected WebDriver driver() { return DriverFactory.get(); }
 
  @Parameters("browser")
  @BeforeMethod(alwaysRun = true)
  public void setUp(@Optional("chrome") String browser) {
    DriverFactory.set(switch (browser) {
      case "firefox" -> new FirefoxDriver();
      case "safari"  -> new SafariDriver();
      default        -> new ChromeDriver();
    });
    driver().manage().window().maximize();
  }
 
  @AfterMethod(alwaysRun = true)
  public void tearDown() {
    if (DriverFactory.get() != null) {
      DriverFactory.get().quit();
      DriverFactory.remove();
    }
  }
}
 
@Listeners(ScreenshotOnFailure.class)
public class LoginTest extends BaseTest {
 
  @Test(groups = {"smoke", "login"}, retryAnalyzer = RetryAnalyzer.class)
  public void successfulLogin() {
    new LoginPage(driver())
        .open()
        .loginAs("admin@test.com", "Admin123");
    assertTrue(new DashboardPage(driver()).isDisplayed());
  }
 
  @Test(dataProvider = "invalidCreds", groups = {"regression", "login"})
  public void invalidLogin(String email, String pw, String expectedError) {
    new LoginPage(driver()).open().loginAs(email, pw);
    assertEquals(loginPage.errorText(), expectedError);
  }
 
  @DataProvider
  public Object[][] invalidCreds() {
    return new Object[][] {
      {"admin@test.com",  "wrong",   "Invalid email or password"},
      {"missing@x.com",   "any",     "Invalid email or password"},
      {"",                "",        "Email is required"},
    };
  }
}

Run it:

mvn test -DsuiteXmlFile=testng-regression.xml
mvn test -Dgroups=smoke -DexcludedGroups=wip

For richer reports, drop in ExtentReports or Allure — both ship ITestListener adapters that auto-capture step output, screenshots from the listener above, and produce branded HTML.