TestNG Annotations, Priority, DataProvider & testng.xml

TestNG Annotations Used in a Project

TestNG is the test framework behind most Java Selenium suites. It decides what runs, in what order, with which data, and how results are reported. This guide covers the annotations and their execution order, priority rules (including the edge cases interviewers love), DataProviders, groups, parameters, dependencies, soft vs hard assertions, testng.xml and listeners.

Annotations and Their Order ⭐

Order:

Suite → Test → Class → Method → test → Method → Class → Test → Suite

Setup opens from the outside in; teardown closes from the inside out.

There are also @BeforeGroups and @AfterGroups, which run around the first and last method of a group.

For a class with two tests, the output order is:

BeforeSuite → BeforeTest → BeforeClass
  → BeforeMethod → testA → AfterMethod
  → BeforeMethod → testB → AfterMethod
→ AfterClass → AfterTest → AfterSuite

A Typical BaseTest

public class BaseTest {
    protected WebDriver driver;

    @BeforeMethod
    public void setUp() {
        driver = new ChromeDriver();
        driver.manage().window().maximize();
        driver.get(ConfigReader.get("baseUrl"));
    }

    @AfterMethod(alwaysRun = true)
    // runs even if setup or the test failed
    public void tearDown(ITestResult result) {
        if (result.getStatus() == ITestResult.FAILURE) {
            // take a screenshot for the report
        }

        if (driver != null) {
            driver.quit();
        }
    }
}

A fresh browser per test using @BeforeMethod keeps tests independent. For parallel runs, use a ThreadLocal driver — see Parallel Execution & Grid.

Priority: Rules and Edge Cases ⭐

public class PriorityDemo {

    @Test(priority = 2)
    public void a_test() {}

    @Test(priority = 3)
    public void c_test() {}

    @Test(priority = 1)
    public void b_test() {}

    @Test
    public void d_test() {}     // no priority → default 0

    @Test(priority = -1)
    public void e_test() {}
}

// Run order:
// e_test (-1) → d_test (0) → b_test (1)
// → a_test (2) → c_test (3)

• Lower values run first. Negative values are allowed.
• The default priority is 0 — methods without a priority run before positive priorities, but after negative ones.
• Same priority → alphabetical by method name.
• Configuration annotations aren't affected — @BeforeMethod still runs before each test, whatever its priority.
• Priority isn't a dependency — if b_test fails, a_test still runs. Use dependsOnMethods for that.

Good frameworks rely on priority very little because independent tests shouldn't care about execution order.

DataProvider ⭐

A DataProvider runs the same test with several data sets. It returns Object[][] — each row is one run, each column one parameter.

@DataProvider(name = "logins")
public Object[][] logins() {
    return new Object[][] {
        {"standard_user",   "secret_sauce", true},
        {"locked_out_user", "secret_sauce", false},
        {"standard_user",   "wrong",        false},
    };
}

@Test(dataProvider = "logins")
public void login(String user, String password, boolean shouldSucceed) {
    // runs three times — once per row
}

• It can also return Iterator<Object[]> — handy for large data read lazily from a file.
• @DataProvider(parallel = true) runs the rows in parallel.
• Keep providers in a separate class and reference them with dataProviderClass = LoginData.class. The method must then be static.
• Feed it from Excel or JSON: Data-Driven Testing with Apache POI.

Groups and Parameters

@Test(groups = {"smoke", "regression"})
public void loginWorks() {}

@Parameters({"browser"})
@BeforeMethod
public void setUp(@Optional("chrome") String browser) {
    /* start the chosen browser */
}

Run one group with:

mvn test -Dgroups=smoke

You can also include or exclude groups in testng.xml.

@Parameters reads values from testng.xml — the usual way to pass a browser or environment per <test> block.

Other Useful @Test Attributes

Dependencies

@Test
public void loginTest() {}

@Test(dependsOnMethods = "loginTest")
public void dashboardTest() {
    // skipped if loginTest fails
}

@Test(dependsOnMethods = {"loginTest", "createPostTest"})
public void verifyPostTest() {}

dependsOnGroups does the same for whole groups.

A dependent test is skipped rather than failed when its dependency fails, which keeps reports honest.

Add alwaysRun = true to run it regardless.

Use dependencies sparingly because they make tests order-dependent and harder to run in parallel.

Hard Assert vs Soft Assert

TestNG has two assertion styles. Older Selenium material often calls them "assert" and "verify."

// Hard assert — stops the test at the first failure
Assert.assertEquals(driver.getTitle(), "Dashboard");

// Soft assert — collects failures and reports them together
SoftAssert soft = new SoftAssert();

soft.assertEquals(
    header.getText(),
    "Welcome, Asha"
);

soft.assertTrue(
    logo.isDisplayed(),
    "Logo missing"
);

soft.assertEquals(
    cartCount.getText(),
    "0"
);

soft.assertAll(); // required

Without assertAll(), the test can pass even if the soft assertions failed.

Use hard asserts when later steps depend on the result. For example, there is no point continuing if login has failed.

Use soft asserts when you want to check several independent things on one page.

testng.xml

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">

<suite name="Regression" parallel="classes" thread-count="3">

  <listeners>
    <listener class-name="com.company.listeners.TestListener"/>
  </listeners>

  <test name="Chrome - smoke">

    <parameter name="browser" value="chrome"/>

    <groups>
      <run>
        <include name="smoke"/>
        <exclude name="wip"/>
      </run>
    </groups>

    <classes>
      <class name="com.company.tests.LoginTest"/>
      <class name="com.company.tests.CartTest"/>
    </classes>

  </test>

</suite>

Key testng.xml Elements

• <suite> — the root element; this is also where parallel and thread-count go.
• <test> — a block of classes with its own parameters and groups, for example, one per browser.
• <classes> / <packages> / <methods> — define what to run, down to individual methods with <include name="…"/>.
• <listeners> — classes that react to test events.

Run it from the IDE or through Maven Surefire with:

mvn test

See Maven for Selenium.

Reports and Listeners

After each run, TestNG writes its built-in reports to test-output/:

• index.html

• emailable-report.html

When run through Maven, XML results are generated in:

target/surefire-reports/

These results can be consumed by Jenkins.

For richer reports such as Extent or Allure and screenshots on failure, write a listener — a class implementing ITestListener — and register it in testng.xml or with @Listeners.

public class TestListener implements ITestListener {

    @Override
    public void onTestStart(ITestResult r) {
        /* create a report entry */
    }

    @Override
    public void onTestSuccess(ITestResult r) {
        /* mark as passed */
    }

    @Override
    public void onTestFailure(ITestResult r) {
        /* attach screenshot and error */
    }

    @Override
    public void onFinish(ITestContext c) {
        /* flush the report */
    }
}

There's no built-in ExtentReporterNG class in TestNG. Extent reporting comes from a listener that you or a reporting library provides.

See how the browser side of each test works step by step in the Selenium WebDriver Visualizer.

From Real Projects

On Canolog and Testsigma our Selenium scripts ran on TestNG: annotations for setup and cleanup, groups for choosing what to run, and parallel and batch execution for the full suite. Once you're comfortable with those features, organising a growing test suite becomes much easier. Learn the exact order in which TestNG runs its annotations — it explains many setup problems.

📚 Official documentation: TestNG documentation

FAQs

What is the order of TestNG annotations?

BeforeSuite → BeforeTest → BeforeClass → BeforeMethod → Test → AfterMethod → AfterClass → AfterTest → AfterSuite.

What is the default priority in TestNG?

The default priority is 0. Methods without a priority run before those with positive priorities and after negative ones. Methods with equal priority generally execute alphabetically by method name.

What does a DataProvider return?

A DataProvider can return Object[][], where each row represents one test execution and each column represents a parameter. It can also return Iterator<Object[]>.

@BeforeTest vs @BeforeMethod?

@BeforeTest runs before each <test> block in testng.xml, while @BeforeMethod runs before every individual @Test method.

What happens when a dependency fails?

The dependent test is skipped when its dependency fails, unless alwaysRun = true is configured where applicable.

Hard assert vs soft assert?

A hard assert stops the test at the first failure. A soft assert collects failures and reports them when assertAll() is called.

How do you run only smoke tests?

Tag the tests with groups = "smoke" and include that group in testng.xml, or run:

mvn test -Dgroups=smoke

Related

• TestNG Cheat Sheet
• TestNG Interview Questions
• The Complete Automation Framework Guide
• TestNG with REST Assured