Generating Test Reports

A test suite that fails silently is almost as useless as no tests at all. Reports are how the team — developers, QA leads, and managers — understands what ran, what broke, and what was skipped. JUnit 5 integrates with three distinct reporting layers: the XML files Surefire writes automatically (read by Jenkins, GitHub Actions, and every major CI tool), the HTML report the Surefire Report Plugin generates, and Allure — the richest option — which produces interactive dashboards with timelines, attachments, and severity labels.

Surefire XML — the CI standard

Every mvn test run writes two files per test class into target/surefire-reports/:

• TEST-com.mycompany.CalculatorTest.xml — standard JUnit XML consumed by CI tools
• com.mycompany.CalculatorTest.txt — plain text summary for human reading

You don't configure anything for this — it happens automatically. Jenkins' JUnit plugin, GitHub Actions' test summary, GitLab's test report integration, and CircleCI's test metadata all parse this XML format. In GitHub Actions:

- name: Run tests
  run: mvn test
 
- name: Publish test results
  uses: EnricoMi/publish-unit-test-result-action@v2
  if: always()  # publish even when tests fail
  with:
    files: target/surefire-reports/TEST-*.xml

The if: always() is important — without it, a test failure stops the workflow before the report is published.

Surefire Report Plugin — basic HTML

The Maven Surefire Report Plugin reads the XML files and generates a static HTML page:

<reporting>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-report-plugin</artifactId>
            <version>3.2.5</version>
        </plugin>
    </plugins>
</reporting>

Generate it:

# Run tests and generate the HTML report in one command
mvn surefire-report:report
 
# Or as part of the Maven site lifecycle
mvn verify site

The report lands at target/site/surefire-report.html. It shows per-class pass/fail counts, test names, durations, and failure messages with stack traces. It is not interactive, but it requires no extra tooling and works offline.

Allure — professional interactive reports

Allure produces the kind of report that product owners can read: test names from @DisplayName, grouped by feature, with screenshots, request/response payloads, and severity ratings. It is the most widely used third-party reporting library for JUnit 5.

Add the dependency:

<dependency>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-junit5</artifactId>
    <version>2.25.0</version>
    <scope>test</scope>
</dependency>

Add the Allure Maven plugin:

<plugin>
    <groupId>io.qameta.allure</groupId>
    <artifactId>allure-maven</artifactId>
    <version>2.12.0</version>
</plugin>

Configure the Allure aspect agent in Surefire (required for @Step annotations):

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.2.5</version>
    <configuration>
        <argLine>-javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/1.9.21/aspectjweaver-1.9.21.jar</argLine>
    </configuration>
</plugin>

Allure annotations in test code

import io.qameta.allure.*;
import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.*;
 
@Feature("User Login")
class LoginTest {
 
    @Test
    @Story("Successful login")
    @Severity(SeverityLevel.CRITICAL)
    @DisplayName("should navigate to dashboard after valid login")
    void shouldLoginSuccessfully(WebDriver driver) {
        openLoginPage(driver);
        enterCredentials(driver, "alice@test.com", "password");
        submitForm(driver);
        assertEquals("Dashboard", driver.getTitle());
    }
 
    @Step("Open login page")
    private void openLoginPage(WebDriver driver) {
        driver.get("https://myapp.com/login");
    }
 
    @Step("Enter credentials: {email}")
    private void enterCredentials(WebDriver driver, String email, String password) {
        driver.findElement(By.id("email")).sendKeys(email);
        driver.findElement(By.id("password")).sendKeys(password);
    }
 
    @Step("Submit the login form")
    private void submitForm(WebDriver driver) {
        driver.findElement(By.cssSelector("[data-testid='submit']")).click();
    }
}

@Feature and @Story group tests in the Allure Behaviors view. @Severity sets a colour-coded priority label. @Step breaks the test body into named steps visible in the Allure timeline — each step shows whether it passed, and for failures, exactly where the test broke.

Generate and serve the report:

mvn test                          # run tests, write allure-results/
allure serve target/allure-results # open browser with live report

Or generate a static HTML site:

allure generate target/allure-results --clean -o target/allure-report
allure open target/allure-report

Custom reporter with TestWatcher extension

For lightweight custom reporting without Allure — for example, a simple JSON summary or a Slack notification — implement the TestWatcher extension from Chapter 4:

import org.junit.jupiter.api.extension.*;
import java.util.*;
import java.io.*;
import java.nio.file.*;
 
public class JsonReporter implements TestWatcher, AfterAllCallback {
 
    private final List<Map<String, String>> results = Collections.synchronizedList(new ArrayList<>());
 
    @Override
    public void testSuccessful(ExtensionContext ctx) {
        results.add(Map.of("name", ctx.getDisplayName(), "status", "PASSED"));
    }
 
    @Override
    public void testFailed(ExtensionContext ctx, Throwable cause) {
        results.add(Map.of(
            "name", ctx.getDisplayName(),
            "status", "FAILED",
            "error", cause.getMessage()
        ));
    }
 
    @Override
    public void afterAll(ExtensionContext ctx) throws Exception {
        String json = buildJson(results);
        Path report = Path.of("target/test-summary.json");
        Files.createDirectories(report.getParent());
        Files.writeString(report, json);
        System.out.println("Report written to " + report.toAbsolutePath());
    }
 
    private String buildJson(List<Map<String, String>> results) {
        // Use Jackson or Gson in a real implementation
        StringBuilder sb = new StringBuilder("[\n");
        for (int i = 0; i < results.size(); i++) {
            var r = results.get(i);
            sb.append(String.format("""
                  { "name": "%s", "status": "%s"%s }""",
                r.get("name"), r.get("status"),
                r.containsKey("error") ? ", \"error\": \"" + r.get("error") + "\"" : ""
            ));
            if (i < results.size() - 1) sb.append(",");
            sb.append("\n");
        }
        return sb.append("]").toString();
    }
}

Register globally via META-INF/services/org.junit.jupiter.api.extension.Extension so it applies to every test class without any per-class annotation.

Reporting pipeline

⚠️ Common mistakes

• Not using if: always() in GitHub Actions. The mvn test step exits with a non-zero code when tests fail, which stops subsequent steps from running. A report-upload step without if: always() therefore never runs when you need it most — when tests fail. Add if: always() to every step that uploads or publishes reports.
• Running allure serve on a CI machine without a display. allure serve opens a browser. In a headless CI environment, use allure generate to produce a static site, then upload the directory as an artifact. GitHub Actions' actions/upload-artifact can store target/allure-report/ and make it downloadable from the run summary.
• Forgetting the AspectJ weaver for @Step. Without the -javaagent in Surefire's <argLine>, @Step annotations are silently ignored — test methods still run, but the Allure report shows no steps. The AspectJ weaver intercepts @Step calls at runtime; without it, Allure only sees the test-level events.

🎯 Practice task

Set up the full reporting pipeline. 25–35 minutes.

1. Run mvn test and open target/surefire-reports/. Find the XML file for one of your test classes. Read the <testcase> elements and note how JUnit 5 display names appear in the XML name attribute.
2. HTML report. Add the Surefire Report Plugin to the <reporting> section. Run mvn surefire-report:report. Open target/site/surefire-report.html. Find the test you intentionally failed earlier.
3. Allure integration. Add allure-junit5 2.25.0 to dependencies. Add @Feature("Login") and @Severity(SeverityLevel.CRITICAL) to one test class. Add @Step(...) to two helper methods. Run mvn test then allure serve target/allure-results. Explore the Behaviors view — confirm @Feature and @Story grouping is visible.
4. CI upload. Create a .github/workflows/test.yml with steps: checkout → setup Java → mvn test → upload target/surefire-reports/ as an artifact (use if: always()). Push and confirm the artifact appears in the Actions run even when a test fails.
5. Stretch — custom reporter. Implement the JsonReporter shown in the lesson. Register it globally via META-INF/services. Run your suite and open target/test-summary.json. Confirm the file contains one entry per test with the correct status.

You have completed Chapter 5. The capstone chapter puts everything together: a real test suite from scratch with unit tests, parameterisation, extensions, Maven configuration, and a reporting pipeline.