Generating Test Reports

A failed assertion in your terminal is enough when you're iterating locally. Once you push the suite into CI, the audience widens — managers want a green/red dashboard, developers want a one-click view of what broke, ops want machine-readable XML to feed into Jenkins. pytest produces all three from the same run: HTML for humans, Allure for rich interactive dashboards, and JUnit XML for CI integrations. This lesson covers each, plus the configuration patterns that hide the flag soup behind a single pytest command.

What you get out of the box

A bare pytest already prints a usable summary:

============================= test session starts ==============================
collected 12 items

tests/test_login.py ......                                              [ 50%]
tests/test_users.py ...F.F                                              [100%]

=================================== FAILURES ===================================
... full tracebacks ...

=========================== short test summary info ============================
FAILED tests/test_users.py::test_user_role_admin
FAILED tests/test_users.py::test_user_email_lowercase
========================= 2 failed, 10 passed in 1.24s =========================

pytest -v adds one line per test (passed or failed). pytest --tb=short shortens tracebacks; --tb=line reduces each failure to one line; --tb=no hides them entirely. For local dev that's plenty.

For a CI run, you want the result captured to a file. Three formats cover the cases you'll meet.

HTML reports — pytest-html

Install once:

pip install pytest-html

Generate a report:

pytest --html=report.html --self-contained-html

Output: a single report.html you can open in any browser. --self-contained-html inlines the CSS so the file works as an email attachment or a CI artefact without external assets.

The report shows pass/fail per test, the full traceback for each failure, and any captured stdout. It's the simplest "one file per CI run" option — easy to attach to a build artefact, easy to send around.

For richer reports, fall through to Allure.

Allure reports — interactive, history-aware

Allure is the de-facto rich reporter for test runs. It ingests results from a directory and renders a browsable dashboard with steps, attachments, history, and trends. Most QA teams that publish reports to a portal use Allure.

Install the pytest plugin and the Allure CLI:

pip install allure-pytest
brew install allure                   # macOS — or download from allure.qameta.io

Run:

pytest --alluredir=allure-results
allure serve allure-results           # opens a local browser view

The first command writes per-test JSON files into allure-results/. The second processes them and serves a dashboard.

In CI you'd usually call allure generate allure-results -o allure-report --clean instead of serve, and publish the resulting allure-report/ folder as a static site (GitHub Pages, S3, the Allure server).

You can attach extra context to each test from your code:

import allure
 
@allure.feature("Authentication")
@allure.story("Valid login redirects to dashboard")
@allure.severity(allure.severity_level.CRITICAL)
def test_valid_login(api_client):
    with allure.step("submit credentials"):
        response = api_client.login("alice@test.com", "...")
    with allure.step("expect dashboard URL"):
        assert response["redirect"].endswith("/dashboard")

Each @allure.feature / @allure.story / @allure.severity shows up as a tag in the dashboard. Each with allure.step(...) becomes a collapsible step in the failure view — invaluable when a long test fails on one line and you want to see how far it got.

JUnit XML — for Jenkins, GitHub Actions, GitLab CI

Every CI system in existence understands the JUnit XML format. pytest produces it natively:

pytest --junitxml=results.xml

Output: a results.xml with one <testcase> element per test plus pass/fail/skip and message. CI systems parse it into their own dashboards — the GitHub Actions "Test results" tab, GitLab's pipeline view, Jenkins's per-build trend graphs.

A typical CI workflow:

# .github/workflows/tests.yml
- run: venv/bin/pytest --junitxml=results.xml --html=report.html --self-contained-html
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: test-results
    path: |
      results.xml
      report.html

The if: always() ensures the artefacts are uploaded even when tests failed — exactly when you most want them.

Adding context to reports — autouse fixtures

Use an autouse=True fixture (lesson 2) to log information every test should record. The captured output appears in pytest's terminal output, the HTML report, and Allure:

import pytest
 
@pytest.fixture(autouse=True)
def log_test_info(request):
    print(f"\n--- start: {request.node.name} ---")
    yield
    print(f"--- end:   {request.node.name} ---")

request.node is the test object — name, nodeid, keywords (the markers), originalname (without parametrize ID).

For richer logging, write to a file or call out to your team's structured logger. Don't over-instrument — every print is content the next reader has to skim.

Screenshots on failure — Playwright + pytest

The pattern that makes a UI test suite usable: take a screenshot whenever a test fails, save it next to the report. The standard recipe uses a request.node.rep_call attribute pytest exposes via a hook in conftest.py:

# conftest.py
import pytest
 
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    rep = outcome.get_result()
    setattr(item, f"rep_{rep.when}", rep)
 
 
@pytest.fixture(autouse=True)
def capture_on_failure(request, page):
    yield
    if request.node.rep_call.failed:
        path = f"screenshots/{request.node.name}.png"
        page.screenshot(path=path)
        if "allure" in request.config.pluginmanager.list_name_plugin():
            import allure
            allure.attach.file(path, name="failure", attachment_type=allure.attachment_type.PNG)

That's a lot at once — copy it into a real project, adapt the path, and you'll have screenshots dropping into screenshots/ for every failed test. With allure-pytest enabled, the screenshot is also attached to the failed test in the Allure report.

pytest-playwright (the Playwright plugin) supplies a similar hook out of the box if you'd rather not write the boilerplate yourself.

Configuring once — pyproject.toml

Reaching for the same flags every run is busywork. Move them to pyproject.toml:

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = """
  -v
  --tb=short
  --strict-markers
  --html=report.html
  --self-contained-html
  --junitxml=results.xml
"""
markers = [
    "smoke: smoke subset run on every commit",
    "slow:  slow tests excluded from local runs",
]

Now pytest alone produces both HTML and JUnit reports with verbose output. CI scripts, local runs, and a teammate's first time all behave the same.

--strict-markers is especially worth turning on: it makes pytest fail on an unregistered marker rather than warn. Catches typos like @pytest.mark.smoek immediately.

Showing only what you care about

A few flags that shape the output:

pytest -q                       # quiet — one char per test, fewer headers
pytest --tb=line                # one-line tracebacks
pytest -rfE                     # short summary of fails (f), errors (E)
pytest -rxX                     # also include xfails (x) and unexpected passes (X)
pytest --durations=10           # print the 10 slowest tests at the end
pytest --capture=no             # show prints from inside tests (same as -s)

--durations=10 is genuinely useful — surface the slowest tests so they're visible candidates for the slow marker.

The reporting pipeline, end to end

One run, many outputs. Pick the format(s) that match your audience: terminal for the developer, HTML for a quick share, Allure for a reviewable dashboard, JUnit XML for CI integration. Most teams produce JUnit XML and one human-friendly format simultaneously.

A real-world pyproject.toml

A reasonable starting point for a Playwright + pytest project:

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = """
  -v
  --tb=short
  --strict-markers
  --strict-config
  --html=report.html
  --self-contained-html
  --junitxml=results.xml
  --alluredir=allure-results
  --durations=10
"""
markers = [
    "smoke: critical-path tests run on every commit",
    "slow: skip in local runs (pytest -m 'not slow')",
    "regression: regression-only suite",
]
filterwarnings = [
    "error",                # treat warnings as errors — usually catches real issues
    "ignore::DeprecationWarning",
]

Three reports in one command, strict marker checking, top-10 slowest tests printed at the end, deprecation warnings suppressed (a deliberate choice, document why). pytest alone now produces everything — no per-developer flag memorisation.

⚠️ Common mistakes

• Reports that aren't uploaded by CI. Generating report.html is half the job; the other half is configuring your CI to publish it as an artefact. Without that, you're left re-running the failure locally to see what happened. Always attach the generated files (and --alluredir/) as build artefacts, especially on failure.
• Allure CLI version skew. allure-pytest (the plugin) and allure (the CLI) are different tools that need to be compatible. After upgrading either, regenerate a small report and confirm it renders. Pin the plugin version in requirements.txt.
• Over-instrumenting with autouse logging. Every print and step decorator adds bytes to your reports — pleasant in moderation, painful when every test contributes pages of context. Keep autouse logging to a few crisp lines and let interesting steps be opt-in.

🎯 Practice task

Wire up reports for a tiny suite. 25-30 minutes.

1. Activate your venv and pip install pytest pytest-html allure-pytest.
2. Create a small project with three test files (3-5 trivial tests each — assert 1 + 1 == 2, etc.).
3. Run pytest --html=report.html --self-contained-html. Open report.html and confirm the dashboard renders. Make one test fail and re-run — confirm the failure shows the rewritten diff.
4. Run pytest --junitxml=results.xml. Open results.xml and find your test names inside <testcase> elements.
5. Run pytest --alluredir=allure-results. If you have the Allure CLI installed, run allure serve allure-results to view the report. (If you don't have Allure CLI handy, that's fine — the JSON files in allure-results/ are the artefact a CI job would publish.)
6. Add at least one @allure.story("...") and one with allure.step("..."): to one of your tests. Re-run and confirm the step appears in the dashboard.
7. Move every flag into a [tool.pytest.ini_options] block in pyproject.toml. Confirm bare pytest produces the same outputs.
8. Add a --durations=5 to the config and read the slowest tests printed at the end of the run. Add a time.sleep(0.5) to one test to confirm it surfaces.
9. Stretch: add an autouse=True fixture that prints start: <name> and end: <name>. Run with pytest -v -s and verify the lines appear. In Allure, attach a small text body via allure.attach("hello world", name="debug-info") from inside a test. Confirm the attachment appears in the report.

You now have the full pytest toolkit — discovery, fixtures, parametrize, assertions, reports — and that's everything you need to build the test suite for the capstone in chapter 8.