Running Tests on Every Pull Request

A test suite that only runs on your laptop isn't a quality gate — it's a personal safety net. The first thing a CI pipeline gives your team is tests that run automatically on every pull request, before any reviewer approves it and before any code merges. This lesson shows you a complete, production-ready PR test workflow for each of the major QA automation frameworks, with the patterns that make it reliable in CI.

The patterns that make PR testing work

Before the full examples, four patterns appear in every reliable CI test setup:

Headless mode. CI runners have no display. Browsers need to run in headless mode — rendering pages to memory rather than a screen. Forget this and your Selenium or Playwright job hangs indefinitely waiting for a display that doesn't exist.

if: always() for reports. By default, GitHub skips a step if a previous step failed. If your tests fail, you need the report upload step to still run — otherwise you can't see why they failed. if: always() overrides this.

Dependency caching. Without caching, every CI run re-downloads Maven dependencies, npm packages, or Python packages from scratch. Adding cache: 'maven' or cache: 'npm' to setup actions cuts this from 60–90 seconds to 2–5 seconds on cache hits.

timeout-minutes. A hung browser, a test waiting for a server that never starts, or a deadlock in a parallel runner — all of these will leave a job running until GitHub kills it at 6 hours if you don't set your own limit. Set it to 2× your expected runtime.

Complete workflow: Selenium + TestNG + Maven

# .github/workflows/selenium-tests.yml
name: Selenium Tests
 
on:
  pull_request:
    branches: [main]
 
jobs:
  selenium:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
 
      - name: Set up Java 21
        uses: actions/setup-java@v4
        with:
          java-version: '21'
          distribution: 'temurin'
          cache: 'maven'            # caches ~/.m2 between runs
 
      - name: Run smoke tests
        run: |
          mvn clean test \
            -DsuiteFile=smoke.xml \
            -Dheadless=true \
            -B
        env:
          BASE_URL: ${{ secrets.STAGING_URL }}
 
      - name: Upload Surefire reports
        if: always()                # upload even when tests fail
        uses: actions/upload-artifact@v4
        with:
          name: surefire-reports
          path: target/surefire-reports/
          retention-days: 7
 
      - name: Upload failure screenshots
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: screenshots
          path: target/screenshots/
          retention-days: 14

The -Dheadless=true system property works because your WebDriver setup reads it:

ChromeOptions opts = new ChromeOptions();
if (Boolean.parseBoolean(System.getProperty("headless", "false"))) {
    opts.addArguments("--headless=new", "--no-sandbox", "--disable-dev-shm-usage");
}

The --no-sandbox and --disable-dev-shm-usage flags are required on Linux CI runners — without them Chrome crashes on startup due to sandboxing restrictions.

Complete workflow: Playwright + TypeScript

# .github/workflows/playwright-tests.yml
name: Playwright Tests
 
on:
  pull_request:
    branches: [main]
 
jobs:
  playwright:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
 
      - name: Install dependencies
        run: npm ci
 
      - name: Install Playwright browsers
        run: npx playwright install --with-deps chromium
 
      - name: Run smoke tests
        run: npx playwright test --grep @smoke --reporter=html
 
      - name: Upload Playwright report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 7

--with-deps chromium installs the browser and all its OS-level dependencies in one step. Install only the browsers you actually run in this workflow — installing all three browsers adds ~400MB and 2 minutes unnecessarily.

Complete workflow: Cypress

# .github/workflows/cypress-tests.yml
name: Cypress Tests
 
on:
  pull_request:
    branches: [main]
 
jobs:
  cypress:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
 
      - name: Cypress run
        uses: cypress-io/github-action@v6
        with:
          browser: chrome
          spec: 'cypress/e2e/smoke/**'
        env:
          CYPRESS_BASE_URL: ${{ secrets.STAGING_URL }}
 
      - name: Upload screenshots on failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: cypress-screenshots
          path: cypress/screenshots/
          retention-days: 14
 
      - name: Upload videos
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: cypress-videos
          path: cypress/videos/
          retention-days: 7

The official cypress-io/github-action handles npm ci, caching, and the Cypress binary installation in one step. Use it — don't replicate what it does manually.

Making status checks block merging

Once your workflow runs, the job appears as a status check on every PR. To make it a hard requirement — so a PR literally cannot merge while tests are failing:

1. Go to your repository on GitHub → Settings → Branches
2. Edit or create a branch protection rule for main
3. Enable Require status checks to pass before merging
4. Search for and add your job name (e.g., selenium, playwright, cypress)
5. Optionally enable Require branches to be up to date before merging

From that point, every PR shows either a green check ("Tests passed — ready to merge") or a red X with a link to the failing workflow run. No manual intervention required — CI enforces the standard.

⚠️ Common mistakes

• Forgetting --no-sandbox for Chrome on Linux. On CI runners (containerised Linux), Chrome requires --no-sandbox --disable-dev-shm-usage. Without these flags, the browser crashes immediately and your test reports show zero tests with no useful error.
• Using if: failure() for all artifact uploads. On a green run you still want reports for auditing and trend analysis. Use if: always() for reports and if: failure() only for things that only matter when something goes wrong (like raw screenshots or video recordings you don't want to store every run).
• Not reading the workflow logs after the first failure. GitHub Actions logs are verbose and searchable. The failure message, the exact failed assertion, the line number — they're all there. A QA engineer who files "CI failed" without reading the logs is leaving the most useful debugging information on the table.

🎯 Practice task

Set up a PR test workflow for your own project — 40 minutes.

1. Pick the workflow template that matches your framework (Selenium, Playwright, or Cypress). Copy it into .github/workflows/.
2. Replace the test command with your project's actual command. Confirm it works locally first.
3. Commit and push to a feature branch. Open a pull request. Watch the workflow run.
4. Find the artifact upload in the Actions run. Download the test report and open it locally.
5. Set up branch protection on main to require the status check. Make a failing commit. Confirm the PR cannot be merged until the failure is fixed.
6. Stretch: add a second job to your workflow that runs if: always() and posts a summary of the test results as a PR comment. The dorny/test-reporter@v1 action reads JUnit XML and does this automatically for Selenium/TestNG. Read its documentation and wire it up.

The next lesson adds matrix builds — running the same test suite across multiple browsers or Java versions simultaneously, with each combination as a separate parallel job.