Matrix Builds for Cross-Browser and Multi-Version Testing

Running your Playwright suite against Chrome takes 8 minutes. Running it against Chrome, Firefox, and WebKit serially takes 24 minutes. Running all three in parallel takes 8 minutes — the same as one browser. That's what matrix builds do: they fan a single job out into N parallel jobs, one per combination of parameters. For QA engineers testing across browsers, operating systems, or framework versions, it's the single most impactful CI configuration change you can make.

The strategy.matrix block

A matrix is a set of variables that GitHub Actions uses to create parallel job instances:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        browser: [chromium, firefox, webkit]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npx playwright install --with-deps ${{ matrix.browser }}
      - run: npx playwright test --project=${{ matrix.browser }} --reporter=html
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: report-${{ matrix.browser }}
          path: playwright-report/
          retention-days: 7

This creates three parallel jobs: test (chromium), test (firefox), test (webkit). Each runs identically except for the value of ${{ matrix.browser }}. The PR status check shows all three; all three must pass for the check to be green.

fail-fast: false — the most important setting

By default, fail-fast is true: if one matrix job fails, GitHub cancels all the others immediately. For cross-browser testing, this is almost always wrong. If Chrome fails, you still want to know whether Firefox passed — a Firefox-specific bug would be hidden by the early cancellation. Set fail-fast: false on every cross-browser matrix.

The only time you want fail-fast: true is when the jobs are truly redundant — for example, a smoke test that runs sequentially-dependent steps and you want to abort the moment any step fails.

Multi-version Java/Node

strategy:
  fail-fast: false
  matrix:
    java: ['17', '21']
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          java-version: ${{ matrix.java }}
          distribution: 'temurin'
          cache: 'maven'
      - run: mvn clean test -DsuiteFile=smoke.xml -Dheadless=true -B

This runs your Selenium/TestNG suite against Java 17 and Java 21 in parallel — useful when your team is mid-migration between Java versions and you want to confirm compatibility.

Cross-browser AND environment: a 2D matrix

Matrix variables multiply:

strategy:
  fail-fast: false
  matrix:
    browser: [chrome, firefox]
    environment: [staging, production]

This creates four jobs: chrome+staging, chrome+production, firefox+staging, firefox+production. The ${{ matrix.environment }} value feeds into secrets.STAGING_URL or secrets.PROD_URL via an env: block.

Use 2D matrices carefully — they scale fast. 3 browsers × 2 environments × 2 OS = 12 parallel jobs. Each job costs minutes from your GitHub Actions budget.

include and exclude for fine control

Some combinations don't make sense. Safari only runs on macOS. Firefox on Windows has different quirks than Firefox on Ubuntu. Use exclude to remove invalid combinations:

strategy:
  matrix:
    browser: [chrome, firefox, safari]
    os: [ubuntu-latest, macos-latest]
    exclude:
      - browser: safari
        os: ubuntu-latest    # Safari is not available on Linux
  fail-fast: false
 
jobs:
  test:
    runs-on: ${{ matrix.os }}

Use include to add specific combinations with extra properties not in the main grid:

matrix:
  browser: [chrome, firefox]
  include:
    - browser: edge
      os: windows-latest    # Edge gets a specific OS not in the matrix

Speed impact: serial vs parallel

A concrete example from a real project:

The total compute time is the same in both cases — parallel execution uses 3× the machines simultaneously. You pay in GitHub Actions minutes either way, but developers get feedback 3× faster.

Naming artifacts in a matrix

Each parallel job must upload to a unique artifact name. Use the matrix variable in the name:

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: report-${{ matrix.browser }}-${{ matrix.environment }}
    path: playwright-report/

Without this, matrix jobs overwrite each other's artifacts — you get only the last one uploaded.

⚠️ Common mistakes

• Leaving fail-fast: true (the default) on cross-browser matrices. A Chrome failure cancels Firefox and WebKit. You miss Firefox-specific bugs. Always add fail-fast: false to independent cross-browser jobs.
• Building a 3 × 3 × 2 matrix for smoke tests. An 18-job matrix for a 5-minute smoke suite is overkill. Full cross-browser matrices belong on nightly regression, not on every PR. On PRs, smoke against one browser fast; run the matrix nightly.
• Duplicate artifact names across matrix jobs. Without ${{ matrix.browser }} in the artifact name, the second job to finish overwrites the first job's report. Always include at least one matrix variable in artifact names.

🎯 Practice task

Convert a single-browser workflow to a matrix — 30 minutes.

1. Take the PR workflow you created in the previous lesson (or the template for your framework).
2. Add a strategy.matrix block with at least two browsers or two Java/Node versions.
3. Add fail-fast: false.
4. Update the test command to reference ${{ matrix.browser }} or ${{ matrix.java }}.
5. Update the artifact upload to use the matrix variable in the name.
6. Push and open a PR. Confirm both matrix jobs appear as separate status checks. Confirm fail-fast: false works by making one browser fail intentionally — verify the other browser job still completes.
7. Stretch: create a nightly workflow (on: schedule: - cron: '0 3 * * *') that runs a full cross-browser matrix against your smoke suite. Keep the PR workflow to a single browser for speed.

The next lesson covers secrets, environment variables, and artifacts — the three mechanisms for getting sensitive data into your workflow and getting results out of it.