How would you decide which test suites run on every commit versus nightly?

Two design constraints: how fast devs need feedback, and how expensive the suite is to run.

Per-commit (fast feedback) — typical contents:

• Unit tests (sub-minute, 100% of suite).
• Linters and type-checkers.
• Fast integration tests (against in-memory DB or test containers).
• A small E2E smoke covering the 3-5 most critical user journeys.
• Build + lightweight contract tests if applicable.

Goal: tell the dev "your change is broken" within 10 minutes of pushing. If it takes 45 minutes, devs context-switch, lose flow, and trust in CI erodes.

Nightly (broad coverage) — typical contents:

• Full E2E across browsers and devices.
• Performance load and soak tests.
• Security scans (SAST/DAST), dependency audits.
• Visual regression tests across viewports.
• Cross-version API compatibility tests.

Goal: catch what PR-time can't reasonably check. Failures are triaged in the morning, not blocking PRs.

Decision rules:

1. If the suite catches > X% of regressions per Y unit of runtime, promote to PR. A 30-second smoke that catches 80% of regressions belongs on PR; a 30-min E2E that catches 5% extra belongs nightly.
2. If the suite is flaky, demote it to nightly. PR-time flake erodes trust. Nightly flake just needs morning triage.
3. If infra is expensive (cross-browser farms, perf runners), demote. Pay the cost once a night, not 100x a day.

Tag-driven setup: tag tests with @pr, @nightly, @critical. CI picks the set by tag. New tests start nightly; promote to PR after they prove stable.

The honest split is data-driven. Track which tests catch real bugs and which never do. Tests that never catch anything in 6 months should be deleted, not promoted.