What is a flaky test and how do you handle it in CI?

A flaky test is one whose outcome isn't determined by the code under test — it sometimes passes and sometimes fails for the same SHA. Common causes: hard-coded timing or sleeps, shared mutable state between tests, network or third-party dependencies, race conditions in async code, time-zone or date assumptions.

The wrong response is to add an auto-retry that masks flakiness — the test still flaps, the bug it might be catching gets ignored, and trust in CI erodes. The right response is quarantine and triage:

1. Detect: track flake rate per test in CI metadata. A test that fails ≥ 1% of runs on the main branch is flaky by definition.
2. Quarantine: move flaky tests to a separate job that's allowed to fail without blocking PRs, but is tracked. Don't delete them yet — they might be catching real bugs.
3. Own: assign every quarantined test to a person or team with a deadline (e.g. 2 weeks).
4. Fix or delete: if the flake is a test bug, fix it (replace sleeps with explicit waits, isolate state, mock the network). If the flake is a real product bug surfacing intermittently, fix the product. If neither — the test is testing something genuinely unstable — delete it; flake without intent is worse than no test.

Auto-retry is acceptable only as a temporary mitigation while a fix is in flight, never as the permanent answer.