What I learned from a failed regression cycle

This is a case study, details blurred, about a regression cycle that did everything by the book and still missed the bug that mattered. It's the most useful kind of failure to study, because nobody was careless. The process ran exactly as designed, and the design was the problem.

Context

A mature product with a large manual regression suite — a couple of hundred test cases run before each major release. The release in question touched the search feature. We ran the full regression pass, two days of work, every case executed, everything green. We shipped.

Symptoms

Within hours, support lit up: search was returning wrong results for a common query pattern — exactly the kind of thing half the user base does every day. It was an obvious, high-traffic regression, and we'd just spent two days "testing" and declared the build clean.

Investigation

The post-mortem was uncomfortable, because the regression suite had search cases and they'd all passed. The problem was what they covered. The suite had been built years earlier and grown by accretion: it tested the search scenarios that mattered when it was written, plus whatever got bolted on after old bugs. It had deep coverage of features barely anyone used and near-zero coverage of the query pattern that had since become the most common one. The cases were green because the cases were stale — they tested an old picture of how the product was used.

Worse, the pass had been a box-ticking march: execute every case, mark it green, move on. Two days of effort spent re-confirming low-risk behaviour while the actual change — the search modification — got the same shallow attention as everything else.

Root cause

Two failures compounding:

• The suite hadn't tracked reality. Test coverage reflected the product and usage of years ago, not today. No one had asked "are we still testing what users actually do?" — so effort was spread evenly across cases of wildly unequal importance.
• The cycle ignored the change. A regression pass should weight toward what was modified and what's risky. Ours weighted toward what was already written down. The search change — the whole point of the release — was never tested any harder than a dormant settings screen.

This is what happens with no risk-based prioritisation: you confuse "executed every case" with "tested the right things," and a green pass rate hides a coverage gap aimed straight at production.

What the tests missed

The tests missed the change. Two hundred green cases, and not one exercised the new behaviour against the most common real-world input. The suite measured itself — did the documented cases still pass — instead of measuring the release — is the thing we just changed safe. A pass rate on a stale suite is a confident answer to the wrong question, which is precisely why readiness is more than passed cases.

The reusable lesson

We changed two things. First, every regression cycle now starts from the diff: what changed, what's risky, what's high-traffic — and that gets disproportionate, fresh testing, including exploratory work, not just the canned cases. Second, the suite gets pruned and re-pointed, not just appended to: dead cases for unused features come out, real high-traffic patterns go in, so the coverage tracks how the product is actually used. A regression suite is a living thing; left untended it slowly tests a product that no longer exists. Running every case is not the goal — testing the right things is, and "every case green" can be the most dangerous sentence in the report.

Regression-cycle case lessons

• Start every regression cycle from the diff: weight effort toward what changed and what's high-traffic
• "Executed every case" is not "tested the right things" — a stale suite gives a confident wrong answer
• Prune dead cases for unused features; add cases for how users actually use the product now
• Add fresh exploratory testing around the change — canned cases miss new behaviour by design
• A green pass rate on a stale suite hides a coverage gap pointed straight at production
• Treat the suite as living: re-point it to current usage, don't just append after each release