Q26 of 37 · API testing

Walk me through your strategy for catching breaking changes in a public API.

API testingSeniorapibreaking-changesschemaversioningsenior

Short answer

Short answer: Layered: schema diff in CI on every PR; contract tests against a representative consumer suite; deprecation header + sunset checks; an OpenAPI-driven test that fails on any incompatible change. Frame breaking change as a process problem — checks block merges, communications fire automatically, customers get advance notice.

Detail

Public APIs have customers who depend on every quirk. Breaking changes shipped quietly become incidents; the right strategy makes them impossible to ship by accident.

The five-layer defence:

1. Schema diff on every PR.

  • The OpenAPI / GraphQL schema is checked in.
  • A tool (oasdiff, schemathesis, graphql-inspector) compares the PR's schema to main.
  • Any incompatible change (removed field, renamed parameter, narrowed enum) blocks merge unless tagged breaking and approved.
- run: oasdiff breaking spec-main.yaml spec-pr.yaml --fail-on-diff

2. Schema-driven test generation.

  • Tools like Schemathesis generate property-based tests from the OpenAPI schema.
  • They send malformed inputs, edge cases, and confirm responses match the documented schema.
  • Catches "API and docs disagree" — the most common breaking-change vector.

3. Contract tests with representative consumers.

  • Pact contracts from each major consumer (web, iOS, Android, partner SDK).
  • Provider verification in CI fails if the change breaks any consumer's recorded expectations.
  • For public APIs without internal consumers, maintain "synthetic consumer" tests that exercise every documented response shape.

4. Versioned snapshot tests.

  • For each major version (v1, v2), a frozen test suite that asserts on the exact response shape.
  • New versions of the API may change responses; old versions cannot.
  • Run continuously against main; failures mean someone changed v1's behaviour.

5. Deprecation + sunset enforcement.

  • Every deprecated endpoint carries Deprecation and Sunset headers.
  • Tests assert these are set correctly.
  • A separate test fails 30 days before sunset to force review.

Process:

Communicate breaking changes through:

  • Changelog entry on every release, with migration guidance.
  • Emails to API customers using the affected endpoints (telemetry-driven).
  • Deprecation period of 6-12 months for SaaS, longer for partner integrations.
  • Blog post for substantial changes.

SemVer or date-based versioning. /v1, /v2 for major; date-based (?api-version=2026-05-10) for finer granularity. Stripe's date-versioning is the gold-standard reference.

Internal escalation. A breaking change PR with the right tag goes through:

  • Eng manager review.
  • Customer success awareness (any major customer affected?).
  • Optional product / legal sign-off for material changes.

Anti-patterns:

  • Catching breaks in production telemetry only. By then, customers are paged.
  • A single E2E test labelled "the API contract" that someone updates without thought when it breaks. Schema diff is non-negotiable.
  • Deprecating endpoints without sunset dates. They linger forever and the surface area grows.

The senior signal: framing this as a process + automation combo. Tooling alone misses the human side; process alone is forgettable. Both are needed.

// WHAT INTERVIEWERS LOOK FOR

Five-layer defence (schema diff, generated tests, contracts, snapshots, deprecation), process around communication and escalation, and bonus reference to Stripe's date-based versioning.

// COMMON PITFALL

Relying on review-by-eyeballs to catch schema breaks. Reviewers miss things; automation should be the floor, not the ceiling.