Q20 of 38 · Test design
How do you design tests for race conditions in a concurrent system?
Short answer
Short answer: Combine deterministic concurrency tests (run the same scenario many times in a tight loop with controlled timing), property-based fuzzing (random schedules), stress tests (artificial contention), and explicit invariant checks (no double-charge, no negative balance) that hold regardless of interleaving.
Detail
Race conditions are bugs that manifest only in specific interleavings of concurrent operations. They're notoriously hard to catch because they're non-deterministic, depend on timing, and often involve shared state across requests, threads, or services.
Layered test design:
Stress / load tests with concurrency. Run the suspect operation N times in parallel, then check invariants. Tools: k6, JMeter, custom thread/coroutine pools. Focus on operations with shared state: balance updates, inventory decrement, idempotent endpoints, distributed locks.
Property-based concurrency testing. Tools like jepsen.io (databases), Hypothesis with state-machine testing (general logic), or stateright (model-checked Rust) generate random interleavings and check invariants. The closest you get to "exhaustive" race testing.
Deterministic schedules where possible. For tightly scoped concurrent code, use thread-control primitives: CountDownLatch in Java, asyncio.gather with synchronisation, or pause-and-step debugger-style controls in test frameworks. Construct the specific interleaving you suspect is buggy.
Chaos engineering at the system level. For distributed systems, inject delays, drops, and partitions (LinkedIn's litmus, Netflix's Chaos Monkey). Race conditions in distributed systems often involve network re-ordering, retries, or timing.
Static analysis and contracts. Race detectors (Go's
-race, Java's ThreadSanitizer, Rust's borrow checker) catch some races at the language level.
Test design moves: identify invariants explicitly ("total balance never decreases below zero", "no order is double-charged"); run race tests in a loop (1000 iterations); test idempotency explicitly with the same idempotency key; test under realistic contention (10,000 RPS surfaces races differently than 2 requests).
// EXAMPLE
test_no_double_charge.py
import pytest
from concurrent.futures import ThreadPoolExecutor
from billing import charge
def run_concurrent(fn, threads=100):
with ThreadPoolExecutor(max_workers=threads) as pool:
futures = [pool.submit(fn) for _ in range(threads)]
return [f.result() for f in futures]
def test_idempotent_charge_does_not_double_charge():
user = create_user_with_balance(100)
key = "abc-123"
results = run_concurrent(
lambda: charge(user_id=user.id, amount=10, idempotency_key=key),
threads=100,
)
user.refresh()
assert user.balance == 90, "expected exactly one £10 charge"
successful = [r for r in results if r.success]
assert len({r.charge_id for r in successful}) == 1, \
"expected one logical charge across all retries"