Performance Test Data Strategy

Plan a safe, realistic test data strategy for a performance test run — covering data requirements, synthetic data generation, uniqueness, cleanup, sensitive-data handling, and refresh approach.

Role

Performance QA

Difficulty

Intermediate

Time limit

90 min

Scenario

Your team is about to run a 500-VU load test against the ShopFlow e-commerce platform. The test covers three journeys: user registration, product search, and checkout. The pre-production environment shares a database with the integration test suite, and the product team has asked you to ensure no test run can corrupt live-like data, create orphaned records, or trigger real email notifications. You must produce a written test data strategy before the first test execution.

Requirements

1.Identify the data requirements for each of the three journeys: what records must exist before the test starts, what records are created during the test, and what records are read-only
2.Design a synthetic data plan for the user pool: how many unique accounts are needed, the naming/email convention to make test accounts identifiable, and how to generate them at scale
3.Define a CSV or data-generation strategy for parameterising VUs with unique credentials — explain the JMeter CSV Data Set Config or k6 SharedArray approach and why uniqueness matters
4.Specify a cleanup strategy: what records must be deleted after each run, what can be retained across runs, and how orphaned records (e.g. incomplete orders) are handled
5.Define sensitive-data rules: what types of data must never appear in test scripts, results files, or CI logs — and how to substitute realistic-but-fake values
6.Describe a data refresh approach: how to reset the environment to a known state before each major test run, and how frequently a full refresh is needed vs. an incremental top-up
7.Identify at least two data-related risks that could invalidate test results (e.g. VU collision on shared accounts, email quota exhaustion) and propose mitigations

Starter data

›System under test: ShopFlow — user registration (/register), product search (/search), guest and registered checkout (/checkout)
›Environment: pre-production DB shared with integration test suite; email notifications are live (connected to SendGrid sandbox); payment gateway is in Stripe test mode
›Known constraints: SendGrid sandbox has a 100-email/day cap; Stripe test mode has no rate limit but does log all events; the product catalogue has 50,000 items and is read-only for performance tests
›Current state: no test data strategy exists; the team has been reusing a set of 10 shared test accounts, causing intermittent 'account already in use' failures during concurrent runs
›Tooling available: JMeter CSV Data Set Config, k6 SharedArray, a Python script that can call the /register API to bulk-create accounts, and a direct DB connection for cleanup

Expected deliverables

✓A data requirements matrix: journey name, data category (pre-existing, created during run, read-only), record type, and quantity needed for 500 VUs
✓A synthetic user pool plan: account count (justify the number), email convention (e.g. loadtest+{n}@example.com), password strategy, and the generation method (API-based seeding vs. direct DB insert)
✓A CSV or SharedArray strategy: file structure, variable names, recycle/stop-on-EOF settings, and how uniqueness is guaranteed across threads
✓A cleanup runbook: which tables/records to clean post-run, the cleanup method (API calls, SQL deletes, or a teardown script), and the order of operations to avoid FK constraint errors
✓A sensitive-data rules list: minimum three data categories that must not appear in scripts or logs, with the substitution approach for each
✓A data refresh plan: trigger conditions for a full refresh vs. incremental top-up, estimated time for each, and the team member responsible
✓A risk register with at least two data-related risks, their likelihood, impact on test validity, and mitigations

Evaluation rubric

Dimension	What reviewers look for
Data realism	Does the synthetic data plan produce records that exercise the application the same way real data would? A user pool of 10 accounts shared across 500 VUs will cause lock contention or session conflicts — the plan must size the pool at or above the peak VU count. Product search queries should cover a realistic distribution of search terms, not a single repeated keyword that would always be cached.
Uniqueness and collision avoidance	Are user accounts unique per VU per run? Does the CSV or SharedArray ensure no two concurrent VUs share the same credentials? Does the plan account for overlapping runs (nightly load test vs. an ad-hoc run) that could compete for the same account pool? A good strategy allocates accounts in ranges or uses a run-ID prefix to namespace each execution.
Sensitive-data handling	Does the plan explicitly identify which data categories are sensitive (real names, email addresses, payment card numbers, PII) and substitute them with realistic-but-fake equivalents (Faker-generated names, loadtest+n@example.com addresses, Stripe test card 4242424242424242)? Does it state that Stripe test-mode tokens and SendGrid sandbox keys must never appear in script files committed to version control?
Cleanup and refresh discipline	Is the cleanup order correct — child records before parent records to avoid FK violations (e.g. delete order_items before orders before users)? Is the cleanup triggered automatically at the end of each run or manually? Does the refresh plan specify a testable 'known good state' so the team can verify the environment is ready before the next run?
Risk awareness	Are the identified risks specific to this system and test design — not generic boilerplate? The best risk entries explain the failure mode: 'if two VUs register with the same email simultaneously, the second registration returns 422 and the VU's session is invalid for the rest of the run, artificially inflating error rate by up to 10%'. Mitigations should be actionable (pre-generate accounts before run, not 'be careful with data').

Sample solution outline

›Data requirements matrix: registration journey — 500 unique email/password pairs (created during run, must not collide); search journey — uses product catalogue (read-only, no test data needed); checkout journey — 500 unique registered accounts + 50 guest sessions (created during run); shared pre-existing data — product catalogue (50k items, read-only)
›Synthetic user pool: generate 600 accounts before the run (20% buffer above VU count) using the /register API with bulk-create script; convention: loadtest+{uuid}@shopflow-test.com; password: LoadTest!{uuid}; store in users.csv with columns email,password
›CSV strategy: JMeter — CSV Data Set Config, filename users.csv, variables email,password, delimiter comma, recycle=false, stop_on_EOF=false, sharing mode 'Current thread' (prevents two threads reading the same row); k6 — SharedArray loaded once, each VU picks row index by VU_ID % array.length
›Cleanup runbook: (1) DELETE FROM order_items WHERE created_at > run_start; (2) DELETE FROM orders WHERE created_at > run_start; (3) DELETE FROM user_sessions WHERE user_id IN (SELECT id FROM users WHERE email LIKE 'loadtest+%'); (4) DELETE FROM users WHERE email LIKE 'loadtest+%'; run as post-test teardown script, not manually
›Sensitive-data rules: (1) Real customer email addresses — substitute with loadtest+n@shopflow-test.com; (2) Real payment card numbers — use Stripe test cards only (4242424242424242); (3) SendGrid production API key — use SENDGRID_TEST_KEY env var, never commit to .jmx or .js files; (4) Production DB credentials — use a separate read/write test DB user with INSERT/DELETE only on the loadtest schema
›Data refresh: full refresh (drop and re-seed all test accounts + orders) before each major load test campaign (weekly); incremental top-up (add 100 new accounts, delete run-specific orders) before each daily smoke run; refresh takes ~5 min; ownership: performance QA engineer on-call
›Risk register: (1) VU collision on shared accounts — likelihood high if pool < VU count, impact: inflated error rate masks real performance issues — mitigation: always size pool ≥ 1.2× peak VUs; (2) SendGrid email cap exhaustion — 500 VU registration × 1 confirmation email = 500 emails, exceeds 100/day sandbox cap — mitigation: disable email notifications in pre-prod via feature flag before load test, verify flag is off before run

Common mistakes

Reusing the same 10 test accounts across 500 concurrent VUs — causes lock contention, session conflicts, and 'account already in active session' errors that inflate error rate without indicating a real performance problem
Generating test data inside the k6 or JMeter script at runtime (e.g. POST /register as the first step of every VU) — this mixes data setup with the actual test, inflates the registration endpoint's load count, and creates thousands of orphaned accounts if the run is aborted midway
Not specifying the cleanup order — deleting users before orders causes FK constraint violations; always clean child records first
Using real personal data (names, emails scraped from the production DB) in a load test — a results file or log with real email addresses is a PII leak, especially if the file is archived in CI
Setting CSV recycle to true without checking account collision — if the CSV has 100 rows but 500 VUs cycle through them, multiple VUs will log in with the same account simultaneously
Treating the data refresh as a manual step with no documented trigger — without a trigger condition ('run full refresh before every campaign kick-off, not just when it feels stale') the environment drifts and results become incomparable across runs

Submission checklist

Data requirements matrix covering all three journeys with record types and quantities
Synthetic user pool plan: count justified, email convention, generation method
CSV or SharedArray strategy with recycle/stop settings and uniqueness guarantee
Cleanup runbook with correct FK-safe deletion order and trigger condition
At least three sensitive-data categories identified with substitution approach
Data refresh plan: trigger conditions for full vs. incremental refresh, time estimate
Risk register with at least two data-specific risks and actionable mitigations
No hard-coded production credentials, real email addresses, or real card numbers anywhere in the deliverable

Extension ideas

+Write a Python or Node.js seed script that calls /register in batches of 50 concurrent requests and writes the resulting credentials to users.csv — making the pool generation reproducible and version-controllable
+Add a pre-test health check step to the load test script that verifies the account pool is large enough for the configured VU count before the main test starts — fail fast if the pool is undersized
+Design a data versioning approach: prefix each run's test accounts with a run ID (loadtest+run42+{n}@example.com) so multiple historical runs' data can coexist in the DB and be cleaned up independently