Q20 of 21 · AI for testing
How do you handle AI hallucinations in test-generation output?
Short answer
Short answer: Treat all AI-generated test output as a draft that requires verification against the real system. Hallucinations in tests manifest as assertions on values the system doesn't produce, selectors that don't exist, or API field names invented from plausible conventions rather than the actual spec.
Detail
AI hallucination in test generation is not exotic — it's the default risk. A model asked to write a test for an endpoint it hasn't seen will invent plausible but wrong response field names, status codes that don't exist, or parameter names based on conventions rather than the actual contract.
Three common hallucination patterns:
Selector hallucination: the model generates getByTestId("submit-button") when the real attribute is data-cy="login-submit". The test runs; the selector fails; you debug.
Value hallucination: the model asserts expect(response.userId).toBeDefined() when the real field is response.user.id. If the path resolves to undefined, the toBeDefined() assertion may pass vacuously depending on the test framework.
Behaviour hallucination: the model describes behaviour the API doesn't have — assumes a 201 response when the endpoint returns 200, or assumes a body when the endpoint returns empty.
Mitigation: verify generated tests against the source of truth — the live API spec, the running application, or the source code. Running the test once and seeing it pass is not sufficient; an assertion on an unexpected undefined path can pass silently.