Q23 of 26 · Mobile QA
How do you decide which tests run on emulators versus real devices in a build pipeline?
Short answer
Short answer: Emulators for fast PR-gate feedback on core functional flows — they start in seconds and parallelise freely. Real devices for hardware-dependent features, OEM-specific behaviour, performance measurement, and pre-release confidence. Nightly suites run on real devices; PR gates run on emulators.
Detail
The decision framework is based on two axes: speed requirement and hardware dependency.
Always run on real devices: camera and microphone usage, biometric authentication (Face ID, fingerprint), NFC, Bluetooth, GPS with real signal, haptic feedback, OEM-specific UI overrides (Samsung font substitution, Xiaomi process management), battery and thermal behaviour, and any test that must verify behaviour on a specific commercial device for legal or compliance reasons.
Appropriate for emulators: functional UI flows that don't use hardware sensors, API integration tests, navigation and routing, form validation, offline mode (network toggled via ADB), and accessibility checks. On the Android emulator, all of these work identically to a real device for functional purposes.
Pipeline tiering:
- PR gate (every push): emulator suite on a single Pixel emulator, targeting your minimum supported API level. Runs in under 10 minutes. Blocks merge.
- Post-merge nightly: real device matrix (top 5 device/OS combos), full regression suite. Runs in 30-60 minutes. Notifies the team of failures overnight.
- Pre-release: real device matrix, full suite plus manual exploratory on the key devices. Run 24-48 hours before submission.
The cost model drives this: emulators are free and parallelise to infinity; real device slots have a per-minute cost and a concurrency limit.