🎬 Media & Streaming QA

Model answer

I break startup latency into three measurable phases and instrument them separately so a regression in any one can be attributed without guessing. The first phase is DRM license acquisition: from the moment the player calls requestMediaKeySystemAccess() (or the equivalent EME entry point) to the moment the license is returned and the key is loaded. I measure this by intercepting the license server request in a proxy (Charles or the browser's network panel) and recording the time delta from request sent to response received. On a well-functioning stack this should be under 500 ms; a regression here points to the license server or the entitlement check, not the CDN. The second phase is manifest and first-segment fetch: from the CDN request for the .m3u8 or .mpd to the moment the player has buffered enough of the first segment to begin decoding. I measure this by instrumenting the player's 'buffering' and 'playing' lifecycle events alongside the network waterfall. A cold CDN edge adds latency here; a fully populated edge should be under 1 s on a 4G connection. The third phase is ABR startup decision: how much data the player buffers before it commits to a quality level and emits the first frame. Some players wait for 2+ seconds of buffer before starting render even when segments arrive fast — this is a player-side configuration issue, not a network or DRM issue. I measure this by logging the player's time-to-first-frame event relative to the moment the first segment started downloading. To triage a four-second startup: I add timestamps around each phase boundary in a test harness, replay the scenario against a throttled network profile (e.g. 4G at 15 ms RTT), and identify which phase expanded. If license acquisition is 3 s of the 4 s total, the license server is the culprit — not the CDN. This decomposition is why 'check the network speed' is insufficient: the bottleneck might be the license server on an otherwise fast connection.

Model answer

The failure mode I'm testing has two parts: the technical one (the entitlement API returns a 403 with the right context) and the UX one (the player shows a message the user can act on, not a generic 'playback error'). I test them separately. For the technical part, I mock or seed the entitlement service — or use a test environment that lets me configure per-content region restrictions and window expiry — so I have deterministic fixtures: a content ID that is geo-blocked for France, a content ID whose window closes at a known timestamp. I assert that when the player calls the license acquisition endpoint for the French user on the geo-blocked content, the response is a 403 with a body that includes the reason (geo restriction) and the user's region. I do the same for the post-midnight UK user: the window-expiry fixture should return a 403 with a window-closed reason. I assert these against the actual API response, not just the UI. For the UX part, I drive the player through the same scenarios and assert the rendered error message is region-specific ('This title is not available in your country') and window-expiry-specific ('This title is no longer available') — not a generic 'We encountered a playback error' that leaves the user with no recourse. I also assert that a title that passes the entitlement check but fails for a different reason (e.g. DRM hardware level mismatch) shows a different, correctly attributed message — so I'm testing the error mapping, not just that an error appears. This separation matters: the entitlement API and the DRM license server are different systems, and a bad 403 from one should not produce the same user message as a bad response from the other.

Model answer

I instrument the player's ABR event stream and run the test against an injected network profile rather than relying on a real variable network, so the test is deterministic and repeatable in CI. The injection method depends on the platform: in a browser I use Charles Proxy or the Chrome DevTools Protocol bandwidth throttle; on mobile I use a proxy-based profile or the OS network conditioner. I configure three phases: a 30-second window at 8 Mbps (baseline), a drop to 400 kbps, then a recovery back to 8 Mbps after 60 seconds. The assertions fall into two distinct groups. For the descent: I listen to the player's quality-level-changed event (Shaka Player exposes this as adaptation events; Video.js has a qualitychange event; all HLS.js-based players have a levelSwitched event) and assert that within 5 seconds of the bandwidth drop, the player has selected a quality rung at or below the 400 kbps bitrate ceiling. I also assert that no stall event (the player's 'waiting' or 'buffering' lifecycle event) fires for more than 200 ms during the descent — the player should descend proactively before the buffer drains, not reactively after it stalls. For the step-up: after bandwidth recovers, I assert that within the configured hysteresis window (say, 20 seconds of stable bandwidth), the player emits a quality-level-changed event selecting a rung at or above 720p. This is the assertion most test suites omit, which is exactly where quality-stuck regressions hide. I also add a 120-second hold after recovery and assert the final quality level is ≥720p — a player that steps up once to 360p and then gets stuck there still fails this assertion. The oracle throughout is the player's event stream and buffer-depth readings, not visual inspection or a screen recording. If I want to verify a regression was introduced in the step-up hysteresis configuration, I can compare the time-delta between recovery and first step-up event across builds.

Model answer

I use a Widevine test proxy — a local license server shim that can be configured to respond to license requests with whatever I specify — so I can inject an expiry without waiting for the real token TTL. I set the proxy to issue a license with a TTL of, say, 90 seconds, which makes the expiry reproducible in a test run. The happy path: the player must request a license renewal before the token expires. I configure the proxy to respond successfully to renewals and assert that a renewal request arrives at the proxy within the last 20% of the TTL window (not after expiry). If no renewal request arrives before the token expires, the player has a renewal-timing bug. I also assert continuous playback through the renewal — no stall event, no quality drop — because a renewal that succeeds but causes a 2-second stall is a QoE regression. The failure path: I configure the proxy to return a license error on renewal (I use a specific EME error code, e.g. a MEDIA_ERR_ENCRYPTED response). The assertions here are two separate checks. First, the player must emit an 'encrypted' or 'error' event with an error code that I can map to the right UX message — I listen for this event in the test harness and assert it fires within 5 seconds of the failed renewal, not silently. Second, the player must render a user-visible error message that is actionable — something like 'Your session has expired — please refresh the page' — rather than freezing with the last frame still on screen while the progress bar stops moving. The silent-freeze detection is the hardest assertion: I drive the test with an automated player that can read the current playback position every 2 seconds and assert it is advancing. If playback position stops advancing and no error event has been emitted, that is the silent-freeze bug — the player has stopped but has not told the user why. This is distinct from generic API auth testing: the EME error code comes from the DRM stack's hardware-software negotiation, not from an HTTP 401, and the renewal flow involves the license server validating the device certificate again, not just checking a session token.

Model answer

I need two things to make this test reliable: test accounts at each subscription tier with a known limit, and a test environment where the TTL evaluation window is short enough to hit the race condition predictably. In production the TTL window might be 5–10 seconds; in a test environment it should be configurable to 1–2 seconds so parallel-script timing is not fragile. The basic enforcement test: I open two sessions for a 2-stream account sequentially (not concurrently) and assert both succeed; then I open a third and assert it is rejected with an error code and a user message that names the limit and offers an action. But this misses the race condition. The race-condition test: I use a scripted parallel session opener — two threads or processes that each call the stream-start API at the same time, within the TTL window. Both requests may succeed momentarily before the server counts them and triggers eviction. I assert that within the evaluation window, exactly one eviction fires (one session receives an eviction signal). Then I assert the eviction target: the documented policy might say 'evict the session with the earliest start timestamp' or 'evict the non-premium device type'. I assert the eviction signal is received by the correct session, not just by one of them. To test the eviction UX: I instrument the evicted session and assert it receives a signal that the player renders as an actionable message ('Your account is streaming on another device — upgrade your plan or end another session'), not a silent kill where the video freezes and the user sees no explanation. I also test the TTL-window edge: the two sessions open at exactly the TTL boundary — one just inside the window and one just outside — and assert the server evaluates them correctly. I run this matrix across all subscription tiers (1-stream plan, 2-stream plan, 4-stream plan) because enforcement logic often has per-tier bugs. This is session-level enforcement — the server is counting active stream-init calls and TTL-bounded session handles — which is fundamentally different from SaaS seat provisioning, where an admin allocates user accounts in a management panel and there is no real-time eviction of live sessions.

Model answer

I test this in three distinct phases. Phase 1 — position accuracy: I set up a test account, start playback on a TV emulator or a controlled device, pause at a known timestamp (say, 2,847 seconds), and then immediately call the watched-position API directly to assert the saved value is within ±5 s of 2,847. This tests the position write, not the resume — I want to confirm the API saved the value before I test whether the phone reads it correctly. Phase 2 — resume sequence on phone: I open the app on a phone (real device or emulator), trigger the resume flow, and instrument the player lifecycle: the player must call the watched-position API, receive the saved timestamp, and issue a seek() call to that position before calling play(). I assert the seek event fires with a position within ±5 s of the saved value, and that the first video frame rendered corresponds to approximately that position. A player that plays from zero and then jumps at 2 s fails this assertion even though the final state looks correct. Phase 3 — DRM continuity: the phone must acquire its own DRM license. I intercept the license server traffic on the phone and assert a new license request is made for the phone's device certificate, not a reuse of the TV's license. The TV license is device-bound in the DRM stack and is not transferable. I also test the failure paths: position API returns an error → player falls back to the beginning with a notice, not a crash; saved position is beyond the content's current duration (e.g. the content was trimmed) → player clamps to the end and shows a helpful message; DRM license acquisition fails on the phone → player surfaces an actionable error, not a silent black screen. The cross-device resume test is a scalar handoff — the only shared state is a timestamp integer in the watched-position API — which is why the test surface is the API value and the seek sequence, not a complex conflict resolution mechanism. This is not IoT device-cloud state sync (where the physical device is the ground truth) or SaaS collaborative editing (where concurrent writes from multiple users must be reconciled).

Model answer

Three live-specific test cases that don't apply to VOD: First, end-to-end latency. A VOD stream has no concept of 'how old is this frame' because it is pre-recorded. A live stream has an encoder timestamp in the HLS/DASH manifest that represents when the segment was produced. I measure latency as the delta between the encoder-produced timestamp in the segment manifest and the player's wall-clock time when that segment is rendered. I instrument this in a test harness by parsing the manifest, extracting the PROGRAM-DATE-TIME tag (for HLS) or the publishTime (for DASH), and comparing it to Date.now() at the moment the player renders the corresponding frame. I assert this delta is within the SLA — for example, ≤10 s for near-live HLS or ≤3 s for LL-HLS. This is not just 'does it play fast' — it is a measurement with a defined threshold and a regression test. Second, DVR window boundary behaviour. VOD content has a fixed duration with no sliding boundary. A live DVR window has a start that advances as the live event progresses — segments older than the window duration are removed from the CDN. I test three boundary conditions: seek to the current live edge (assert the player goes to the most recent segment without error), seek to the earliest available segment (assert the player loads the oldest valid segment correctly), and seek to a timestamp that was in the window when the seek was initiated but has since slid out (the window advanced between the seek being requested and the segment being fetched). This last case is the production bug: the player requests a segment that has just become unavailable and receives a 404; without a graceful boundary handler it enters a 404-retry loop and freezes. I assert the player clamps to the new earliest available segment rather than freezing. Third, CDN origin failure and failover. VOD CDN testing is about cache behaviour; live CDN testing is about origin health and failover speed. I simulate origin failure by blocking the primary CDN origin at the proxy level — returning 5xx errors for all segment requests — and assert that the player switches to a secondary CDN within the defined failover threshold (e.g. within 2 failed requests or 5 seconds). I measure the stall duration from the first failed segment request to the first successful segment from the secondary CDN, and assert it is within the user-perceived stall SLA (≤5 s). I then assert the player resumes at the correct live position after failover, not from the beginning of the DVR window.

Model answer

I work from the manifest layer outward rather than trying to reach the ad server. The stitched manifest is the single source of truth for what the player will do — if the manifest is correct, the player will behave correctly; if the manifest is wrong, the ad server's configuration is irrelevant because the player follows the manifest. My approach has four steps. Step 1 — intercept and parse the stitched manifest. I capture the HLS playlist or DASH MPD returned by the SSAI stitcher (not the origin manifest) using a proxy, and parse the segment list. In HLS, each ad segment has an EXT-X-DISCONTINUITY tag at the break boundary, and each segment has an EXTINF duration value. I build the full segment timeline from the manifest: content segments before the break, ad segments, content segments after the break. Step 2 — validate declared durations. I download each segment in the ad pod using a tool like Bento4 or ffprobe and measure the actual media duration of each segment. I assert that the declared EXTINF duration in the manifest matches the actual media duration within a tolerance of ±100 ms. A discrepancy here is the root cause of content-skip bugs: the player calculates the resume offset from the declared duration, not the actual duration. If an ad creative is 32 s but the manifest declares 30 s, the player will seek 2 s into the post-break content, skipping 2 s of the film. Step 3 — validate content offset post-break. I calculate the expected byte offset for the first post-break content segment: it should equal the sum of (pre-break content duration) + (actual ad pod duration). I assert the manifest's segment URL for the first post-break content segment corresponds to the correct position in the content — I can verify this by checking the segment's sequence number or its byte-range offset against the unstitched origin manifest. Step 4 — edge cases. I inject a 0-duration ad creative using a VAST mock server (a lightweight HTTP server that returns a crafted VAST XML, which I control even without access to the real ad server) and assert the player emits a VAST error callback and resumes content immediately, without hanging at the stitching point. I also test an ad pod that is longer than the declared slot duration by configuring the mock VAST to return a creative with an actual duration 2 s longer than the slot, and assert the stitcher either truncates the creative or carries the overflow forward correctly rather than silently skipping content.

Model answer

I define all three metrics with a precise measurement methodology, a baseline, and a regression gate — not just a threshold. Rebuffer ratio: the total time the player spent in a buffering state divided by the total time the player was in a playing or buffering state, expressed as a percentage. The measurement comes from player lifecycle events — 'waiting', 'stalled', 'playing' — not from network measurements alone. Industry baseline for a well-served stream is ≤0.5% rebuffer ratio; a VOD start page load is not counted. For regression detection I use a rolling p95 over a 1-hour synthetic canary window compared to the 7-day trailing p95 for the same content and network profile. If the current p95 exceeds the baseline by more than 0.2 percentage points, I flag a regression. This approach catches gradual CDN degradation that a point threshold misses, and it filters out one-off spikes that a naive alert would page on. Startup latency: measured from the user-intent event (play button tap or autoplay trigger) to the first decoded and rendered frame, decomposed into three sub-phases as I described earlier (license round-trip, manifest and first-segment fetch, ABR startup decision). I instrument this in a synthetic canary that runs on a scheduled cadence across representative device and network profiles. The regression gate is on each sub-phase separately, not on the total — a 1-second regression in license acquisition is a different root cause than a 1-second regression in CDN segment delivery, and a total-only gate would require post-hoc decomposition to triage. CDN failover recovery time: measured as the duration from the first failed segment request (5xx or timeout from the primary CDN) to the first successfully decoded frame from the secondary CDN, including the stall duration the user experiences during failover. I inject CDN failure synthetically by blocking the primary CDN origin at the proxy layer in a canary environment and measure the failover time per run. The SLA is ≤5 s user-perceived stall. For the regression gate: if median failover recovery time across 10 canary runs in a deploy pipeline exceeds 5 s, the deploy is blocked pending investigation. This is distinct from a production threshold alert — it is a pre-deploy gate that catches CDN configuration regressions before they reach users. The overall approach: synthetic canaries for pre-deploy gates (you control conditions, fast feedback), RUM metrics for post-deploy confirmation (you observe real users at scale, slower signal), and distribution-based regression detection for both — because streaming QoE metrics are noisy and point thresholds generate both false positives (spikes) and false negatives (slow drift).