Optimizely Experiment QA Checklist: Validate Before Launch

Most experiments that produce garbage results were not undone by bad statistics — they were undone before launch, by a variation that only rendered on desktop, an event that fired twice, or a "winner" that was really QA traffic polluting the numbers. A pre-launch QA pass is the cheapest reliability you will ever buy, and it is the step teams skip most under deadline pressure. This is the repeatable, gated procedure — with the right Optimizely tool for each check, and, just as important, what each tool cannot prove.

If you want the interactive version, the Experiment QA Checklist tool walks the same steps and can auto-verify parts of your setup when connected to Optimizely. This article is the reasoning behind it.

QA Protects the Experiment's Causal Chain

An experiment is only trustworthy if an unbroken chain holds from eligibility to analysis. QA is not "does the variation look right" — it is verifying every link:

Eligibility — the right user, on the right URL or request, qualifies.
Assignment — that user gets the expected variation, consistently, on return visits.
Delivery — the intended UI or code path executes without regressing anything else.
Measurement — decision and conversion events carry the correct identifiers and values.
Analysis — Optimizely and any external analytics receive clean, non-duplicated data.

A break anywhere invalidates everything downstream. A flawless variation whose conversion event fires twice is worthless; a perfect event on a variation that only renders for half its audience is worse, because it looks fine.

Freeze the Spec and Define Pass/Fail Evidence

QA against a moving target is theatre. Before testing, freeze the specification: the hypothesis; primary, secondary, and guardrail metrics; the exact experiment, variation, page/flag, event, and audience API names and IDs; the minimum detectable effect; traffic allocation; and start/stop rules. Record a configuration summary so any later edit is detectable.

Then decide what evidence counts as a pass before you look. Build a small matrix — test case, environment, user/device, expected decision, action, expected event, evidence, owner, status — and require concrete artifacts: screenshots for visual checks, captured payloads or logs for data checks. "It looked fine" is not evidence. Separate blocking defects (must fix before launch) from accepted limitations (documented and signed off).

Test Every Variation, Journey, and the Control

Walk the control and every treatment across the real matrix: desktop, mobile, tablet, each supported browser, responsive breakpoints, logged-in and logged-out, localization, accessibility, and a slow network/device. Exercise dynamic states, not just the happy path — modals, validation errors, empty/loading/error states, SPA route changes, and back/forward navigation.

Watch for the regressions that A/B changes commonly introduce: flash-of-original-content or flicker, layout shift, duplicated event handlers, console errors, failed requests, and handlers that fail to clean up when a page deactivates. Measure a key performance indicator before and after the variation activates — a visual change can quietly cost you load time, and a slow variation loses for reasons that have nothing to do with your hypothesis.

Treat the control as a real test case. Confirm the original experience still activates and measures correctly, and that shared/project code has not accidentally altered it. A contaminated control biases every comparison.

Verify Targeting, Activation, and Assignment

Test both directions. Positive cases: every included URL, route, audience, attribute, and user state activates at the correct lifecycle moment, across every page of a multi-step journey. Negative cases: excluded URLs and audiences do not activate; users outside the traffic allocation or inside an exclusion group see nothing; missing or malformed attributes are handled; and deactivating a page removes the changes and their event handlers where designed.

Then confirm stable assignment: revisit, refresh, open new tabs, and continue the funnel to verify the user keeps the same variation. For server-side SDKs, make sure the same bucketing identifier is supplied across services and devices, and never switch identifiers mid-funnel — that re-buckets the user and shreds the data. (For deliberately overriding assignment during QA, see Overriding Variation Assignments in Optimizely.)

Pick the Right QA Mode: Web vs Feature Experimentation

The single most common QA mistake is trusting a tool to prove something it structurally cannot. Match the mode to the question.

flowchart TD
    A[Freeze specification] --> B[Variation and journey QA]
    B --> C[Targeting and assignment QA]
    C --> D{Product}
    D -->|Web / Performance Edge| E[Preview + force checks]
    D -->|Feature Experimentation| F[Preproduction + allowlist / QA audience]
    E --> G[Guarded live telemetry check]
    F --> G
    G --> H{Decision, conversion, analytics<br/>payloads correct?}
    H -->|No| I[Fix and rerun affected cases]
    I --> B
    H -->|Yes| J[Cross-functional sign-off]
    J --> K[Controlled launch + first-hour monitoring]

Web Experimentation:

Preview is right for draft visual, audience, activation, and event-feed checks without publishing — but it slows snippet execution and does not reproduce live timing, its event feed follows the code path but sends no network POST (preview events never reach the Results page), and crucially analytics integration logic does not run in Preview. Never let Preview be your proof of timing, event delivery, or analytics.
Force parameters (optimizely_x…) pin a specific variation for a focused check, bypassing audience and traffic — but they do not satisfy URL targeting or activation, and forcing updates the visitor's local variation mapping, so QA from a clean/reset profile. Keep them disabled by default.
A test cookie or QA audience is what you want when QA must cross pages, fire real events, and exercise live integrations while ordinary visitors stay excluded.

Feature Experimentation:

QA in a separate development/preproduction environment first — environments have separate Results data — then copy rules to production.
Allowlisting is the easiest exact-variation path: up to 50 user IDs per A/B or MAB rule, forced via the datafile's forcedVariations map, overriding audience targeting and traffic allocation. It proves the variation path, not ordinary eligibility. The SDK user ID must match the allowlisted ID exactly. To isolate allowlisted users, target an attribute no real user has or set traffic to 0%.
For more than 50 testers, or to exercise real audience evaluation, use a QA audience on an internal attribute — Optimizely warns that large forced groups bias results and bloat the datafile.
In automated suites, mock Decide for each important variation path in integration tests, but keep at least one test that uses the real SDK and configuration so the contract is exercised.

Validate Decision and Conversion Telemetry

Visual delivery and telemetry are separate QA layers — verify them independently.

Decision event: confirm the experiment/flag key, variation key, visitor/user ID, timestamp, attributes, and page/URL, and that a single intended exposure does not generate duplicate decisions.

Every conversion event: confirm the exact API name, experiment context, timestamp, and tags/properties. For revenue and numeric metrics, validate units, currency, sign, decimal scaling, and aggregation — a value off by 100× from a cents/dollars mismatch is a classic. Test each trigger path separately: click, keyboard submit, programmatic submit, delayed elements, redirects, and failure paths.

Inspect the live network path. In Web, look at the /events POST and its body — a 204 means Optimizely accepted the batch, not that the payload is semantically correct, so read the decision and event arrays and check the IDs, keys, UUIDs, and tags. In Feature Experimentation, attach decision and track notification listeners (and log-event callbacks) to observe assignments and event calls, including what flows to downstream integrations.

Reconcile. After the processing delay, confirm the expected variation appears on the Results page, compare a small known QA count against both Optimizely and your analytics destination, and verify third-party integrations live — remember they do not run in Preview. If Optimizely and your analytics disagree, resolve it before launch (see GA / Optimizely data discrepancies).

Protect Production Data and Launch Gradually

QA traffic in production results is its own silent failure. Mark allowlisted/forced users in your evidence and never claim a forced test proves audience matching or traffic allocation. Prefer non-production environments with separate results; use documented test IDs and internal attributes; and if you reset results, record exactly what and when. When the implementation changed materially, run an A/A or low-risk smoke test to validate the end-to-end randomization and measurement pipeline — not as a ritual before every copy tweak.

Gate the launch on real sign-offs, each owning a slice: product (hypothesis and UX), engineering (code paths, performance, rollback), analytics (payloads, unit of analysis, reporting), and QA (all blocking cases passed with evidence). Then launch gradually when risk warrants — start with controlled traffic, watch delivery errors and data health, and ramp on a written plan. In the first hour and first cycle, monitor allocation, decision/conversion rates, errors, latency, analytics parity, sample ratio mismatch, and support signals, with pause/rollback thresholds defined in advance. Confirm no configuration was edited after sign-off.

The Copyable Pre-Launch Checklist

Specification frozen; experiment/variation/event/audience IDs and API names recorded.
Control and all variations passed the supported browser/device/journey matrix.
Included and excluded targeting cases both verified.
Assignment persists across revisits with the intended user identifier.
Preview checks done; timing-sensitive behavior and integrations tested live, not in Preview.
Decision and conversion payloads verified, including numeric/revenue units and no duplicates.
Third-party analytics verified live and reconciled with the Results page.
QA identities and traffic isolated from production users and results.
Rollback owner and pause thresholds documented.
Product, engineering, analytics, and QA approvals recorded.

The discipline is simple to state and easy to skip: decide what proof you need before you look, use each Optimizely tool only for what it actually demonstrates, and never let a green Preview or a 204 response stand in for evidence it cannot provide. A launch you can defend line by line is worth the hour it costs.