Experiment Design for A/B Testing: A Practical Framework

An A/B test can be implemented perfectly and still answer the wrong question. Experiment design is the written contract that connects a business decision to a causal comparison: who is eligible, what is randomized, what changes, how outcomes are measured, and what evidence will trigger action.

This guide turns those choices into a one-page specification, then maps the same design to Optimizely Web Experimentation and Feature Experimentation. It complements the deeper guides to an A/B testing hypothesis, experiment metrics, sample size and statistical power, test duration, and pre-launch QA.

Experiment design is the blueprint for a causal answer

The output of experiment design is not a list of variation ideas. It is a precise description of the comparison that will support a decision. Optimizely's own basic experiment plan asks why the experiment runs, where and when variations run, who sees them, what changes, and how success is measured. A rigorous design adds the unit of randomization, estimand, power assumptions, and decision rule.

flowchart LR
  A[Business decision] --> B[Causal question]
  B --> C[Unit and eligibility]
  C --> D[Control and treatment]
  D --> E[Metrics and estimand]
  E --> F[MDE, sample, and duration]
  F --> G[Optimizely configuration]
  G --> H[QA and launch]
  H --> I[Ship, iterate, stop, or conclude]

Start with the decision the result must support

Write the action before the hypothesis. A useful decision is specific enough that a result can change it:

Ship the checkout recommendation module to all eligible customers if it produces a practically valuable increase in completed orders without materially increasing refund rate; otherwise iterate or stop.

Turn that decision into one falsifiable causal question:

Among eligible signed-in shoppers, what is the effect of showing the recommendation module during checkout, compared with the current checkout, on completed orders within seven days of assignment?

This order prevents a common failure: designing an attractive variation, choosing an available metric, and only later deciding what the test was meant to prove.

Define the estimand before the dashboard metric

The estimand is the effect the team wants to estimate. Define five elements:

Use an intention-to-treat estimand by default: analyze people according to the variation to which they were assigned, even if rendering fails or they do not engage with the module. This preserves the benefit of random assignment and estimates the effect of the policy as deployed.

An exposure-only analysis answers a different question because it conditions on behavior or technical events that may happen after assignment. It can be useful as a diagnostic, but it should not silently replace the primary analysis.

Know when an experiment is the wrong tool

Do not randomize merely because the platform makes it possible. An experiment is a poor choice when:

• No plausible outcome would change the decision.

• Available traffic cannot detect an effect worth acting on within a useful timeframe.

• The treatment cannot be assigned independently enough to create a credible control.

• Random assignment would be unethical, unlawful, or operationally unacceptable.

• Instrumentation cannot distinguish assignment, exposure, and outcomes reliably.

When assignment cannot be randomized but a decision still needs evidence, use a quasi-experimental design and state its assumptions explicitly. That design should not be presented as equivalent to a randomized test.

Choose the experimental unit and assignment mechanism

The experimental unit is the entity independently assigned to control or treatment. It determines what must remain stable, where interference can occur, and which observations are independent in the analysis.

User, session, account, device, or cluster

Match the unit to how the treatment is delivered and how people can influence one another:

For a B2B account-level recommendation feature, randomizing individual members can contaminate the comparison: one member may configure the experience for colleagues, or team members may discuss it. Assign the account, then analyze with methods that respect account-level assignment.

Keep assignment stable

Use a durable identifier for the unit of randomization. If the treatment must persist across browser sessions and services, a transient session ID is not sufficient. Document how anonymous identities merge with authenticated identities and what happens when a user changes devices.

Optimizely Web Experimentation uses deterministic bucketing: it hashes a user ID or cookie with MurmurHash, so the same identifier maps to the same variation while the relevant configuration remains stable. Optimizely also warns that some traffic changes can rebucket visitors; the official bucketing guide recommends avoiding allocation changes after results accumulate.

The randomization unit is not necessarily the metric denominator. A user-randomized experiment may measure revenue per eligible user, orders per user, or average order value among orders. Each denominator answers a different question and must be fixed in the measurement plan.

Detect spillovers and network effects

Random assignment does not prevent treated units from changing control units' outcomes. Before launch, ask:

• Can users communicate about or share the treatment?

• Does treatment change inventory, marketplace liquidity, queue time, or prices for everyone?

• Can one account member alter settings seen by other members?

• Does a recommender learn from behavior pooled across variations?

If interference is plausible, assign clusters that contain the spillover, use a switchback or marketplace-specific design, or redefine the estimand. Record the remaining interference risk rather than assuming it away.

Define control, treatment, and eligibility precisely

A reproducible comparison needs versioned experiences and frozen entry rules. “Business as usual” and “all users” are not adequate specifications.

Make the control a real production baseline

Describe the control as thoroughly as the treatment:

• User interface and copy.

• Code path and service dependencies.

• Default feature-flag values.

• Existing personalization or recommendation logic.

• Analytics events expected from the baseline.

For the checkout example, the control is “the current checkout with no recommendation component and no recommendation API call,” not simply “original.” This matters if production changes while the experiment is being prepared.

Change one causal idea at a time

One hypothesis can require several implementation edits. Adding a recommendation module may change layout, make an API request, add loading behavior, and introduce new tracking. Those edits express one causal idea if the intended decision is whether to deploy the module as a package.

If the decision is which component causes the effect—placement, copy, image, or algorithm—use separate experiments or a multivariate test. A bundled treatment cannot identify the effect of each ingredient.

Freeze inclusion and exclusion rules

Define eligibility before anyone inspects results:

• Target pages, routes, or flag call sites.

• Geography, language, device, browser, and app version.

• Account tier, authentication state, and product entitlement.

• Prior exposure and concurrent-experiment exclusions.

• Employee, bot, fraud, and QA-traffic handling.

• The exact moment a unit enters the experiment.

Changing eligibility after viewing outcomes creates a new analysis population. If a safety issue forces a change, record the time and configuration, then decide whether to restart rather than blending incompatible cohorts.

Design the measurement plan

Metrics operationalize the estimand. They should be specified as formulas and event rules, not only selected by name in a dashboard.

Choose one primary metric

The primary metric must be affected by the treatment, interpretable by stakeholders, and tied to the ship decision. Define:

• Numerator and denominator.

• Aggregation unit.

• Attribution or conversion window.

• Event source and deduplication key.

• Direction of improvement.

• Practical effect threshold.

For the checkout example, “conversion” is ambiguous. A better definition is “distinct eligible users with at least one completed order within seven days divided by distinct assigned eligible users.”

Optimizely Web Experimentation treats the first metric added as the primary metric; its experiment setup guide recommends choosing an event directly affected by the change. Put the decision metric first deliberately.

Add restrained secondary and guardrail metrics

Secondary metrics explain mechanism or downstream impact. For the recommendation module, useful secondary measures might include recommendation clicks, units per order, and revenue per eligible user.

Guardrails describe harms that can veto a nominal win: refund rate, checkout latency, payment errors, support contacts, or gross margin. State each veto rule in advance. A dashboard containing dozens of metrics invites selective interpretation; include only metrics with a defined role.

Instrument assignment and outcomes separately

An outcome event is not proof that assignment or exposure worked. Validate three layers:

1. Assignment: the unit was bucketed into a variation.

2. Exposure: the intended experience was actually delivered.

3. Outcome: the conversion or guardrail event occurred and was attributed correctly.

Specify how the pipeline handles duplicate events, bots, missing identifiers, delayed events, identity merges, and events arriving after the experiment stops. Run an A/A or staged validation when the cost of a broken decision pipeline is high.

Design for enough information, not merely significance

A design should collect enough information to make the business decision. Statistical significance alone does not establish that an effect is large enough, safe enough, or measured without bias.

Pick an MDE worth acting on

The minimum detectable effect (MDE) should reflect the smallest effect that would justify implementation and opportunity cost. Derive it from baseline performance, expected value, engineering cost, risk, and competing roadmap items.

Do not inflate the MDE simply to make the required sample fit available traffic. That creates a faster test by declaring smaller—but potentially valuable—effects invisible.

Calculate sample size and calendar duration

Use the baseline rate, MDE, significance threshold, desired power, and number of variations to estimate the required sample. Then convert eligible sample into calendar time using:

• The share of traffic satisfying eligibility.

• Total experiment allocation.

• Variation traffic distribution.

• Weekly and seasonal cycles.

• Ramp time and planned outages.

• Conversion delay and data-processing latency.

The sample-size guide covers the calculation; the design document should retain the inputs and result. The duration guide explains why reaching a nominal sample early does not erase calendar-cycle or delayed-conversion requirements.

Predefine the stopping and decision rules

Write the complete rule before launch:

• Statistical method and confidence policy.

• Minimum runtime and information requirement.

• Practical-effect threshold or acceptable interval.

• Guardrail vetoes.

• Safety-stop conditions.

• Treatment of inconclusive results.

Do not stop because a preferred variation briefly looks significant, change the primary metric after seeing results, or segment repeatedly until a favorable result appears. Optimizely Stats Engine supports ongoing results monitoring, but operational visibility is not permission to rewrite the design after outcomes are known.

Map the experiment design into Optimizely

The same causal design can be implemented in either Web Experimentation or Feature Experimentation. The platform configuration enforces parts of the design; it does not choose the question, unit, estimand, MDE, or decision threshold for the team.

Start from one design card

Use a one-page design card as the source of truth:

flowchart TB
  D[Shared experiment design card]
  D --> W1[Web: URL or saved page]
  D --> W2[Web: audience]
  D --> W3[Web: original and variation]
  D --> W4[Web: allocation, distribution, metrics]
  D --> F1[Feature: flag and variables]
  D --> F2[Feature: attributes and audience]
  D --> F3[Feature: A/B test rule]
  D --> F4[Feature: decide, events, metrics]

Web Experimentation mapping

The current official Web Experimentation workflow maps the design as follows:

Set total traffic allocation to 100% when every eligible visitor should enter the experiment, then distribute that traffic 50/50 between original and treatment. Optimizely distinguishes the eligible share entering the test (allocation) from the share assigned to each variation (distribution) in its traffic documentation. Freeze both before launch.

Feature Experimentation mapping

Feature Experimentation implements the same design in application code and a flag ruleset:

Optimizely's official Feature Experimentation getting-started guide describes the sequence: create a flag, implement it with decide, define an audience and flag rule, then analyze the Experiment Results page. The newer A/B test configuration guide adds the hypothesis, test plan, traffic allocation, metrics, distribution mode, and flag variations.

const user = optimizelyClient.createUserContext(
  durableUserId,
  {
    market: checkout.market,
    account_tier: checkout.accountTier,
  },
)

const decision = user.decide('checkout_recommendations')

if (decision.variationKey === 'treatment') {
  renderCheckout({ recommendations: decision.variables })
} else {
  renderCheckout({ recommendations: null })
}

The durable user ID implements user-level assignment. Attributes implement declared eligibility. The flag key defines the decision point, while the returned variation selects the control or treatment code path. Track the completed-order event with the same user context so the result can be attributed to the decision.

Preserve the analysis contract in Collaboration

If the organization uses Optimizely Collaboration, store the structured design alongside the work. The official hypothesis documentation provides a Brief tab for a structured plan, a Variations tab for images or links, an Experiment tab for linking the implementation, and History for an activity trail.

Attach the design card, metric definitions, MDE inputs, QA evidence, risks, and assumptions. Collaboration can preserve the contract and make changes visible; it does not repair a design that was never specified.

Run a pre-launch design review

Review the design before launch, when weaknesses can still be fixed without contaminating results. Separate validity from feasibility so “we can build it” is not mistaken for “it will answer the question.”

Validity review

Challenge the causal comparison:

• Selection: Are eligibility rules evaluated before treatment, and are exclusions symmetrical?

• Spillover: Can treated units alter control outcomes?

• Attrition: Can missing outcomes differ by variation?

• Novelty: Could a temporary reaction dominate the intended long-run effect?

• Instrumentation: Can assignment, exposure, and outcomes be reconciled?

• Concurrent tests: Can another experiment interact with this treatment or metric?

• Identity: Can one randomization unit receive multiple IDs or variations?

Record mitigations and residual risks. Randomization balances expected pre-treatment differences; it does not fix differential loss, broken event collection, or post-assignment filtering.

Feasibility review

Confirm the operating plan:

• Required sample and realistic calendar duration.

• Engineering, design, analytics, and QA ownership.

• Event latency and conversion window.

• Rollout and rollback procedure.

• Safety monitoring and incident owner.

• Browser, device, and environment coverage.

• Production baseline freeze and change control.

If the design cannot be executed as written, revise it before exposure rather than improvising after launch.

One-page experiment-design checklist

• Decision, causal question, and hypothesis are explicit.

• Population, randomization unit, and analysis unit are distinct.

• Control and treatment are versioned production specifications.

• Eligibility, exclusions, and prior-exposure rules are frozen.

• One primary metric has a complete formula and conversion window.

• Secondary metrics explain mechanism; guardrails have veto rules.

• Assignment, exposure, and outcome events can be reconciled.

• MDE, sample, duration, and stopping rule are recorded.

• Optimizely pages or flags, audiences, traffic, variations, and metrics match the design.

• QA evidence, rollback owner, and launch approval are documented.

Use the Optimizely experiment QA checklist to turn the approved design into a production launch review.

Frequently asked questions

What is experiment design in A/B testing?

Experiment design is the specification that connects a decision to a causal comparison. It defines the population, randomization unit, eligibility, control, treatment, metrics, power assumptions, duration, and decision rule before traffic is exposed.

Is the randomization unit the same as the analysis unit?

Not always. A test can randomize accounts and observe users or transactions, but the analysis must account for dependence within each randomized account. Treating correlated observations as independently randomized overstates the amount of information in the test.

Should an A/B test change only one page element?

It should test one causal idea, which may require several coordinated implementation changes. If the decision requires isolating the contribution of individual elements or their interactions, use separate experiments or a multivariate design.

Can traffic allocation change after an experiment starts?

Operationally, Optimizely permits traffic changes, but they can affect bucketing and interpretation. Predefine allocation and distribution, avoid changing them after results accumulate, and document any safety-driven intervention.

Does Optimizely replace an experiment design document?

No. Optimizely implements pages or flags, audiences, variations, traffic rules, and metrics. The team must still define the causal question, unit, estimand, MDE, duration, risks, and action threshold. The configuration should be a faithful implementation of that prior design.