Quasi-Experimental Design: How to Measure Impact When You Cannot Randomize

Randomized experiments create a credible counterfactual by assigning comparable units to treatment and control. Product teams lose that protection when a migration, policy, threshold, or market-wide change determines who receives treatment. Quasi-experimental design is the disciplined alternative: it uses a non-random assignment mechanism to construct a comparison, then makes the causal claim conditional on a specific, testable assumption.

That qualification matters. A quasi-experiment is not an A/B test without an experimentation platform, and a regression does not turn an arbitrary before-and-after chart into causal evidence. This guide shows how to choose among difference-in-differences, interrupted time series, regression discontinuity, and synthetic control—and when to redesign the rollout so an Optimizely randomized experiment is possible instead.

What makes a design quasi-experimental?

A quasi-experimental design estimates causal impact without researcher-controlled random assignment. Its credibility comes from how treatment was assigned and why the untreated observations approximate what would have happened to treated units without the intervention.

Treatment changes, but assignment is not randomized

Every impact estimate needs a counterfactual: the outcome treated units would have produced without treatment. That outcome is unobservable, so the design must approximate it.

Selection bias appears when treatment assignment is related to potential outcomes. A checkout migration launched first in high-performing regions may appear successful because those regions were already improving. The analysis needs a design that separates the launch from that selection process.

Identification is the argument that lets observed data recover a causal quantity under stated assumptions. It is not a model fit statistic. A narrow estimate with an implausible identifying assumption is precisely estimated, not credible.

The design creates the causal argument

Begin with the assignment mechanism:

• Who received treatment?

• At what time?

• According to what rule?

• Who could not receive it?

• Could units influence one another?

Then select a method whose assumption matches that mechanism. A difference-in-differences model needs a credible comparison group's untreated trend. Regression discontinuity needs a threshold that creates local comparability. Interrupted time series needs a stable pre-intervention trajectory and no coincident shock that explains the break. Synthetic control needs untreated donor units that reproduce the treated unit before treatment.

Diagnostics can probe these assumptions, but no chart proves them. Every result should travel with the assumption, falsification checks, and the plausible ways the assumption could fail.

Product situations where randomization may be unavailable

Common examples include:

• A legal or compliance policy that applies to everyone on one date.

• An infrastructure migration sequenced by region or data center.

• A benefit or restriction determined by a fixed score threshold.

• A marketplace rule that changes liquidity or prices for treated and untreated users.

• An irreversible account migration with no safe way to revert individual units.

• A pricing or operational decision made centrally for one country.

The absence of user-level randomization does not automatically justify a quasi-experiment. First ask whether another unit or rollout mechanism can restore random assignment.

First, try to recover a randomized experiment

Quasi-experimental analysis accepts stronger assumptions than random assignment. Before accepting that tradeoff, test whether the treatment can be isolated at a different level or separated from deployment.

Randomize a different unit

When users interact, randomize a unit large enough to contain the spillover:

• Accounts for shared B2B settings.

• Stores for staff workflows and local inventory.

• Markets for marketplace or media effects.

• Routes for logistics interventions.

• Sellers for supply-side changes.

• Time blocks for systems where simultaneous treatment and control would interfere.

Cluster randomization changes the power calculation because outcomes inside a cluster are correlated. It may still be more credible than pretending individual users are independent.

Use a flag to separate deploy from release

Deploying code does not require exposing it to everyone. Optimizely Feature Experimentation can keep a new path dormant behind a flag, then use an A/B test rule to randomize eligible units. The official Feature Experimentation overview describes running experiments throughout the stack and rolling changes back without redeploying.

For a regional checkout migration, prefer this sequence where feasible:

1. Deploy both checkout paths.

2. Create legacy and migrated flag variations.

3. Use a stable user, account, or region identifier.

4. Pass declared location and eligibility attributes.

5. Add an A/B test rule with a control and treatment variation.

6. Send decision and conversion events for the same identity.

7. Keep a safe off or legacy path for rollback.

Optimizely's user-ID guidance warns that inconsistent IDs can rebucket users and corrupt attribution. The randomization unit and identifier policy belong in the design, not only in SDK code.

Know when randomization is unethical or operationally incoherent

Do not force an A/B test when:

• Safety or compliance requires universal treatment.

• Withholding the treatment would cause unacceptable harm.

• A migration is irreversible and rollback cannot be supported.

• One unit's treatment necessarily changes every other unit's outcome.

• The intervention is a single external event rather than an assignable product change.

In those cases, state why randomization was unavailable and design the strongest feasible quasi-experiment before rollout.

Choose a quasi-experimental design from assignment

Choose the method from the source of treatment variation—not from the analyst's preferred package. The assignment mechanism determines which counterfactual and assumption are defensible.

flowchart TD
  A[Can treatment be randomized ethically and coherently?]
  A -->|Yes, individual exposure is isolated| B[Individual A/B test]
  A -->|Yes, users interact| C[Cluster or time-block randomized test]
  A -->|No| D{How was treatment assigned?}
  D -->|Treated and comparison groups over time| E[Difference-in-differences]
  D -->|Clear intervention date| F[Interrupted time series]
  D -->|Eligibility cutoff| G[Regression discontinuity]
  D -->|One treated unit plus donor pool| H[Synthetic control]

Difference-in-differences for groups over time

Difference-in-differences (DiD) compares the treated group's before-after change with the comparison group's change:

DiD = (Treated after − Treated before) − (Comparison after − Comparison before)

Suppose Region A migrates to a new checkout while Region B remains on the legacy system:

The DiD estimate is +0.5 percentage points. The naive before-after estimate for Region A is +0.8 points; DiD subtracts the +0.3-point change observed without migration.

flowchart LR
  TB[Treated before: 4.0%] --> TA[Treated after: 4.8%]
  CB[Comparison before: 4.2%] --> CA[Comparison after: 4.5%]
  TA --> D[Net change: 0.8 - 0.3 = 0.5 pp]
  CA --> D

The central identifying assumption is parallel trends: without treatment, the treated group would have changed like the comparison group over the relevant period. The current practitioner's guide to DiD emphasizes that designs beyond the simplest two-group, two-period case require deliberate choices about estimators, covariates, weights, multiple periods, and staggered adoption.

Plot long pre-treatment trends, estimate event-time effects, and test placebo intervention dates. Similar pre-trends support credibility but do not prove the unobserved post-treatment counterfactual.

Interrupted time series for a clear intervention date

Interrupted time series (ITS) estimates whether an outcome's level or slope changes at a known intervention date relative to its pre-intervention trajectory. It fits a universal policy or migration applied at one time when a separate comparison group is unavailable.

Use enough pre- and post-intervention periods to estimate:

• Baseline level.

• Pre-intervention trend.

• Immediate level change.

• Change in post-intervention slope.

The peer-reviewed ITS tutorial highlights features unique to time-series data. Product analyses should model seasonality and autocorrelation, predefine lagged effects, and investigate releases, campaigns, outages, or economic changes occurring near the intervention.

An ITS result is weakened when the launch date coincides with a holiday campaign or tracking migration. A control series or negative-control outcome can help distinguish the intervention from a common shock.

Regression discontinuity for a threshold rule

Regression discontinuity design (RDD) applies when treatment changes at a cutoff in a continuous assignment variable: accounts above a risk score receive additional verification, sellers above a volume threshold receive a new workflow, or customers reaching a date-based eligibility rule enter a program.

Compare units just above and below the threshold. If potential outcomes would otherwise vary smoothly through the cutoff and units cannot precisely manipulate assignment, the discontinuity estimates a causal effect near that cutoff.

The estimate is local. It describes units near the boundary, not necessarily the entire customer base. The methodological review Regression Discontinuity Designs organizes RDD around continuity and local-randomization frameworks and stresses estimation, inference, validation, and falsification.

Vary the bandwidth, inspect outcome and covariate continuity, test for sorting around the threshold, and use placebo cutoffs. Avoid high-order global polynomials that let distant observations drive the local comparison.

Synthetic control for one treated market or unit

Synthetic control is useful when one aggregate unit is treated and several untreated units form a donor pool. It selects weights so the weighted donor combination reproduces the treated unit's pre-treatment outcomes and predictors.

For one migrated country, a synthetic control might combine 50% Country B, 30% Country C, and 20% Country D. After launch, compare the treated country's actual outcome with that fixed weighted counterfactual.

The foundational synthetic-control study demonstrates the method for aggregate interventions and comparative case studies. Product teams should report pre-period fit, donor weights, leave-one-donor-out sensitivity, and placebo effects obtained by treating donor units as if they had launched.

Poor pre-treatment fit means the donor pool cannot reproduce the treated unit; weighting does not manufacture a credible counterfactual from unsuitable controls.

Build the analysis before the rollout

Pre-registration is valuable even when the change cannot be randomized. It prevents method, outcome, and comparison choices from drifting toward the result after the launch.

State the assignment mechanism and estimand

Document:

• Who receives treatment and when.

• The operational, legal, or threshold rule that assigns it.

• Why randomization is unavailable.

• The population represented by the estimate.

• The effect scale and time horizon.

“Impact of the migration” is not an estimand. “Change in completed checkout sessions per eligible account during the first eight post-launch weeks for regions migrated on July 1, relative to the counterfactual under the legacy checkout” is.

Freeze outcome definitions and windows

Define the primary outcome's numerator, denominator, aggregation unit, data source, deduplication rule, lag, and exclusions. Specify:

• Pre-period and post-period.

• Expected ramp or transition window.

• Missing-data policy.

• Bot, employee, and test traffic.

• Identity changes during migration.

• Backfilled or late-arriving events.

Preserve raw assignment, decision, and outcome events. A tracking change introduced with the product change can create the apparent effect.

Preserve a usable comparison

Do not contaminate every possible control at once. If operationally feasible:

• Retain unaffected markets or accounts.

• Avoid migrating the entire donor pool simultaneously.

• Preserve the assignment score around a threshold.

• Collect sufficient pre-period data before the intervention.

• Stagger launches only when the sequence has a genuine operational rationale.

Do not invent a convenient control after seeing which region performed best. Comparison construction is part of the design and should be fixed before outcomes are known.

Test assumptions rather than hiding them

Diagnostics determine how much confidence a design deserves. A failed check weakens the causal interpretation; it is not merely a warning badge to acknowledge and ignore.

Pre-trends and placebo dates for difference-in-differences

• Plot treated and comparison outcomes across a long pre-period.

• Estimate event-time coefficients before and after treatment.

• Test dates when no intervention occurred.

• Re-estimate with alternative defensible comparison groups.

• Account for anticipation if users or teams change behavior before launch.

A non-significant pre-trend test is not proof of parallel trends; it may have low power. Use domain knowledge and visual magnitude alongside uncertainty.

Bandwidth, continuity, and sorting checks for RDD

• Show the assignment-variable distribution around the cutoff.

• Test whether units cluster suspiciously on one side.

• Examine pre-treatment covariates and unaffected outcomes at the threshold.

• Report estimates across defensible bandwidths and polynomial orders.

• Test placebo cutoffs where treatment does not change.

If units can precisely manipulate the score, near-cutoff observations may no longer be comparable.

Seasonality, autocorrelation, and shocks for time series

• Plot multiple seasonal cycles before launch.

• Model serial dependence rather than treating periods as independent.

• Include predeclared seasonal structure.

• Test alternate intervention lags.

• Inventory concurrent releases, campaigns, outages, and external shocks.

• Add a control series when one shares the same shocks but not the treatment.

A visible break at the launch date can still be caused by another event at the same time.

Donor fit and placebo units for synthetic control

• Report pre-treatment prediction error.

• Publish donor weights and justify donor exclusions.

• Refit after removing influential donors.

• Assign placebo treatments to untreated units.

• Compare post-treatment gaps relative to each unit's pre-period fit.

An unusual treated-unit gap is more persuasive when placebo units with comparable pre-fit do not show similar breaks.

Common product-analysis traps

Quasi-experimental labels do not rescue analyses that lack a credible counterfactual.

Before-after is not automatically a quasi-experiment

A single pre-period and post-period comparison leaves seasonality, product mix, acquisition campaigns, macro conditions, and tracking changes confounded with treatment. ITS requires a modeled trajectory and sufficient periods; DiD requires a comparison group's change.

Matching does not balance unknown confounders

Matching or propensity scores can balance measured covariates under strong assumptions. They do not balance variables that were not measured, were measured poorly, or were affected by treatment. Matching is not equivalent to random assignment.

More rows do not fix biased assignment

Millions of events can produce a tiny standard error around a biased estimate. Effective information may also be determined by the number of markets, accounts, or time periods—not the number of clicks.

Launch timing can be endogenous

Teams often migrate regions first because they have the best engineering support, lowest risk, greatest need, or strongest expected performance. That timing is related to outcomes. Record why each unit launched when it did and choose an estimator appropriate for staggered adoption rather than treating timing as arbitrary.

An Optimizely-first decision framework

Use Optimizely to recover randomization when the treatment can be isolated. Use the warehouse for quasi-experimental estimation when assignment is externally fixed. Do not blur that boundary.

If individual exposure is isolatable, run an A/B test

Implement:

• A flag with legacy and migrated variations.

• A stable randomization ID.

• Location and eligibility attributes.

• An audience and A/B test rule.

• A decision event from decide.

• Conversion events tracked for the same user context.

Optimizely's audience documentation states that Feature Experimentation evaluates attributes passed through user-based Decide and Track calls. Keep attribute types and values consistent with the audience conditions.

If users interact, randomize clusters or time blocks

Pass a stable cluster identifier as the randomization identity or use a documented bucketing-ID policy, then log cluster and time-block attributes to the warehouse. The analysis must use the randomized unit and account for within-cluster correlation.

Avoid using ad hoc audience segments as a substitute for random assignment. An audience defines eligibility; the A/B test rule provides randomized variation assignment within that eligible population.

If assignment is fixed, use the appropriate quasi-design

Export exposure and outcome data to the warehouse and estimate DiD, RDD, ITS, or synthetic control with dedicated causal-analysis tooling. Optimizely Stats Engine reports randomized experiment results; it does not natively estimate these quasi-experimental designs.

Optimizely documents that A/B test decisions send impressions used on the Experimentation Results page, while targeted-delivery-only decisions do not send experiment impressions. See impressions and decisions in Feature Experimentation. A delivery rule can release a migration, but its dashboard is not a substitute for a warehouse counterfactual.

For interpretation boundaries, use the guide to reading Optimizely results. For interference across experiments, see running concurrent A/B tests. Post-hoc subgroup exploration is covered separately in A/B test segmentation.

Reporting checklist

• Explain the assignment mechanism and why randomization was unavailable.

• Name the design, estimand, and identifying assumption.

• Define treatment timing, pre-period, post-period, and comparison construction.

• Freeze outcomes, denominators, exclusions, lags, and missing-data policy.

• Report pre-treatment fit and design-specific diagnostics.

• Include placebo tests and sensitivity analyses.

• Report uncertainty and both absolute and relative effect sizes.

• Separate statistical precision from identifying credibility.

• State limitations and the population to which the estimate applies.

• Define the decision boundary: what this evidence can and cannot support.

Frequently asked questions

What is a quasi-experimental design?

It is a causal design in which treatment assignment is not controlled through randomization. The analysis uses a policy, threshold, timing change, comparison group, or donor pool to construct a counterfactual under a design-specific identifying assumption.

Is a before-and-after analysis a quasi-experiment?

Not by itself. A two-point comparison cannot separate treatment from seasonality, trends, campaigns, external shocks, or tracking changes. Interrupted time series requires enough periods to estimate a pre-intervention trajectory and assess a level or slope break.

What is the main assumption in difference-in-differences?

The key assumption is that, without treatment, the treated group's outcome would have followed the same relevant trend as the comparison group. Pre-trend and placebo checks probe credibility but cannot prove the unobserved counterfactual.

When should a product team use regression discontinuity?

Use it when treatment changes at a known cutoff in a continuous assignment variable and units near that cutoff are plausibly comparable. The resulting effect is local to the threshold and should not be extrapolated automatically.

Can Optimizely Stats Engine analyze quasi-experiments?

No. Optimizely Stats Engine analyzes randomized experiment results. When assignment is fixed externally, export data and estimate DiD, RDD, ITS, or synthetic control in the warehouse. If exposure can be randomized safely, redesign the rollout as an Optimizely A/B test instead.