Optimizely Global Holdouts: Measure the Real ROI of Your Experimentation Program

Every experimentation program eventually faces an awkward question from finance: "You reported 40 winning tests last year, each with a lift — so where is the 30% revenue growth those lifts add up to?" The honest answer is that summed experiment wins almost never equal the real, aggregate impact of a program. Winners are inflated by selection, effects decay, experiments interact, and untested changes and market drift muddy the picture. A global holdout is the one measurement that cuts through all of it: keep a small slice of users on the unchanged product, and compare them against everyone who received the stream of shipped winners. This guide explains what a global holdout estimates, how Optimizely's native Global Holdouts work, and how to run and read one without overselling the result.

A Global Holdout Answers the Question Individual Tests Cannot

A normal experiment control isolates one treatment during one test. It tells you whether to ship that change. It says nothing about the combined effect of every feature you shipped over a quarter or a year.

A global holdout flips the unit of analysis from the experiment to the program. You hold back a small, stable group from all experiments and newly rolled-out winners, let everyone else experience the full stream of changes, and compare the two groups on a top-line business metric. The estimand is aggregate incremental impact — what your experimentation program actually added — not the arithmetic sum of dashboard lifts.

Why the sum is almost always too optimistic:

Winner's curse. Shipped "wins" over-represent lucky overestimates; regression to the mean pulls the true effect down.
Decay and interaction. A lift measured in week one can fade (novelty) or collide with a later change.
Non-independence. Experiments reuse the same users and metrics, so their effects are not additive line items.
Everything else. Product-wide movement also includes untested releases and external drift. A holdout is the clean baseline that separates your program's contribution from the noise.

Holdout vs Control vs Holdback vs Exclusion Group

These terms get used interchangeably and mean very different things:

Concept	What it is	Scope
Experiment control	Baseline for a single hypothesis	One test; users may still join other experiments
Global holdout	Persistent group kept on the unchanged product across the whole program	Project/environment-level, cumulative reporting
Traffic holdback	Unallocated experiment traffic	Not a persistent, reported program control
Exclusion group	Ensures a user joins at most one member experiment	Prevents overlap; does not keep users on the unchanged product

The distinction that matters most: an exclusion group stops a user from being in two experiments at once, but those users still receive experiments. A global holdout keeps its users on the default experience entirely, so it can measure the program's cumulative effect.

Is Your Program Ready for a Holdout?

A holdout costs something real — every held-out user forgoes new features — so it earns its keep only under the right conditions.

Strong fit:

Many experiments and feature rollouts per quarter.
A stable, shared primary business metric leadership already trusts.
Flags whose default off behavior is safe and meaningful.
A genuine need to report aggregate program ROI, not just individual wins.

Poor fit:

Only a handful of unrelated experiments (nothing to aggregate).
A product that cannot safely maintain the old experience for months.
Unstable identity across devices/services (membership won't hold).
A period of large seasonal or migration upheaval, where "unchanged control" is operationally incoherent.

And an ethical line: critical security, compliance, reliability, or contractual fixes should never be withheld from a holdout group just to preserve a clean measurement. Keep the allocation small and review material harms.

How Optimizely Global Holdouts Work

Optimizely shipped native Global Holdouts for Feature Experimentation, configured in the Flags area for a specific environment. The mechanics are precise and worth understanding before you launch one.

Held-out users see the default off variation. A visitor assigned to the holdout only ever receives the default off variation for any flag within the holdout's scope — regardless of any A/B test, targeted delivery, or multi-armed bandit they would otherwise qualify for. This is why your preflight must confirm that off is a safe, functional status-quo experience for every affected flag.

Assignment is deterministic and identity-bound. Feature Experimentation is stateless and buckets a stable user identifier deterministically. Durable holdout membership therefore depends entirely on supplying consistent user IDs across your services — if the same person arrives with different IDs, they'll drift in and out of the holdout.

Allocation is small by design. Optimizely lets you set the held-back percentage and recommends not exceeding 5%, warning you if you configure more. More withheld traffic means more opportunity cost and can slow your ordinary experiments, since it removes users from their samples.

flowchart TD
    A[Eligible users with stable IDs] --> B{Global holdout bucketing}
    B -->|Up to 5%| C[Persistent holdout]
    B -->|Remaining users| D[Normal experimentation program]
    C --> E[Default off variations]
    D --> F[A/B tests and feature rollouts]
    F --> G[Concluded and deployed winners]
    E --> H[Primary company metric]
    G --> H
    H --> I[Aggregate incremental program impact]

Configure and Operate the Holdout

Preflight the flag defaults. Inventory every flag the holdout can affect and verify that off means the intended status quo and stays functional. Flag any release that must bypass the holdout for safety reasons.

Create it. Give it a name/key and an executive hypothesis, choose the environment, set the metrics (one executive primary metric plus a restrained set of secondary/monitoring metrics), and set the traffic allocation (kept ≤5%). You can apply an audience, but for a program-ROI question you generally want it to apply to everyone.

Health-check before you start. Validate stable identity, expected allocation, off decisions, event flow, and metric coverage in development or staging first — the same discipline as an A/A test. Record the start date, the affected surface, the exclusions, and the owner.

Understand the one-way lifecycle. This is the critical operational constraint: once a holdout starts, you cannot pause it — you can only permanently conclude it. When you conclude, every held-out user is rebucketed and begins receiving live variations, and a concluded holdout cannot be re-enabled. Started holdouts also cannot be deleted (only concluded, then archived), which deliberately protects a quarter's worth of data from accidental loss. Treat launching a holdout as a governed program decision, not a dashboard toggle.

Keep the treatment side truthful. Optimizely's aggregate comparison depends on you concluding experiments and deploying their winning variations — the holdout compares held-out users against those exposed to the deployed winners, so a win that was never concluded and deployed won't be reflected accurately. Assign an owner and an SLA for that conclude-and-deploy step, and maintain a change ledger (date, rule, shipped variation, affected population, primary metric, rollback) so you can interpret the aggregate later.

Read the Results Without Overselling

Optimizely computes the holdout result by comparing unique conversions per visitor and conversion rate between the In Holdout and Not in Holdout groups, producing a confidence interval, significance, and a cumulative improvement percentage — essentially one large-scale A/B test for your whole program. Read it with discipline:

Lead with the interval and the absolute value. Report the aggregate effect and its uncertainty on the primary metric. Translate to revenue only with an explicit denominator, period, and population stated.
Compare against the bottom-up sum. Put the holdout estimate next to the summed/compounded estimate from individual winners. A lower holdout number is expected and healthy — the gap is your prompt to audit power, selection, novelty, interaction, and rollout adherence.
Don't use it to rank experiments. A holdout estimates the bundle. It cannot say which feature drove the movement — use the original experiments or targeted follow-ups for attribution.
Watch time-varying risk. Long-running holdouts accumulate code and configuration complexity and can drift from a supportable product experience. Set a review cadence and a maximum decision horizon before launch, and monitor guardrail metrics on both groups.

When native capabilities don't fit, a self-service holdout — a deterministic eligibility attribute that excludes a fixed bucket from later experiments — is the fallback, at the cost of more manual configuration and the need to manage membership yourself as the user base grows.

Global Holdout Launch Checklist

The executive question and single primary metric are written down.
Every affected flag's off default is safe and represents the intended status quo.
A stable user identifier is used consistently across SDKs and services.
Allocation is justified and ≤5%.
Mandatory/safety releases have an explicit bypass policy.
Concluding experiments and deploying winners has an owner and an SLA.
Decision and conversion telemetry were QA'd before start.
Review dates, a maximum horizon, and ethical escape criteria are documented.
Reporting distinguishes the aggregate program effect from individual-experiment attribution.

A global holdout is how an experimentation program graduates from "we ran a lot of winning tests" to "here is the defensible, incremental value we added." It is the one number that survives the winner's curse — and the one most likely to keep leadership investing in experimentation at all.