Winner's Curse and Regression to the Mean in A/B Testing

A variation wins your A/B test with a measured +12% lift. You ship it, watch the metric for a quarter, and the gain settles at +3% — or disappears entirely. Nothing broke, the tracking is fine, and the test reached significance. The lift was overstated from the start. This is the winner's curse: the act of selecting the winning variation systematically inflates its measured effect above the true effect. Its close cousin, regression to the mean, is why the inflated number shrinks once you re-measure it in production. This article explains the mechanism precisely, quantifies it with a simulation, and lays out the safeguards that keep your launch forecasts honest.

Why the Winning Variation's Lift Is Usually Overstated

Every measured effect is the sum of a true effect and noise:

observed lift = true lift + random noise

The noise is symmetric — over many hypothetical repeats of the same test, it is equally likely to push the observed lift above or below the truth. On its own, that is harmless: an unbiased estimate.

The bias enters at the moment you select. You do not report a randomly chosen variation. You report the one that looked best, or the one that crossed the significance threshold. Both selection rules preferentially pick variations that caught a positive draw of noise. A variation whose true effect is modest but whose noise happened to be large and positive is exactly the kind of result that wins a test and gets shipped. A variation with the same true effect but unlucky negative noise never makes the cut.

So the population you launch from is not a fair sample of your ideas — it is filtered for good luck. Conditional on having won, the expected observed lift is strictly greater than the true lift:

E[observed lift | selected as winner] > true lift

That inequality is the winner's curse in one line. It holds even when your statistics are impeccable, your sample ratio is clean, and your significance threshold is correct. It is a property of selection, not of a bug.

The Winner's Curse, Formally

Statisticians describe this with two error types, following Gelman and Carlin's work on design analysis:

• Type M (magnitude) error — the estimate is correct in sign but exaggerated in size. A real +4% lift is reported as +11%.

• Type S (sign) error — the estimate has the wrong sign entirely. A variation that is truly slightly negative is reported as a significant winner.

The severity of both errors is governed by statistical power. When a test is well powered, a significant result usually reflects a real, well-estimated effect, and the exaggeration is small. When a test is underpowered, the only way to reach significance is to catch an unusually large positive draw of noise — so every significant result is, by construction, an overestimate. Low power does not just make winners rarer; it makes the winners you do find more misleading.

This is why "we reached significance" is not a defense against the winner's curse. Significance is the selection filter that creates the curse. The relevant question is not whether the result is significant but how much the result was inflated by the fact that you only looked at, and only shipped, the significant ones.

The same logic scales with the number of things you compare. If you run one variation, you select on a single noisy estimate. If you run ten variations, or track twelve metrics and report the one that moved, you are taking the maximum of many noisy estimates — and the maximum of a set of noisy draws is biased upward far more than any single draw.

flowchart TD
  A[Test many variants or metrics] --> B[Each observed lift = true effect + noise]
  B --> C{Select the highest observed lift}
  C --> D[Winner likely caught positive noise]
  D --> E[Reported lift exceeds true lift]
  E --> F[Launch: noise no longer helps]
  F --> G[Realized lift regresses toward the smaller true value]

Regression to the Mean: The Same Effect Over Time

Winner's curse and regression to the mean are two views of one phenomenon. The winner's curse is the cross-sectional view: at decision time, the selected estimate is biased high. Regression to the mean is the temporal view: when you measure the same thing again, an extreme first measurement tends to be followed by a less extreme second one.

The winning variation was, almost by definition, an extreme measurement — that is why it won. When you re-observe its performance after launch, in a fresh stream of traffic, the noise component is redrawn from scratch and no longer conspires in the variation's favor. The second measurement drifts back toward the variation's true effect, which is lower than what the test reported. Nothing changed about the variation. The world simply stopped handing it a lucky draw.

This is distinct from a novelty or primacy effect, where the true effect itself changes over time as users habituate to a new experience. Regression to the mean happens even when the true effect is perfectly stable — it is a statistical artifact of having selected on an extreme value, not a behavioral shift. In practice both can act on the same launch, which is why realized lift so often lands well below the tested lift.

A Worked Example: How Big Is the Exaggeration?

The theory is easier to trust with numbers. The two simulations below are illustrative — they use simple binomial conversion data and a standard two-proportion z-test — but the effect they show is general.

First, the pure-selection case. Suppose you test ten variations against a control, and none of them is actually different: every arm converts at a true 5.00%. With 20,000 visitors per arm, pick the best-looking variation and record its measured lift.

import numpy as np
rng = np.random.default_rng(7)

n_variants, n, p = 10, 20_000, 0.05
best_lifts, any_significant = [], 0

for _ in range(20_000):
    ctrl = rng.binomial(n, p) / n
    variants = rng.binomial(n, p, size=n_variants) / n
    lifts = (variants - ctrl) / ctrl
    se = np.sqrt(2 * p * (1 - p) / n)
    z = (variants - ctrl) / se
    best_lifts.append(lifts[np.argmax(variants)])
    if np.any(np.abs(z) > 1.96):
        any_significant += 1

print(round(np.mean(best_lifts) * 100, 2), "% avg lift of best variant")
print(round(any_significant / 20_000 * 100, 1), "% of experiments show a 'winner'")

+4.85 % avg lift of best variant
28.4 % of experiments show a 'winner'

There is no real effect anywhere in this design, yet the best-looking variation shows an average lift of nearly +5%, and in about 28% of these all-null experiments at least one variation crosses the 95% significance threshold. Selecting the maximum manufactures a winner out of pure noise.

Second, the underpowered-but-real case. Now give the treatment a genuine +5% relative lift (5.00% to 5.25%), but run it underpowered at 15,000 visitors per arm. Compare the average measured lift across all such tests against the average measured lift among only the significant winners.

p0, rel, n = 0.05, 0.05, 15_000
p1 = p0 * (1 + rel)
all_lifts, winner_lifts = [], []

for _ in range(40_000):
    c = rng.binomial(n, p0) / n
    t = rng.binomial(n, p1) / n
    lift = (t - c) / c
    se = np.sqrt(c * (1 - c) / n + t * (1 - t) / n)
    all_lifts.append(lift)
    if (t - c) / se > 1.96:
        winner_lifts.append(lift)

print(round(np.mean(all_lifts) * 100, 2), "% avg over all tests")
print(round(np.mean(winner_lifts) * 100, 2), "% avg among significant winners")

+5.06 % avg over all tests
+13.16 % avg among significant winners

Averaged over every test, the estimate is unbiased — it recovers the true +5%. But conditioned on winning, the average measured lift is +13% — a 2.6x exaggeration of the real effect. The test is only about 16% powered, so the only runs that reach significance are the ones that got lucky, and their luck is baked into the reported number. If you forecast next quarter's revenue on that +13%, you will miss by roughly a factor of three.

How Peeking and Multiple Comparisons Make It Worse

Two common practices amplify the curse well beyond the baseline above.

Peeking and optional stopping. If you watch a running test and stop the moment it crosses significance, you have added a second selection filter on top of the first. You are not sampling the effect once; you are sampling it repeatedly and stopping at whichever look happened to be highest. Early stopping catches the effect at a local peak of its noisy trajectory, which inflates the estimate further — and, with a fixed-horizon test, also inflates the false-positive rate. This is the core reason continuous monitoring requires sequential statistics rather than a fixed 95% threshold checked whenever you feel like it.

Multiple comparisons. Every extra variation and every extra metric is another draw you can select the maximum from. Testing one hypothesis and reporting it is honest; testing twenty and reporting the best is the multiple comparisons problem, and it inflates both the false-positive rate and the magnitude of whatever you crown as the winner. The defense is to fix your primary metric in advance and to correct the significance threshold for the number of comparisons you actually make.

How to Guard Against the Winner's Curse

You cannot eliminate the winner's curse — selection is the whole point of experimentation. You can shrink it, measure it, and stop being fooled by it.

Power tests properly

The single most effective safeguard is adequate power. A well-powered test detects real effects without depending on a lucky draw, which collapses the exaggeration ratio toward 1. Size the sample for a realistic minimum detectable effect rather than an optimistic one, and get there by running longer or by reducing variance. CUPED variance reduction is a direct lever here: lower variance means higher power at the same sample size, which means smaller Type M inflation on whatever wins.

Do not stop at the first significant peek

Decide your analysis method before you start, and hold to it. If you need to monitor continuously, use a method built for it — Optimizely's Stats Engine uses always-valid inference so that peeking does not invalidate the result. Stopping a fixed-horizon test at the first green result is the fastest way to ship an inflated winner.

Read the interval, not the point estimate

The reported lift is a point estimate — the most fragile number on the results page. The confidence or improvement interval tells you the range the true effect plausibly occupies. Plan around the lower bound of that interval, not the headline number. If the interval runs from +1% to +23%, your defensible forecast is closer to +1%. A wide interval is itself a warning that the test is underpowered and the point estimate is exaggeration-prone.

Correct for multiple comparisons

When you test many variations or track many metrics, adjust for it. Optimizely's Stats Engine applies false discovery rate control — a tiered Benjamini-Hochberg procedure — so that adding secondary metrics does not quietly manufacture significant results, while the primary metric keeps its power. Whatever platform you use, name a single primary metric before launch and treat everything else as diagnostic.

Shrink extreme estimates

Shrinkage estimators and Bayesian methods pull extreme observed effects back toward a prior mean, in proportion to how noisy they are. A variation with a huge but uncertain measured lift gets pulled down hard; a precise, moderate estimate barely moves. The shrunken estimate is a better forecast of realized impact than the raw point estimate precisely because it undoes some of the selection bias. If your platform offers a Bayesian readout, the posterior mean is already doing this work for you.

Validate winners with a holdout

The cleanest way to learn the realized effect of a winner is to keep measuring it against a group that never got it. A holdout re-estimates the effect in production, on fresh traffic, after the selection has already happened — so it captures exactly the regressed-to-the-mean value. Optimizely's global holdouts hold back a small slice of traffic (typically up to 5%) from all shipped winners, letting you quantify the cumulative, realized uplift of your program rather than the sum of inflated test-time lifts. This is also the honest number to report to leadership.

What Optimizely Gives You Out of the Box

Several of the safeguards above are built into Optimizely's statistics, though none of them is a magic eraser for selection bias — they reduce specific contributors to it.

• Sequential testing via Stats Engine. Optimizely's Stats Engine produces always-valid results so you can monitor continuously and stop early without inflating the false-positive rate, according to Optimizely's statistical analysis methods overview and its explanation of how Stats Engine calculates likelihoods. This addresses the peeking amplifier, not the underlying selection bias.

• False discovery rate control. Stats Engine uses a tiered Benjamini-Hochberg procedure to correct for multiple metrics and variations, treating the primary metric separately to preserve its power, per Optimizely's false discovery rate control documentation. This limits how often multiple comparisons manufacture a false winner.

• Improvement intervals. The results page reports an interval around each estimate, not just a point lift — the range you should actually plan around.

• Global holdouts. A persistent control group that never sees shipped winners, which Optimizely describes as a way to measure the cumulative, realized impact of your program, per the global holdouts documentation.

Treat these as tools that reduce specific sources of inflation. They do not remove the fundamental fact that choosing the winner biases its estimate — only replication and holdouts measure what survived that choice.

Winner's Curse vs Related Pitfalls

The winner's curse is often confused with other reasons a launch underperforms its test. They have different causes and different fixes.

The tell for the winner's curse specifically is that the test was clean, significant, and underpowered or one of many, and the realized effect is a fraction of the tested effect without any behavioral explanation.

Frequently Asked Questions

Is the winner's curse the same as regression to the mean?

They are two descriptions of one phenomenon. The winner's curse describes the bias at the moment of selection: the estimate you choose is inflated because you chose it for being extreme. Regression to the mean describes what happens when you measure that extreme value again: it drifts back toward the truth. If you select a winner (winner's curse) and then re-measure it in production (regression to the mean), you see the same shrinkage from both angles.

Does a statistically significant result rule out the winner's curse?

No — significance is what creates it. The significance threshold is the filter that selects for large, lucky estimates, especially in underpowered tests where only an inflated draw can cross the line. Significance tells you the effect is probably not zero; it does not tell you the reported magnitude is accurate. In low-power tests, significant results are the most exaggerated ones.

How much smaller will my real lift be?

It depends almost entirely on power. A well-powered test with a decisive result may be exaggerated by only a few percent. A barely significant result from an underpowered test can be inflated two- to threefold, as the simulation above shows, and in extreme cases can even have the wrong sign. As a working rule, the wider your improvement interval and the closer your result sat to the significance line, the more you should discount the point estimate.

Can a bigger sample size fix it?

Larger samples are the most reliable mitigation because they raise power, which shrinks the exaggeration. They do not eliminate the bias — any time you select the best of several noisy estimates, some inflation remains — but a well-powered test moves the exaggeration ratio close enough to 1 that the point estimate becomes a usable forecast. Reducing variance with a technique like CUPED buys the same benefit without more traffic.

Should I re-run every winning test?

Not every one, but validate the ones you plan to build a forecast or a roadmap on. A holdout is cheaper than a full re-run: keep a small percentage of traffic on the old experience after you ship the winner, and read the realized effect over the following weeks. That measurement already incorporates regression to the mean, so it is the number worth reporting and planning against.

Key Takeaways

Selecting the winning variation biases its measured lift upward, and the inflation is worst exactly when the test is underpowered or one of many comparisons. Regression to the mean then pulls the realized effect back down after launch. The result is a systematic gap between tested lift and shipped lift that has nothing to do with broken tracking. Guard against it by powering tests properly, refusing to stop at the first significant peek, planning around the lower bound of the interval rather than the point estimate, correcting for multiple comparisons, shrinking extreme estimates, and — most decisively — validating winners against a holdout so you measure what actually survived the selection.