Incrementality Testing: Measuring What Would Not Have Happened Anyway

Incrementality testing answers one question: how much of this outcome would have happened anyway? It is the discipline of measuring the causal contribution of a channel, a campaign or a feature by deliberately withholding it from a randomly chosen group and comparing what the two groups do.

The question sounds obvious. It is asked seriously far less often than it should be, because attribution reporting already produces a confident-looking number for the same thing, and that number is almost always too large.

What incrementality testing measures

Attribution assigns credit for a conversion to the touchpoints that preceded it. Incrementality measures the conversions that would not have occurred without the touchpoint. These are different quantities, and the gap between them is the entire subject.

A retargeting campaign that shows ads to people who already have items in their basket will be credited with nearly every one of their purchases. Most of those purchases were going to happen. The attributed number can be ten times the incremental one, and no amount of care in the attribution model fixes it, because the model has no counterfactual to compare against.

Incrementality testing supplies the counterfactual by randomising. A random holdout group does not see the ads, the feature or the email. The difference between the groups is caused by the treatment, because randomisation makes the groups exchangeable in every other respect.

The output is usually expressed three ways:

• Incremental lift. The percentage difference in the outcome rate between treated and held-out groups.

• Incremental conversions. The lift applied to the treated population — the conversions that exist because of the treatment.

• Incremental cost per acquisition, or iROAS. Spend divided by incremental conversions, which is the only cost-per-acquisition figure worth putting in a budget conversation.

Why attribution over-counts

Three mechanisms produce the gap, and they compound.

Selection. Ad platforms optimise delivery toward people likely to convert. That is what makes them effective, and it is also what guarantees the treated population converts at a higher base rate than average. The platform then reports that difference as its own contribution.

Cannibalisation. A channel takes credit for a conversion another channel would have delivered. Brand search is the standard example: a large share of clicks on your own brand keywords come from people who would have reached the site through the organic result immediately below.

Latency and windows. A 30-day post-view window will attribute a purchase to an impression the buyer never consciously registered. Lengthening the window raises the attributed number without changing anything real.

None of these are errors in the attribution tool. They are consequences of measuring correlation on a population that was selected for correlation.

Three designs, and when each applies

flowchart TD
  A{Can you withhold<br/>from individuals?} -->|Yes| B[User-level holdout<br/>cleanest, needs identity]
  A -->|No| C{Is the treatment<br/>delivered by region?}
  C -->|Yes| D[Geo holdout<br/>fewer units, wider intervals]
  C -->|No| E{Does the ad platform<br/>support it?}
  E -->|Yes| F[Ghost ads / PSA control<br/>platform-run, platform-reported]
  E -->|No| G[No clean design.<br/>Say so rather than<br/>reporting attribution]

User-level holdout. A random share of users is excluded from the treatment entirely. This is the strongest design and the one to use whenever identity allows it: same time period, same conditions, individually randomised. It is the design an experimentation platform is built for.

Geo holdout. Whole markets are assigned to treatment or control. Necessary when delivery is geographic — broadcast, out-of-home, regional campaigns — and considerably weaker, because the number of randomised units drops from millions of users to dozens of regions. Confidence intervals widen accordingly, and the analysis usually needs a matched-market or synthetic-control approach rather than a simple difference of means. The guide to quasi-experimental design for product teams covers those methods and the assumptions they rest on.

Ghost ads and PSA controls. The platform records who would have been served an ad and either serves nothing or serves an unrelated public-service ad. Randomisation is real, but it is performed and reported by the party whose value is being measured. Useful, and worth reading with that in mind.

Sizing an incrementality test

Incrementality tests fail more often for being too small than for being badly designed, and the reason is structural: the effect being measured is the increment, which is a fraction of the total, while the noise is set by the total.

Work the arithmetic before spending anything. If the base conversion rate is 2%, and the campaign genuinely produces a 10% relative lift, the effect to detect is 0.2 percentage points. That requires a sample most single-campaign tests do not have. The sample size calculator will give the number directly; the guide to minimum detectable effect explains why the answer is so unforgiving.

Three practical consequences follow.

Hold out a meaningful share. A 1% holdout produces an unmeasurable control group. Between 10% and 50% is normal, and the cost of the withheld revenue is the price of knowing whether the spend works at all.

Run to a pre-committed end date. Incrementality tests are watched anxiously because real money is being withheld, which makes them the most peeked-at experiments in any organisation. Fix the duration in advance, or use a method that is valid under continuous monitoring — see the note on sequential testing and peeking.

Accept a wide interval honestly. "Between −5% and +25% lift" is a real result. It says the test could not resolve the question, and reporting the midpoint as if it were the answer is how a 10% figure becomes a budget.

Running a user-level holdout in Optimizely

A holdout is an experiment whose control variation does nothing at all. In Optimizely Feature Experimentation, the decision to treat is read from a flag, so the same call both assigns the user and records the exposure.

// The holdout gate. The control group is excluded from the treatment entirely,
// and the decision itself is the exposure record the analysis joins on.
const decision = optimizelyClient.decide('retargeting_holdout', [], userId, {
  country: user.country,
  ltv_decile: user.ltvDecile,
})

if (decision.enabled) {
  await audienceSync.add(userId, 'retargeting_audience')
}

// Fire the outcome for everyone, treated and held out alike. A conversion that
// is only tracked for the treated group cannot be compared to anything.
optimizelyClient.track('purchase', userId, {}, { revenue: order.totalCents })

Two mistakes account for most broken holdout tests. The first is tracking conversions only in the treated arm, which leaves nothing to compare. The second is letting the held-out users receive the treatment through another route — the same audience synced by a second job, or a lookalike audience rebuilt from the treated group. Check the exclusion on the platform side, not only in the code.

For a holdout that stays in place permanently to measure the whole programme rather than one campaign, see the guide to global holdouts in Optimizely.

Reading the result

Report three numbers together, and never the first alone.

{
  "test": "retargeting_holdout_q3",
  "treated": { "users": 480000, "conversions": 10080, "rate": 0.021 },
  "heldOut": { "users": 120000, "conversions": 2280, "rate": 0.019 },
  "incrementalLift": 0.105,
  "confidenceInterval95": [0.021, 0.194],
  "attributedConversions": 7400,
  "incrementalConversions": 1008,
  "spendUsd": 96000,
  "attributedCpaUsd": 12.97,
  "incrementalCpaUsd": 95.24
}

The last two lines are the point of the exercise. Attribution reported a cost per acquisition of about $13. The randomised comparison says the true cost of an incremental conversion is roughly $95. Both numbers are computed correctly; only one of them answers the question a budget decision asks.

Expect the incremental figure to be worse than the attributed one, often by a large factor. A test that finds them equal is more likely to have a leak between the arms than a perfectly clean channel.

Frequently asked questions

How is incrementality testing different from an A/B test?

It is an A/B test, applied to a channel or spend decision rather than a product change. The control arm receives nothing instead of receiving an alternative, and the outcome is usually expressed as incremental conversions and incremental cost rather than as a conversion-rate lift.

What holdout size should I use?

Large enough that the control group can detect the effect you care about — usually 10% to 50% of the population. Work it out from the base rate and the smallest lift that would change the decision, rather than picking a round number.

Can incrementality be measured without withholding anything?

Not directly. Observational estimates need a quasi-experimental design and assumptions that have to be tested and stated. They are the fallback when withholding is impossible, not an equivalent.

Why is my incremental result so much smaller than the attributed one?

Because attribution credits conversions that would have happened anyway, on a population the platform selected for being likely to convert. A large gap is the normal finding, not evidence that the test went wrong.

How long should an incrementality test run?

Long enough to cover a full purchase cycle plus any delayed conversions, with the duration fixed before the test starts. Stopping when the numbers look good inflates the estimate, which is exactly the error the test exists to avoid.