Effect Size in A/B Testing: Absolute vs Relative Lift

Effect size is the magnitude of the difference your A/B test actually measures — how much better (or worse) the variation performed, expressed as a number rather than a yes/no verdict on significance. A p-value tells you whether an effect is real; the effect size tells you whether it is worth anything. Two tests can both reach 95% significance while one moves conversion by half a percentage point and the other moves it by eight. Only the effect size distinguishes them, and it is the number your stakeholders actually care about when they ask "so what did we win?"

This guide covers what effect size means in experimentation, the difference between absolute and relative lift, why a large-looking effect size can still be a rounding error to the business, how to standardize effect size for conversion rates with Cohen's h, and how effect size feeds directly into sample size and reporting. It deliberately stays close to the practical decisions you make when planning and reading a test. Where it touches the smallest effect you plan to detect, it hands off to the companion guide on the minimum detectable effect rather than re-covering it — the two concepts are easy to conflate and this article keeps them separate on purpose.

What effect size means in A/B testing

Effect size is the estimated magnitude of the true difference between your variation and your baseline. In an A/B test it is whatever quantifies "how big is the difference" for your metric: the gap in conversion rate, the change in revenue per visitor, the shift in average order value. It is a property of the result, measured after the fact, not a threshold you set beforehand.

It helps to separate three questions that beginners tend to collapse into one:

• Is there an effect? Answered by statistical significance (the p-value or, in a sequential framework, the significance level Stats Engine reports).

• How big is the effect? Answered by the effect size — the point estimate of the lift.

• How uncertain is that size? Answered by the confidence interval around the effect size.

Significance alone is a trap. With enough traffic, a trivially small effect becomes statistically significant, because significance measures how confidently you can rule out zero, not how far from zero you are. A test on millions of sessions can declare a 0.1% relative lift a "winner" that is real but commercially meaningless. Effect size is the antidote: it forces the conversation back to magnitude. This is why mature experimentation programs report the effect size and its interval together with significance, never significance on its own.

Absolute vs. relative effect size

The same result can be described two ways, and confusing them is the single most common effect-size error in practice. Suppose your baseline converts at 5% and your variation converts at 6%.

Absolute effect size (percentage points)

The absolute effect size is the raw arithmetic difference between the two rates: 6% − 5% = 1 percentage point (pp). Absolute lift is unambiguous and additive, which makes it the right unit for forecasting revenue: if 1pp of conversion on 200,000 monthly visitors is worth a known amount, you can multiply it out directly. The pitfall is that a "1 point" gain means something completely different at a 2% baseline than at a 50% baseline.

Relative effect size (percent lift)

The relative effect size expresses the change as a fraction of the baseline: (6% − 5%) / 5% = 20% relative lift. This is the figure most experimentation tools headline. Optimizely's results page reports exactly this: its Improvement column is defined as "the relative improvement in conversion rate for the variation over the baseline as a percentage" — so a baseline of 5% rising to 10% is shown as a 100% improvement, not a 5-point one (Optimizely Experiment Results page).

The two framings diverge sharply as the baseline changes, so always state which one you mean:

Every row except the last shares the same absolute lift while the relative lift ranges from 2.5% to 50%. A headline of "20% lift!" is impressive at a 5% baseline and unremarkable at a 40% one. When you read or quote an effect size, pin down the unit before you react to the number.

Why raw effect size is not business impact

A large relative effect size is not the same as a large business impact, and treating them as interchangeable leads teams to ship winners that move no money. Three gaps separate the reported effect size from the value that lands in a P&L.

First, the effect size is a point estimate wrapped in uncertainty. The test measures 20%, but the true effect could plausibly be anywhere inside the confidence interval — perhaps 4% to 36%. If that interval is wide, the "20%" is a midpoint you should not bank on. Reporting the interval, not just the point, is what keeps forecasts honest.

Second, the metric may not be the money. A lift on click-through, add-to-cart, or a micro-conversion does not automatically flow to revenue; downstream steps in the funnel can absorb it. This is precisely why programs pair a primary decision metric with guardrail metrics — so a healthy-looking effect size on one metric is checked against harm elsewhere before it is called a win.

Third, base rates and reach cap the value. A 30% relative lift on a page that 500 people a month see is worth far less than a 2% lift on your checkout. Effect size has to be multiplied by volume and by the economic value of the action before it becomes impact. The discipline is to translate every effect size into an absolute, annualized figure — expected incremental conversions or revenue — and let that drive the ship/no-ship decision. Relative lift is how you compare tests; absolute impact is how you value them.

Standardized effect size: Cohen's h for proportions

Absolute and relative lift are scale-dependent, which makes them awkward for power calculations and for comparing effects across metrics with different baselines. A standardized effect size rescales the difference into units that are comparable regardless of the baseline. For conversion rates (two proportions), the standard measure is Cohen's h, which uses an arcsine (variance-stabilizing) transformation so that the same h implies the same detectability at any baseline.

Computing Cohen's h

Cohen's h is the difference of the arcsine-transformed square roots of the two proportions:

Cohen's h for two conversion rates

  phi(p) = 2 * arcsin( sqrt(p) )        # the variance-stabilizing transform
  h      = | phi(p1) - phi(p2) |        # the standardized effect size

Worked example: baseline p1 = 0.05, variation p2 = 0.06
(the same 5% -> 6% result from earlier: 1 pp absolute, 20% relative)

  phi(0.06) = 2 * arcsin(sqrt(0.06)) = 2 * arcsin(0.24495) = 2 * 0.24747 = 0.49493
  phi(0.05) = 2 * arcsin(sqrt(0.05)) = 2 * arcsin(0.22361) = 2 * 0.22551 = 0.45103
  h         = |0.49493 - 0.45103|     = 0.0439

Interpretation (Cohen's conventional benchmarks):
  h ~ 0.2  small
  h ~ 0.5  medium
  h ~ 0.8  large

  h = 0.044  ->  a VERY small standardized effect.

The lesson lands immediately: a 20% relative lift that sounds substantial is a Cohen's h of about 0.044 — far below even the "small" benchmark of 0.2. That is not a mistake in the arithmetic; it is the reality of low-baseline conversion optimization. Small proportions produce small standardized effects, which is exactly why conversion tests need large samples. Here is the same calculation in Python:

import math

def cohens_h(p1: float, p2: float) -> float:
    """Standardized effect size for two proportions."""
    phi1 = 2 * math.asin(math.sqrt(p1))
    phi2 = 2 * math.asin(math.sqrt(p2))
    return abs(phi1 - phi2)

h = cohens_h(0.05, 0.06)
print(round(h, 4))  # -> 0.0439

For continuous metrics such as revenue per visitor or average order value, the analogous standardized effect size is Cohen's d — the difference in means divided by the pooled standard deviation — but the principle is identical: standardize so the effect is comparable across metrics and directly usable in a power calculation.

Why standardization matters for planning

Because Cohen's h is defined so that equal h means equal statistical detectability, it plugs straight into a two-proportion power formula. The number of visitors you need per variation is approximately:

Approximate sample size per variation (two-sided test)

  n  ~=  (z_alpha/2 + z_beta)^2  /  h^2

For 95% significance (z = 1.960) and 80% power (z = 0.842):

  (1.960 + 0.842)^2 = 2.802^2 = 7.849

  n  ~=  7.849 / h^2

With h = 0.0439 (our 5% -> 6% test):

  n  ~=  7.849 / 0.0439^2  =  7.849 / 0.001927  ~=  4,070 visitors PER variation

Detecting a 5%-to-6% move at conventional significance and power takes on the order of 4,000 visitors per arm — a concrete, defensible number that falls straight out of the standardized effect size. This makes explicit the point most teams learn the hard way: sample size is governed by effect size, and small effects are expensive to detect.

How effect size drives sample size

The relationship above, n ∝ 1 / h², is the hinge between effect size and experiment cost. Required sample size grows with the inverse square of the effect size, so smaller effects are punishingly more expensive to detect. Halving the effect size you want to catch roughly quadruples the traffic you need; cutting it to a third needs about nine times the traffic. This is the same non-linear penalty covered in depth under sample size and statistical power, viewed from the effect-size side.

flowchart LR
  A["Choose the smallest effect<br/>worth detecting (planning)"] --> B["Standardize it<br/>(Cohen's h or d)"]
  B --> C["Power formula<br/>n proportional to 1 / effect^2"]
  C --> D["Required sample size<br/>per variation"]
  D --> E["Run the test"]
  E --> F["Observe the actual<br/>effect size + interval"]
  F --> G["Decision:<br/>ship, iterate, or drop"]

The left half of that flow — choosing the smallest effect worth detecting and turning it into a sample size — is a planning activity. The right half — observing the effect that actually occurred — is a measurement. They use the same units and the same formula, which is exactly why they get confused. The next section pins down the difference.

Effect size vs. minimum detectable effect

Effect size and the minimum detectable effect (MDE) are measured in the same units and linked by the same math, but they are opposite ends of the experiment lifecycle, and mixing them up produces badly designed tests.

• The MDE is a decision you make before the test. It is the smallest effect you want the experiment to be able to reliably catch — a design lever you set to fix the sample size. It is an input, a target, a floor on sensitivity.

• The effect size is a measurement you take after the test. It is the magnitude the experiment actually observed — an output, an estimate, wrapped in a confidence interval.

flowchart LR
  subgraph Plan["Before the test — planning"]
    M["Minimum detectable effect (MDE)<br/>smallest effect worth catching<br/>= an INPUT you choose"]
  end
  subgraph Run["After the test — measurement"]
    O["Observed effect size<br/>the lift actually measured<br/>= an OUTPUT you estimate"]
  end
  M -->|"determines sample size<br/>and test duration"| Run
  O -->|"compare against the MDE<br/>you committed to"| D["Was the result<br/>big enough to matter?"]

The clean mental model: you pick an MDE to size the test, then you read the effect size to judge the result. A well-powered test is one whose observed effect size, if it equals or exceeds the MDE you chose, will show up as significant. If your observed effect size lands below your MDE, that is not a measurement error — it means the true effect may be smaller than the smallest effect you decided was worth powering for, and an inconclusive result is the honest outcome. Do not retrofit a smaller MDE after the fact to make a small effect look detectable; that is peeking's cousin and it inflates false positives. For the mechanics of stopping rules and why post-hoc adjustments are dangerous, see sequential testing.

How to report effect size with confidence intervals

An effect size reported as a bare point estimate invites false precision. The correct unit of reporting is the effect size plus its confidence interval, which communicates both the magnitude and the uncertainty in one line. Optimizely's Stats Engine is built around exactly this pairing: its results page shows the Improvement (the relative effect size) alongside a Confidence Interval that, in Optimizely's words, "measures uncertainty around improvement" and narrows as more data arrives (New A/B Results page Summary tab).

The interval is also how significance and effect size are reconciled into a single read:

• Interval entirely above zero — a statistically significant positive effect. The whole plausible range of the true effect is a gain.

• Interval entirely below zero — a statistically significant negative effect. The variation is a loser.

• Interval straddling zero — not yet conclusive. Zero lift remains plausible, so you cannot rule out "no effect" (Optimizely Experiment Results page).

Reading the interval changes decisions. An effect size of "+18% [+2%, +34%]" and one of "+18% [−5%, +41%]" have the same headline but opposite meanings: the first rules out zero, the second does not. When you report a result, quote the interval, state the units (absolute or relative), and translate the point estimate into an absolute business figure so stakeholders can size the prize without over-trusting the midpoint.

Frequently asked questions

Is effect size the same as statistical significance?

No. Significance answers whether an effect is distinguishable from zero; effect size answers how large that effect is. They are independent — a tiny effect can be highly significant given enough traffic, and a large effect can be non-significant in a small sample. Always report both, because each answers a question the other cannot.

Should I report absolute or relative effect size?

Report both, and label them. Use relative lift (percent) to compare tests and to headline results, since it is baseline-independent in framing. Use absolute lift (percentage points) to forecast business impact, because percentage points multiply cleanly by traffic and value. Quoting only one — especially only the relative figure — is how a trivial gain gets oversold.

What is a "good" effect size in A/B testing?

There is no universal threshold; it depends on your baseline and the cost of the change. Cohen's benchmarks (h ≈ 0.2 small, 0.5 medium, 0.8 large) are a rough guide, but conversion optimization routinely deals in effects far below "small" because low baselines compress the standardized scale. A better test than any benchmark is economic: does the absolute, annualized impact exceed the cost of building and maintaining the change?

Why does a big relative lift give a small Cohen's h?

Because Cohen's h is computed on an arcsine transform of the raw proportions, not on the relative change. At low baselines the proportions are close together in transformed space even when their ratio is large, so a 20% relative lift from 5% to 6% yields h ≈ 0.044. This is not a flaw — it correctly reflects that such an effect is hard to detect and needs a large sample.

How does effect size relate to the MDE I set in a calculator?

The MDE is the effect size you plan for; the observed effect size is what you get. You choose an MDE to size the test, then compare the effect size the test measures against that MDE. If the observed effect size meets or beats the MDE, a well-powered test will call it significant. See the minimum detectable effect guide for choosing that planning value deliberately.