How to Calculate A/B Test Sample Size and Statistical Power
TL;DR
Most A/B tests fail before they collect a single visitor. Not because the hypothesis was wrong, but because the test was never large enough to detect the effect the team hoped to find. Sample size is the decision that determines whether your experiment can answer its own question. Get it right and a flat result is genuinely informative; get it wrong and you are guessing dressed up as data. This guide explains how to calculate the number of visitors an A/B test needs, how statistical power fits in, and how to turn that number into a realistic run time. When you want the arithmetic done for you, the traffic estimator runs every calculation on this page automatically.
Why sample size is a planning decision, not a stopping rule
Sample size is set before the test launches, using assumptions about your traffic and the smallest effect worth detecting. It is a design constraint, not a finish line you discover along the way. Teams get into trouble when they invert this: they start a test, watch the results, and stop the moment significance appears. That practice, called peeking, inflates false positives dramatically because a metric wandering across a threshold by chance gets frozen as a "win."
A pre-committed sample size protects you from that temptation. It answers a precise question: given how often visitors convert today and the minimum improvement you would act on, how many people must see each variation before the test has a fair chance of proving the effect is real? Everything downstream, including how long the experiment runs and whether it is even feasible, flows from that single calculation.
The four inputs every sample size calculation needs
Every sample size formula, and every calculator worth using, depends on the same four inputs. Change any one and the required number of visitors moves, sometimes by an order of magnitude.
Baseline conversion rate
The baseline is the conversion rate of your control experience today: the fraction of visitors who complete the goal you are measuring. If 5 out of 100 checkout-page visitors purchase, your baseline is 5%. Baseline matters because conversion data is binomial, and the variance of a proportion depends on the rate itself. Rates near 50% carry the most variance and need larger samples to pin down; very low rates need large samples for a different reason, because the absolute difference you are chasing is tiny. Pull the baseline from a recent, representative window of your own analytics, not from an industry benchmark.
Minimum detectable effect
The minimum detectable effect (MDE) is the smallest improvement you want the test to be able to catch. It is usually expressed as a relative lift: a 10% MDE on a 5% baseline means you want to detect a move to 5.5%. The MDE is the single most powerful lever over sample size, because it enters the formula squared. Halving the MDE roughly quadruples the visitors you need. Choosing it well is a topic in its own right, covered in how to choose a minimum detectable effect; the short version is that the MDE should be the smallest lift that would actually change a business decision, not the lift you are hoping for.
Statistical significance
Statistical significance is your tolerance for a false positive, the risk of declaring a winner when the variation is truly no different from control. It is set by the significance level, alpha. A 95% significance threshold means alpha is 0.05, or a 5% chance of a false positive on a true null. In Optimizely, this is a project-wide setting under Settings > Advanced, and raising it (say from 90% to 95%) requires a larger sample because you are demanding stronger evidence before calling a result, per Optimizely's documentation on the statistical significance setting.
Statistical power
Statistical power is the mirror image of significance. Where significance controls false positives, power controls false negatives: it is the probability that the test detects a real effect of at least your MDE when one genuinely exists. Power is written as 1 minus beta, where beta is the false-negative rate. The industry convention is 80% power (beta of 0.20), meaning that if the true effect equals your MDE, you have an 80% chance of ending with a significant result. Underpowered tests are the quiet killer of experimentation programs, because a flat, inconclusive result gets misread as "the change did nothing" when the truth is "the test could never have seen it."
The sample size formula
For a two-variation test comparing two conversion rates, the visitors needed per variation is:
n = (Z_alpha/2 + Z_beta)^2 * [ p1(1 - p1) + p2(1 - p2) ] / (p2 - p1)^2
Where p1 is the baseline rate, p2 is the baseline improved by your MDE, Z_alpha/2 is the critical value for your significance level (1.96 for 95%), and Z_beta is the critical value for your power (0.84 for 80%, 1.28 for 90%). The denominator, the squared difference between the two rates, is why the MDE dominates everything: shrink it and the required sample explodes.
A worked example
Take a 5% baseline, a 10% relative MDE (so the target rate is 5.5%), 95% significance, and 80% power.
Z_alpha/2 + Z_beta = 1.96 + 0.84 = 2.80
(2.80)^2 = 7.84
p1(1-p1) = 0.05 * 0.95 = 0.0475
p2(1-p2) = 0.055 * 0.945 = 0.051975
numerator = 7.84 * (0.0475 + 0.051975) = 0.7799
denominator = (0.055 - 0.05)^2 = 0.000025
n per variation = 0.7799 / 0.000025 ≈ 31,200
You need roughly 31,200 visitors per variation, about 62,400 in total for a two-variation test. That single number is the difference between an experiment that can answer its question and one that cannot.
A rule of thumb you can do in your head
For the common case of 95% significance and 80% power, the coefficient (1.96 + 0.84)^2 is almost exactly 7.84, and the two variance terms are close enough that you can approximate the per-variation sample as:
n ≈ 16 * p * (1 - p) / (absolute MDE)^2
For the example above, 16 * 0.05 * 0.95 / 0.005^2 ≈ 30,400 per variation, within a few percent of the exact figure. It is a fast sanity check when someone floats a test idea in a meeting: if the rule of 16 says you need six figures of traffic you do not have, you can say so before anyone builds the variation.
How each input changes the visitors you need
The formula is easier to internalize as a set of sensitivities. Each table below holds everything else fixed and moves one input, using 95% significance and 80% power unless noted.
The MDE is the most punishing lever. On a 5% baseline:
Relative MDE | Target rate | Visitors per variation | Total (2 arms) |
|---|---|---|---|
5% | 5.25% | ~122,100 | ~244,200 |
10% | 5.5% | ~31,200 | ~62,400 |
20% | 6.0% | ~8,200 | ~16,300 |
30% | 6.5% | ~3,800 | ~7,600 |
Baseline conversion rate matters almost as much, because rarer events are harder to move detectably. Holding the MDE at a 10% relative lift:
Baseline rate | Visitors per variation | Total (2 arms) |
|---|---|---|
2% | ~80,700 | ~161,400 |
5% | ~31,200 | ~62,400 |
10% | ~14,700 | ~29,500 |
25% | ~4,900 | ~9,700 |
Significance and power move the number too, though less violently. On a 5% baseline with a 10% relative MDE:
Significance / Power | Visitors per variation | Total (2 arms) |
|---|---|---|
90% significance / 80% power | ~24,600 | ~49,200 |
95% significance / 80% power | ~31,200 | ~62,400 |
95% significance / 90% power | ~41,800 | ~83,600 |
99% significance / 80% power | ~46,500 | ~92,900 |
Two lessons fall out of these tables. First, if your traffic is fixed, the only inputs you can realistically trade are the MDE and, to a lesser extent, power. Second, chasing tiny effects on rare events is often simply infeasible, and the honest response is to change the metric or the hypothesis rather than run a test that can never conclude.
From sample size to test duration
A per-variation sample size is only actionable once you convert it into calendar time, because that is what stakeholders actually plan around. The conversion is straightforward:
days = (visitors per variation * number of variations) / daily eligible visitors
If the worked example needs 62,400 visitors in total and the page that hosts the test receives 4,000 eligible visitors per day, the test needs about 16 days of collection. "Eligible" is the word that trips teams up: it is not total site traffic but the visitors who actually enter the experiment after audience targeting, device filters, and page conditions are applied. Overestimate it and you will conclude the test far too early.
Two practical rules protect the result. Run for whole weeks, not arbitrary spans, so that weekday and weekend behavior are represented evenly. And never stop before the planned sample is reached simply because significance appeared early. The deeper treatment of run length, including business-cycle effects, lives in how long to run an A/B test, and the traffic estimator will turn your sample size and daily traffic into a duration in one step.
Putting it together: from inputs to a launch decision
The full workflow is a short chain: gather the four inputs, compute the required sample, convert it to a duration, and then make a go or no-go decision based on whether that duration is acceptable. If it is not, you loop back and adjust the one input you control.
flowchart TD
A[Baseline rate<br/>from analytics] --> D[Required sample<br/>per variation]
B[Minimum<br/>detectable effect] --> D
C[Significance + power<br/>95% / 80%] --> D
D --> E[Convert to duration<br/>via daily eligible traffic]
E --> F{Duration<br/>acceptable?}
F -->|Yes| G[Launch and run<br/>to planned sample]
F -->|No| H[Raise MDE, reduce variants,<br/>or apply variance reduction]
H --> DWhen the duration comes back too long, you have three legitimate moves before you compromise on rigor: accept a larger MDE (you will only detect bigger wins, but faster), cut the number of variations so traffic is not diluted across arms, or reduce variance with a technique like CUPED, which can meaningfully shrink the sample needed without touching your significance or power. Lowering the significance threshold to force a quicker read is not on that list; it trades away the false-positive protection the test exists to provide.
What statistical power protects you from
Power deserves more attention than it usually gets, because the cost of ignoring it is invisible. An underpowered test does not announce itself. It simply returns a flat result, and the team files the idea under "did not work" and moves on, when the reality is that the effect was there and the test was blind to it.
Consider the worked example sized for 80% power. If you instead run it to only half the required sample, your power collapses toward coin-flip territory: a genuine 10% lift now has roughly an even chance of going undetected. You will still see a p-value, still see a confidence interval, and still feel like you ran a real experiment. You did not. Power is what makes a negative result trustworthy. Without enough of it, "no significant difference" means nothing more than "we did not look hard enough," and a program that repeatedly under-powers its tests slowly teaches itself that experimentation does not work.
How the Optimizely Stats Engine changes the picture
Everything above describes fixed-horizon planning: you commit to a sample size in advance and read the result once at the end. This is exactly how Optimizely's Frequentist (fixed horizon) stats engine works. Per Optimizely's documentation on stats engines, that mode performs the full statistical analysis only once all required visitor data is collected, and you define the horizon up front by a target sample size or number of days. If you choose that engine, the sample size you calculate here is the number you plan to.
Optimizely also offers a Sequential stats engine, which is designed for exactly the peeking problem described earlier. It updates results as data arrives and lets you look whenever you like, using false discovery rate control so that repeated peeking does not inflate your error rate. This is the mechanism behind the classic Stats Engine, and it is covered in depth in the Optimizely Stats Engine explainer and sequential testing and the peeking problem.
The two approaches interact in a way worth stating plainly. Sequential testing lets you stop early when an effect is large, but it does not abolish sample size planning. If the true effect is near your MDE, a sequential test can take as long as or longer than a fixed-horizon test to reach significance, because small effects still need large samples to separate from noise. Sizing the test with the formula above tells you the traffic and time to budget; the sequential engine then gives you the option to conclude sooner if the data is unusually clear. Whichever engine you use, Optimizely provides a built-in Sample Size Calculator when you define metrics, and it also surfaces an estimate of visitors remaining before significance on the results page. Guard the run with a sample ratio mismatch check so a skewed split does not quietly invalidate the sample you worked to collect.
Common mistakes when sizing an A/B test
A handful of errors account for most badly sized tests, and all of them are avoidable at the planning stage.
Sizing to the lift you hope for. Teams pick an MDE equal to the improvement they expect, then are surprised when a smaller-but-real effect goes undetected. Size to the smallest effect that would change a decision.
Using total traffic instead of eligible traffic. Audience and page targeting can cut the visitors entering a test to a fraction of site traffic. Duration estimates built on total traffic run short.
Ignoring power entirely. Most casual calculators default to 80% power silently. If you never set it, you never checked whether a null result is trustworthy.
Stopping at first significance. Without a sequential engine, calling the test the moment it crosses the line inflates false positives. Run to the planned sample.
Counting variations as free. Every extra arm splits your traffic and extends the test. An A/B/C/D test needs its sample in each arm, not spread across them.
Key takeaways
Sample size is the single planning decision that decides whether an A/B test can answer its question. It depends on four inputs, baseline rate, minimum detectable effect, significance, and power, and the MDE dominates because it enters the formula squared. Compute the per-variation sample with the two-proportion formula (or the rule of 16 for a quick check), convert it to a duration using eligible daily traffic, and decide go or no-go on that duration. If the number is infeasible, trade the MDE, cut variants, or reduce variance rather than lowering your standards. The fastest way to run all of this is the traffic estimator, which does the arithmetic and the duration in one place.
Frequently asked questions
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest effect you want to detect. As a concrete anchor, a 5% baseline with a 10% relative minimum detectable effect at 95% significance and 80% power needs roughly 31,200 visitors per variation, about 62,400 total for a two-arm test. Lower baselines and smaller effects push that number up quickly. Use the two-proportion formula or the traffic estimator with your own numbers rather than relying on a generic figure.
How long before an A/B test is statistically significant?
Divide the total required sample by the number of visitors who actually enter the test each day. If you need 62,400 visitors and 4,000 eligible visitors enter daily, that is about 16 days. Round up to whole weeks so weekday and weekend behavior are represented. Significance may appear earlier by chance, but with a fixed-horizon test you should still run to the planned sample. See how long to run an A/B test for the full treatment.
What statistical power should I use for an A/B test?
80% is the widely used convention and a reasonable default: it means that if the true effect equals your MDE, you have an 80% chance of detecting it. Raising power to 90% makes the test more sensitive to real effects but increases the required sample by roughly a third. Lower power is rarely worth it, because it quietly turns real effects into inconclusive results.
Can I stop my A/B test as soon as it hits significance?
Not with a fixed-horizon (frequentist) test. Peeking and stopping at the first significant reading inflates your false positive rate well above the nominal level. If you need the freedom to stop early, use a method built for it, such as Optimizely's Sequential stats engine, which applies false discovery rate control. See sequential testing and the peeking problem.
Do I still need a sample size calculation if I use Optimizely's Stats Engine?
Yes. Sequential and Bayesian engines let you stop early when an effect is large, but they do not remove the need to plan. If the true effect is near your MDE, the test still needs a large sample to separate signal from noise, so sizing tells you the traffic and time to budget. The engine then gives you the option to conclude sooner when the data is unusually clear, not a guarantee that it will.