Statistical Power Analysis: How to Run One Before You Launch

A power analysis is the calculation you run before an experiment starts, to find out whether the experiment you are planning can actually detect the effect you care about. This page is the procedure: what the calculation takes in, how to run it, how to read the number that comes back, and what to do when it tells you the test you wanted is not feasible. If the question you have is how much traffic a test needs — the sizing answer on its own, with the conversion-rate arithmetic worked through — A/B test sample size and statistical power is the page for it. If you want to run the same calculation in code rather than read it off a calculator, power analysis in R with the pwr package shows the function calls; if the answer comes back infeasible and you need the ways out, how to increase statistical power works through the levers one at a time. This page is about the calculation itself and what its answer means.

What statistical power actually is

Power is the probability that your test detects an effect, given that the effect is really there. A test with 80% power run against a true 5% lift will come back statistically significant about 80 times out of 100, and will miss it the other 20.

That framing matters because power is conditional. It says nothing about whether your feature works. It describes the instrument, not the thing being measured: a test with 30% power is a thermometer that reads correctly less than a third of the time, and the reading it gives you on any single run tells you very little.

Two error rates are in play, and power is one of them.

• Alpha (α) — the false-positive rate. The probability of calling a winner when nothing changed. Conventionally 0.05.

• Beta (β) — the false-negative rate. The probability of missing a real effect. Power is 1 − β.

An underpowered test does not fail loudly. It returns "not significant", which reads exactly like "no effect" and is usually filed as one. That is the specific failure a power analysis exists to prevent: the experiment that was never capable of answering its own question, reported as evidence that the idea did not work.

The four quantities, and why fixing three fixes the fourth

Every power calculation relates the same four quantities:

• α, the significance level you will test at

• power, the detection probability you want

• effect size, the smallest difference worth detecting — the minimum detectable effect

• n, the sample size per variation

These are not four independent choices. They sit on one surface, and fixing any three determines the fourth exactly. There is no combination in which you get a small sample, a small detectable effect, a low false-positive rate and high power all at once.

That constraint is what makes the calculation useful, because it can be asked in any of four directions, and the direction you ask in should match the decision you are actually making.

The second row is the one teams underuse. Most sites have a traffic level they cannot change in the next quarter, so "what can I detect with what I have" is a more honest question than "how much traffic do I need", and it is the same calculation turned around.

Choosing a target power, and why 0.80 is a convention

0.80 is the number almost every calculator defaults to. It comes from Jacob Cohen's work in the behavioural sciences, where he proposed it as a reasonable floor on the argument that a false negative is roughly four times less costly than a false positive — which, with α at 0.05, gives β at 0.20. It is a convention with a rationale, not a property of statistics.

Treat it as a decision with a cost on each side.

• Raise it to 0.90 or 0.95 when a miss is expensive: a change you will not get a second chance to test, a migration decision, or a result that will shut down a line of work. The extra power is paid for in sample size — going from 0.80 to 0.90 costs roughly 30% more traffic at the same effect size.

• Keep it at 0.80 for routine iteration, where a missed small win reappears in the next test and the cost of the miss is one cycle.

• Going below 0.80 is rarely defensible. At 0.50 power you are flipping a coin about whether a real effect shows up, and a "no effect" conclusion from that test carries almost no information.

The other half of the choice is the effect size, and this is where power analyses go wrong most often. The minimum detectable effect is not your forecast of what the change will do — it is the smallest result that would change your decision. If a 1% lift would ship and a 0.5% lift would not, then 1% is your MDE. Picking an optimistic number here is how a test comes back sized comfortably and then fails to resolve anything.

Running the calculation

For a conversion-rate metric, the standard two-proportion calculation takes four inputs and returns the sample size per variation.

baseline conversion rate   p1 = 3.0%
minimum detectable effect       = 10% relative  ->  p2 = 3.3%
significance level         alpha = 0.05 (two-sided)
target power                    = 0.80

n per variation ≈ 2 * (z(1-alpha/2) + z(power))^2 * p_bar * (1 - p_bar) / (p2 - p1)^2
                ≈ 2 * (1.96 + 0.84)^2 * 0.0315 * 0.9685 / (0.003)^2
                ≈ 53,200 visitors per variation

Three details decide whether that number is right.

• Relative versus absolute effect. A 10% relative lift on a 3% baseline is 0.3 percentage points absolute. Calculators disagree about which one they want, and entering 10 where 0.3 was expected understates the required sample by two orders of magnitude. Always check which the field means.

• One-sided versus two-sided. Two-sided is the default and the safe choice. A one-sided test buys you a smaller sample by declaring in advance that you do not care about a loss, which is almost never true of a production change.

• Variations count. The formula above sizes one comparison. A test with three variations against a control is three comparisons, needs the sample per variation, and needs a correction to α if you intend to act on any of the three.

The same calculation for a continuous metric — revenue per visitor, items per order, session length — replaces the conversion rates with a mean and a standard deviation, and needs the standard deviation from your own historical data. Revenue metrics are heavily skewed, so the variance is large and the required sample is usually several times what the same lift would cost on a conversion rate.

Run the calculation in whatever tool you already trust: a calculator, a spreadsheet, or the pwr package in R if you want it scripted and reproducible alongside the rest of your analysis. If you want to know where that expression comes from rather than take it as given, the sample size formula derives it term by term.

Reading the answer

The output is a sample size per variation, and it is not yet a decision. Convert it into the two things you will actually be asked about.

n per variation      = 53,200
variations           = 2 (control + treatment)
total sample needed  = 106,400 visitors

eligible traffic     = 9,000 visitors/day into the tested experience
                     = 63,000 visitors/week

runtime              = 106,400 / 9,000 ≈ 12 days  ->  round up to 14 (two whole weeks)

Round the runtime up to a whole number of weeks. Weekday and weekend traffic behave differently, and a test stopped after ten days is weighted towards whichever days it happened to cover.

Two things the answer does not say, which are worth stating plainly because they are frequently read into it:

• It is not a stopping rule. Reaching the calculated sample size is the earliest honest point at which to look at the result, not a guarantee of significance. Continuous peeking at a fixed-horizon test inflates the false-positive rate well beyond the α you sized for; if you need to look early, use a sequential testing method designed for it.

• It is not a post-hoc verdict. Calculating "observed power" from the effect you measured after the fact is a well-documented mistake — it is a deterministic function of the p-value and adds no information. Power is a property of the design, computed before the data exists. If a finished test was underpowered, the honest statement is that the test could not answer the question, not that the effect was absent.

When the answer says the test is not feasible

Often enough the calculation returns a runtime of five months. That is a useful result, and it is the point of running the analysis before the experiment rather than after. Read it as a statement about the design, not about the idea: at the traffic you have, an effect of the size you specified would not reliably show up, so the experiment cannot answer the question it was set.

What does not work is running the test anyway at the traffic you have and hoping. That produces a result that is unlikely to be significant if the effect is real, and — when it is significant despite being underpowered — an effect size that is exaggerated, because only the largest sampling fluctuations clear the significance bar in a small test.

There are ways out, and the two the calculation itself points at are visible in the table above: the required sample scales with the inverse square of the effect, so raising the MDE moves the number further than anything else, and reducing the variance of the metric moves it without any extra traffic at all. Which of those is available to you, and what each costs, is a question in its own right — how to increase statistical power works through the full set of levers.

If none of them gets you to a feasible test, the correct output of the power analysis is a decision not to run it. Ship the change on judgement, or pick a different question. Either is better than spending five months on an experiment that was never going to resolve.

A pre-launch checklist

Before the test goes live, the power analysis should have produced explicit answers to all six of these:

• The primary metric, chosen before the test and not after.

• The baseline rate for that metric, measured on the population the test will actually run on.

• The MDE, justified as the smallest result that would change the decision.

• α and whether the test is one- or two-sided, with the number of variations accounted for.

• Target power, with a reason if it is not 0.80.

• The runtime in whole weeks, and the date the test will be read.

Write those six down with the test. When someone asks six weeks later why the experiment ran as long as it did, or whether a flat result means the change did nothing, that record is the answer.