Power Analysis in R with the pwr Package

An experiment that is too small cannot find the effect it was run to find, and no amount of careful analysis afterwards recovers it. A power analysis is the calculation that sizes the experiment before it starts, and in R the standard tool for it is the pwr package. This page is the implementation: which pwr function to call, how to make it solve for whatever you are missing, and how to read what comes back. If the question you have is how much traffic a test needs, A/B test sample size and statistical power answers it without any code. If it is what power is — how alpha and beta trade against each other, and why 80% became the convention — this page will not re-derive it. This page assumes you already know why you are doing the calculation and want to run it in R.

Installing pwr and the idea that makes it click

pwr is on CRAN and has no dependencies beyond base R.

install.packages("pwr")
library(pwr)

Every function in the package shares one design, and it is the whole trick: a power analysis relates four quantities, and the function solves for whichever one you leave out.

• sample size — n

• effect size — d, h, w, r, f, depending on the test

• significance level — sig.level, the false-positive rate you accept

• power — the probability of detecting the effect if it is really there

Give any three, set the fourth to NULL (or simply omit it), and pwr returns it. That is why there is no separate "sample size function" and "power function" in the package: they are the same function asked a different question.

pwr.t.test(d = 0.2, sig.level = 0.05, power = 0.80, n = NULL)  # solve for n
pwr.t.test(d = 0.2, sig.level = 0.05, n = 500, power = NULL)   # solve for power
pwr.t.test(n = 500, sig.level = 0.05, power = 0.80, d = NULL)  # solve for effect size
pwr.t.test(n = 500, d = 0.2, power = 0.80, sig.level = NULL)   # solve for alpha

Exactly one argument may be NULL. Leave two out and the function stops with an error rather than guessing.

pwr.t.test: sample size for a continuous metric

Use this for a metric that is a number per user — revenue per visitor, items per order, minutes on site. The effect size is Cohen's d: the difference between the two group means divided by the pooled standard deviation.

pwr.t.test(d = 0.2, sig.level = 0.05, power = 0.80, type = "two.sample")

#>      Two-sample t test power calculation
#>
#>               n = 393.4057
#>               d = 0.2
#>       sig.level = 0.05
#>           power = 0.8
#>     alternative = two.sided
#>
#> NOTE: n is number in *each* group

Three details in that output decide whether the number is usable. type = "two.sample" is the A/B shape — two independent groups; the alternatives are "one.sample" and "paired", and both would give a smaller, wrong answer here. alternative defaults to "two.sided", which is what you want unless you have genuinely decided in advance to ignore a negative result. And the note is not decoration: n is per group, so 393.4057 rounds up to 394 in each arm and the design needs 788 users in total.

Solving for power instead of n

The more common real question is the other direction: the traffic is what it is, so what can the test actually detect?

pwr.t.test(n = 500, d = 0.2, sig.level = 0.05, type = "two.sample")$power
#> [1] 0.8848

pwr.t.test(n = 1000, d = 0.15, sig.level = 0.05, type = "two.sample")$power
#> [1] 0.9181

Solving for the effect size you can detect

Leaving d out turns the calculation into a minimum detectable effect, which is usually the number worth putting in front of a stakeholder.

pwr.t.test(n = 500, sig.level = 0.05, power = 0.80, type = "two.sample")$d
#> [1] 0.1776

A d of 0.178 means the test can find a difference of about one-sixth of a standard deviation. If revenue per visitor has a standard deviation of $21, that is a detectable difference of roughly $3.73 per visitor — and seeing that number is usually the moment somebody realises the test as planned cannot answer the question.

Solving for the significance level

The fourth quantity is the one most people forget is solvable. Fix n, d and power, and the question becomes how strict the test can afford to be.

pwr.t.test(n = 500, d = 0.2, power = 0.80, sig.level = NULL, type = "two.sample")$sig.level
#> [1] 0.02053619

With 500 users per group and d = 0.2 in hand, 80% power is reached at an alpha of about 0.021 rather than 0.05 — the surplus traffic can be spent on a lower false-positive rate instead of on a smaller detectable effect, which is worth knowing when several tests are running against the same audience. sig.level is the one argument that has to be written out as NULL: it defaults to 0.05, so omitting it asks for a calculation at 5%, not for the function to solve for it.

Reading the returned object

pwr functions return an object of class power.htest. Printing it gives the block above, but the object is a plain list, so every field is available by name.

result <- pwr.t.test(d = 0.2, sig.level = 0.05, power = 0.80, type = "two.sample")

names(result)
#> [1] "n" "d" "sig.level" "power" "alternative" "note" "method"

ceiling(result$n)        # users per group, rounded up
#> [1] 394
2 * ceiling(result$n)    # users in total
#> [1] 788

That is what makes the package scriptable. A sensitivity table — power across a range of sample sizes, or sample size across a range of effects — is one sapply over the field you care about.

effects <- c(0.1, 0.15, 0.2, 0.3)
data.frame(
  d = effects,
  n_per_group = ceiling(sapply(effects, function(d) {
    pwr.t.test(d = d, sig.level = 0.05, power = 0.80, type = "two.sample")$n
  }))
)
#>      d n_per_group
#> 1 0.10        1571
#> 2 0.15         699
#> 3 0.20         394
#> 4 0.30         176

Halving the effect you want to catch roughly quadruples the traffic you need. That relationship is the single most useful thing a power analysis tells a team.

pwr.2p.test: the A/B case with a conversion rate

Most web experiments compare two proportions, not two means, and that is pwr.2p.test. Its effect size is not a difference in rates — it is Cohen's h, the difference between the arcsine-transformed proportions, and ES.h() computes it.

ES.h(p1 = 0.036, p2 = 0.030)
#> [1] 0.03362181

The transformation exists because a fixed difference in rates is not equally hard to detect everywhere: moving 1% to 2% is a far larger statistical effect than moving 50% to 51%, even though both are "one percentage point". Passing a raw difference where pwr expects h is the most common way to get an answer from this package that is wrong by an order of magnitude.

A worked example on a 3% baseline

Take a checkout with a 3% conversion rate and a target of a 20% relative improvement — 3% to 3.6%.

h <- ES.h(p1 = 0.036, p2 = 0.030)

pwr.2p.test(h = h, sig.level = 0.05, power = 0.80)

#>      Difference of proportion power calculation for binomial distribution
#>      (arcsine transformation)
#>
#>               h = 0.03362181
#>               n = 13886.57
#>       sig.level = 0.05
#>           power = 0.8
#>     alternative = two.sided
#>
#> NOTE: same sample sizes

13,887 visitors per group, so about 27,800 in total. Ask the same question about a more realistic 15% lift and about a 10% lift, and the cost of precision becomes obvious:

lifts <- c(0.10, 0.15, 0.20)
data.frame(
  relative_lift = lifts,
  n_per_group = ceiling(sapply(lifts, function(lift) {
    pwr.2p.test(h = ES.h(0.03 * (1 + lift), 0.03), sig.level = 0.05, power = 0.80)$n
  }))
)
#>   relative_lift n_per_group
#> 1          0.10       53182
#> 2          0.15       24165
#> 3          0.20       13887

Doubling the ambition from a 10% lift to a 20% lift cuts the traffic requirement by almost a factor of four.

What the traffic you have will actually detect

Run it the other way when the sample size is fixed by how much traffic the page gets in a sensible test window.

pwr.2p.test(h = ES.h(0.036, 0.030), n = 6000, sig.level = 0.05)$power
#> [1] 0.4529

At 6,000 visitors per group this test has a 45% chance of detecting a real 20% lift — worse than a coin flip. A power analysis that returns a number like this has done its job: it has told you not to run the test as designed. The sample size calculator does the same arithmetic without R if you want to sanity-check the result.

pwr.2p2n.test is the same calculation for unequal group sizes, which is what you need for an unbalanced split or when comparing against a much larger existing control.

pwr.2p2n.test(h = ES.h(0.036, 0.030), n1 = 20000, n2 = 6000, sig.level = 0.05)$power
#> [1] 0.5567

pwr.chisq.test: more than two variations, or a contingency table

A chi-square test compares an entire table of counts, which covers an A/B/n experiment with several variations and any comparison of a categorical outcome across groups. Its effect size is w, and the degrees of freedom describe the table: (rows - 1) * (columns - 1).

# Three variations x converted/not converted: df = (3 - 1) * (2 - 1) = 2
pwr.chisq.test(w = 0.1, df = 2, sig.level = 0.05, power = 0.80)

#>      Chi squared power calculation
#>
#>               w = 0.1
#>               N = 963.4689
#>              df = 2
#>       sig.level = 0.05
#>           power = 0.8
#>
#> NOTE: N is the number of observations

Note the difference from the two functions above: N here is the total across all cells, not the count per group. Divided across three variations that is about 322 users each. Conventionally w of 0.1 is a small effect, 0.3 medium and 0.5 large, and unlike d or h there is no simple business quantity that maps onto it — which is why this function is the one most often run backwards, from the sample size you have to the effect it can detect.

pwr.chisq.test(N = 5000, df = 2, sig.level = 0.05, power = 0.80)$w
#> [1] 0.0439

Choosing the inputs, not just the function

The functions are the easy half. The inputs are where a power analysis goes wrong, and pwr will compute confidently from bad ones.

• The effect size is a decision, not an estimate. It is the smallest improvement worth shipping, not the one you hope for. cohen.ES() returns the conventional small, medium and large values for each test, and they are a last resort — a placeholder for a domain judgement nobody has made yet.

• Never take the effect size from the experiment you are analysing. Post-hoc power computed from the observed effect is a deterministic function of the p-value and tells you nothing new. If you must reason about a finished test, compute the effect the design could have detected, as in the $d example above.

• Convert n into a duration before you accept it. Visitors per group divided by daily eligible traffic gives days; anything under a full week will not survive the weekly seasonality of most sites.

• Match the function to the metric. A conversion rate is pwr.2p.test, a per-user number is pwr.t.test, a table of counts is pwr.chisq.test. Running a proportion through pwr.t.test by pretending it is continuous is a common and expensive mistake.

pwr also covers correlations (pwr.r.test), one-way ANOVA (pwr.anova.test), general linear models (pwr.f2.test) and one-proportion tests (pwr.p.test) — all with the same leave-one-argument-NULL interface, so everything above transfers directly.