The Normality Assumption in A/B Testing: When a T-Test Is Not Valid

A two-sample t-test on revenue per visitor rests on an assumption that the raw data plainly violate: most visitors spend nothing, a few spend a great deal, and the histogram looks nothing like a bell curve. This page is about what that assumption really says, when the violation matters and when it does not, why the Shapiro-Wilk test that most search results recommend is the wrong instrument at experiment scale, and what to do when the test is genuinely not valid for your metric. It is not about which test to choose between a t-test and a z-test — that is t-test vs z-test — nor about experiments with several variations, which is t-test vs ANOVA. It qualifies the two-sample test: it tells you when the p-value that test produces can be trusted.

What the t-test actually assumes

The t-test does not require the metric itself to be normally distributed. It requires the sampling distribution of the difference in means to be approximately normal. Those are different statements, and the difference is the whole subject.

The central limit theorem is what connects them. Average enough independent draws from almost any distribution, and the distribution of that average approaches a normal curve, whatever shape the raw values had. So the question is never "is revenue normal?" — it is not, and it never will be — but "have I averaged enough visitors that the mean behaves as though it were?" For a symmetric metric the answer is yes at a few dozen observations. For a heavily skewed one the answer can be no at several thousand.

When the assumption fails, the failure is specific: the t-distribution no longer describes the statistic, the reported p-value is wrong, and the direction of the error depends on the shape of the skew. For a right-skewed metric such as revenue, the sampling distribution of the mean is itself right-skewed at small samples, confidence intervals are miscentred, and the false-positive rate at a nominal 5% can be materially higher or lower than 5% depending on which tail the effect is in. The number on the results page is then a number, not a probability.

Why revenue per visitor is the hard case

A per-visitor revenue metric has two features that slow the central limit theorem down.

It is zero-inflated: on a site with a 3% conversion rate, 97% of the values are exactly zero. The mean is carried by the 3% who bought.

It is heavy-tailed: among buyers, order values are themselves skewed, and a single bulk order can be fifty times the median. One such visitor landing in the variation group moves its mean by more than the lift the experiment is looking for.

The statistic that captures both is the skewness coefficient — the third standardised moment. A symmetric distribution has skewness 0; a conversion metric at a 3% rate has skewness around 5.5; revenue per visitor on an e-commerce site commonly runs from 5 to 15, and higher where bulk orders exist. That coefficient is what governs how many visitors the mean needs before it behaves normally.

A rule of thumb for skewed metrics

Kohavi, Tang and Xu give a usable rule in Trustworthy Online Controlled Experiments: for the t-test to be reasonably accurate on a metric with skewness s, each group needs at least 355 × s² observations.

skewness   minimum n per group
   1              355
   3            3,195
   5            8,875
  10           35,500
  15           79,875

Two things follow. For a conversion rate, the 0/1 values give a known skewness — (1 - 2p) / sqrt(p(1 - p)) — and at a 3% rate the rule asks for about 11,000 visitors per group, well under what any properly sized conversion test collects, so normality is rarely the constraint there. For revenue per visitor with a skewness of 10, the rule asks for 35,000 visitors per group before the t-test is trustworthy — and that is a condition on validity, separate from and in addition to the sample size needed for statistical power.

The rule is a rule of thumb, derived from simulation rather than theorem, but it is the right order of magnitude and it turns an unanswerable question ("is my metric normal enough?") into a measurement you can make from historical data before launch.

The Shapiro-Wilk test, and why it is the wrong tool at experiment scale

Search for "test for normality" and the first answer is the Shapiro-Wilk test. It computes a statistic W between 0 and 1 that measures how closely the ordered sample values track the values a normal distribution would produce; W near 1 means close to normal, and a small p-value rejects normality.

import numpy as np
from scipy import stats

revenue = np.loadtxt("control_revenue.csv")   # one value per visitor

# Shapiro-Wilk on a small pilot sample
pilot = revenue[:500]
w, p = stats.shapiro(pilot)
print(f"Shapiro-Wilk: W = {w:.4f}, p = {p:.2e}")

# The statistic that actually matters at experiment scale
skew = stats.skew(revenue)
print(f"skewness = {skew:.2f}, rule-of-thumb n per group >= {355 * skew**2:,.0f}")

Three problems make it the wrong instrument for an A/B test.

It tests the wrong thing. Shapiro-Wilk asks whether the raw values are normal. The t-test needs the mean to be normal. Revenue per visitor will fail Shapiro-Wilk at any sample size, and that tells you nothing about whether a t-test on 50,000 visitors is valid — it almost certainly is.

It always rejects at experiment scale. The test's power grows with n, so with thousands of observations it detects departures from normality far too small to affect a t-test. SciPy's implementation warns that its p-value is not accurate beyond 5,000 observations, which is a smaller sample than most experiments collect per group in a day. A test that returns p < 0.001 for every metric on every experiment is not a diagnostic.

It answers a yes/no question when the real question is "how much". Normality is not a switch. The useful quantity is the degree of skew and the sample size relative to it, and Shapiro-Wilk reports neither.

Where the test does belong is small-sample analysis — a pilot with forty accounts, a lab study, a per-segment cut with a few hundred users — where the sample is small enough that the raw shape matters and the test has not yet become oversensitive. Even there, a Q-Q plot shows more than the p-value does.

Diagnostics that work at any sample size

Measure skewness on historical data. One number, computed before the experiment launches, compared against the 355 × s² rule. This is the check that should be part of choosing a primary metric.

Bootstrap the sampling distribution. Resample the historical metric with replacement at the planned group size, compute the mean each time, and look at the histogram of those means. If it is symmetric and bell-shaped, the t-test is safe; if it is visibly skewed, it is not. This is the assumption itself, observed directly.

Run an A/A simulation. Split historical data randomly into two groups of the planned size, run the t-test, and repeat a few thousand times. The fraction of p-values under 0.05 should be 5%. If it is 8%, the test is not valid at that sample size for that metric — and no theory is needed to say so. The same simulation is the standard way to validate a platform's statistics, as described in A/A testing.

Inspect the tail. List the twenty largest values. If one of them is larger than the total lift the experiment could plausibly produce, a single visitor can decide the test, and the remedy is outlier handling rather than more data.

What to do when the assumption fails

flowchart TD
  A[Numeric metric planned as primary] --> B[Compute skewness on historical data]
  B --> C{n per group >= 355 x skewness^2?}
  C -->|Yes| D[Welch t-test is valid as planned]
  C -->|No| E{Can you cap or winsorize the tail?}
  E -->|Yes| F[Cap at a chosen percentile, re-check skewness]
  F --> C
  E -->|No| G{Is the estimand negotiable?}
  G -->|No| H[Bootstrap or permutation confidence interval on the mean]
  G -->|Yes| I[Log transform or Mann-Whitney U on a different question]

Collect enough visitors. If the rule of thumb says 35,000 per group and the traffic plan already delivers 60,000, the assumption holds and nothing else is required. This is the most common resolution and the least discussed.

Cap or winsorize the tail. Replace every value above a chosen percentile — the 99th, or a fixed cap such as $1,000 — with that ceiling. Skewness falls sharply, the rule-of-thumb sample size falls with it, and the metric still measures revenue for all but a handful of visitors. Optimizely's outlier management for numeric metrics does exactly this. The cap must be chosen before the experiment and applied identically to both groups; a cap picked after looking at which group holds the bulk order is a decision made on the outcome.

Reduce variance without changing the metric.CUPED uses pre-experiment data to remove predictable variation from the metric. It does not remove skew, but it shrinks the standard error and often lets the experiment reach the sample size the rule of thumb demands sooner.

Change the question, knowingly. A log transform, or a Mann-Whitney U test, both handle skew — but neither tests the difference in means any more. The log-scale test compares geometric means; Mann-Whitney tests whether a randomly chosen visitor from B tends to exceed one from A. Both are legitimate; both answer a different business question from "did average revenue per visitor rise?" Use them when that other question is the one you actually want answered, not as a way to keep the original claim while switching the test.

Bootstrap or permute the confidence interval. If the estimand must stay the mean and the sample is too small for the rule of thumb, a bootstrap confidence interval on the difference in means, or a permutation test, makes no normality assumption at all. They are slower and less familiar to stakeholders, but they are correct.

What all of these have in common is that they are decided before the experiment launches, from historical data. A t-test that turns out to be invalid after the results are in cannot be repaired by choosing a new test on the basis of which one gives the answer you wanted; that is the same mistake as peeking, moved from the time axis to the method axis.

Frequently asked questions

Does Optimizely's Stats Engine assume normality?

Stats Engine reports that revenue distributions are heavily skewed and applies a skew correction for numeric metrics, alongside the outlier management described above. The assumption discussed here is the one behind the fixed-horizon t-test; the sequential method is built to be robust to the same issue, and its documentation on Stats Engine covers what it does for numeric metrics.

Should I run a Shapiro-Wilk test before every A/B test?

No. At experiment scale it rejects every metric, and it tests the raw values rather than the sampling distribution of the mean. Measure skewness and apply the 355 × s² rule instead; reserve Shapiro-Wilk for small-sample analyses.

Is the normality assumption a problem for conversion rates?

Rarely. The skewness of a 0/1 metric is determined by its rate, and the rule of thumb is comfortably met by any conversion experiment sized for a realistic minimum detectable effect. The exception is a very rare event — a 0.1% rate — where the rule asks for several hundred thousand visitors per group.

Does the assumption apply to a z-test too?

Yes. The two-proportion z-test and the t-test both rely on the sampling distribution of the difference being approximately normal; they differ in how the standard error is treated, not in this assumption. T-test vs z-test covers that difference.