Statistical Significance Calculator for A/B Tests
Enter the visitors and conversions for each variation to get a p-value, a confidence interval on the lift, and a plain verdict on whether the difference is real. Two-tailed by default, and corrected when you test more than one variation.
Results per variation
The first row is the control. Every other row is compared against it.
How to read this, and the traps it avoids
A p-value is the probability of seeing a difference at least this large if the two variations were actually identical. Small p means the data sits awkwardly with “no difference”. It is not the probability that your variation wins, and it says nothing about how big the win is — which is why the confidence interval is shown next to it and is the number worth arguing over.
Three ways significance calculators mislead
- One-tailed by default. A one-tailed test declares significance about twice as easily. This calculator is two-tailed, because you would almost certainly act on a large drop too.
- Ignoring multiple variations. Four variations against one control gives roughly a 19% chance of at least one false positive at 95% confidence. The threshold here tightens automatically.
- Saying nothing about when you looked. A fixed-horizon p-value is only valid if you decided the sample size in advance and looked once. Checking daily and stopping at significance inflates the false-positive rate far above 5% — see sequential testing and the peeking problem.
Decide your sample size before you start with the sample size calculator, confirm the split is not broken with the SRM checker, and if the result is a marginal winner, read the winner’s curse before you forecast the lift into next quarter’s numbers.