Two-Sample T-Test for A/B Testing: How the Statistic Behind Your Results Works

Every A/B testing platform reports a p-value and a confidence interval for a numeric metric such as revenue per visitor, and behind most of those numbers sits one calculation: the two-sample t-test. This page explains how that test works — what it compares, the formula piece by piece, a worked example, and how to run and read it in Python. It does not decide which test you should run; t-test vs z-test covers that choice. It does not handle an experiment with three or more variations; t-test vs ANOVA does. And it assumes the test is valid for your data; the normality assumption is where to check whether it is.

What a two-sample t-test compares

A two-sample t-test (also called an independent-samples or unpaired t-test) asks one question: are the means of two independent groups different by more than random sampling would explain? In an A/B test the two groups are the control and the variation, the metric is a per-user number such as revenue, items per order, or minutes on site, and the two samples are independent because each visitor was randomly assigned to exactly one group.

The null hypothesis is that the two population means are equal. The test computes how far apart the two sample means are, measured in units of the uncertainty of that difference, and asks how surprising that distance would be if the null hypothesis were true. A large distance relative to the uncertainty gives a small p-value; a small distance gives a large one.

Three things about that framing matter for experiments. The test compares means, so it is a test of average lift, not of medians or of whether any individual user changed. The two samples must be independent; a before-and-after comparison on the same users is a paired design and needs a different test. And the test is two-sided unless you deliberately choose otherwise — the trade-off is covered in one-tailed vs two-tailed A/B tests.

The formula, piece by piece

The version you should use by default is Welch's t-test, which does not assume the two groups have the same variance. Its statistic is:

t = (mean_B - mean_A) / sqrt( s_A^2 / n_A  +  s_B^2 / n_B )

Read it from the inside out.

• mean_A and mean_B are the sample means of control and variation. Their difference is the observed lift in absolute units — dollars per visitor, not a percentage.

• s_A^2 and s_B^2 are the sample variances of each group. Variance is the square of the standard deviation, and it measures how spread out the per-user values are.

• s^2 / n is the variance of a sample mean. Dividing by n is why more visitors make the estimate tighter: the noise in an average shrinks with the square root of the sample size.

• The square root of the sum is the standard error of the difference between the two means. It is the unit the numerator is measured in.

So t is the observed lift divided by the standard error of that lift. A t of 2 means the difference is twice as large as the noise you would expect from sampling alone.

The p-value comes from comparing t with a t-distribution. That distribution has one parameter, the degrees of freedom, which for Welch's test is:

df = ( s_A^2/n_A + s_B^2/n_B )^2
     -------------------------------------------------
     ( s_A^2/n_A )^2 / (n_A - 1)  +  ( s_B^2/n_B )^2 / (n_B - 1)

This is the Welch–Satterthwaite approximation. You will never compute it by hand, but it is worth knowing what it does: it makes the reference distribution wider when the samples are small or their variances are unequal, so a given t is less surprising and the p-value is larger. With the sample sizes an online experiment collects, the degrees of freedom run into the thousands and the t-distribution is indistinguishable from the normal distribution.

Welch's t-test vs Student's t-test

The textbook version, Student's t-test, pools the two variances into one estimate:

s_pooled^2 = ( (n_A - 1) s_A^2 + (n_B - 1) s_B^2 ) / (n_A + n_B - 2)
t = (mean_B - mean_A) / ( s_pooled * sqrt( 1/n_A + 1/n_B ) )
df = n_A + n_B - 2

Pooling is valid only if both groups have the same population variance. In an experiment that assumption is exactly the thing a real treatment effect tends to break — a variation that raises average order value usually also changes its spread. When the variances are equal, Welch and Student give almost the same answer; when they are not, Student's version reports the wrong error rate, most severely when the groups are also unequal in size. There is no situation in A/B testing where Student's test is safer, which is why statistical libraries and platforms default to Welch and why this page does.

A worked example on revenue per visitor

Suppose a checkout experiment ran to its planned sample size with 20,000 visitors in each group. Revenue per visitor came out as follows:

The observed lift is 4.62 - 4.20 = 0.42 dollars per visitor, a 10% relative improvement. Is it real?

s_A^2 / n_A = 21.00^2 / 20000 = 441.00 / 20000  = 0.02205
s_B^2 / n_B = 22.50^2 / 20000 = 506.25 / 20000  = 0.02531
standard error = sqrt(0.02205 + 0.02531) = sqrt(0.04736) = 0.2176
t = 0.42 / 0.2176 = 1.93

With roughly 39,000 degrees of freedom the reference distribution is effectively normal, and a two-sided p-value for t = 1.93 is about 0.054. The 95% confidence interval for the lift is 0.42 ± 1.96 × 0.2176, which is roughly -0.01 to +0.85 dollars per visitor.

The result just misses the conventional 5% threshold. Notice what drove that: a standard deviation five times the mean. Revenue metrics are dominated by a few large orders and many zeros, so the noise term is large and the test needs a lot of visitors to resolve a 10% lift. That is the variance problem CUPED exists to shrink, and it is also why a metric this skewed deserves a check of the normality assumption before the p-value is trusted at all.

Running the test in Python

scipy.stats.ttest_ind runs the test. The equal_var=False argument selects Welch's version; leaving it at the default True gives Student's pooled test.

import numpy as np
from scipy import stats

# per-visitor revenue for each group, one value per visitor
control = np.loadtxt("control_revenue.csv")
variation = np.loadtxt("variation_revenue.csv")

result = stats.ttest_ind(variation, control, equal_var=False)

lift = variation.mean() - control.mean()
se = np.sqrt(variation.var(ddof=1) / len(variation) + control.var(ddof=1) / len(control))
ci = result.confidence_interval(confidence_level=0.95)

print(f"lift per visitor: {lift:.4f}")
print(f"t = {result.statistic:.3f}, df = {result.df:.0f}, p = {result.pvalue:.4f}")
print(f"95% CI: [{ci.low:.4f}, {ci.high:.4f}]")

Two details of the input matter more than the call itself. First, each array must hold one value per visitor, including the zeros for visitors who bought nothing. Dropping the non-converters turns revenue per visitor into revenue per buyer, a different metric with a different answer. Second, ddof=1 in the variance is the sample variance the formula above uses; NumPy's default ddof=0 divides by n rather than n - 1 and understates the noise slightly.

Reading the output

The test returns three numbers, and they answer three different questions.

The t statistic is the lift in standard-error units. It is the raw signal-to-noise ratio and the least useful number to report, because nobody can act on "t = 1.93" without the other two.

The p-value is the probability of a difference at least this large if the true lift were zero. It is not the probability that the variation is better, and it says nothing about how big the effect is. A p-value of 0.054 with 20,000 visitors per group and a p-value of 0.054 with 200 visitors per group describe very different situations.

The confidence interval is the range of true lifts consistent with the data. It is the number to put in front of a decision-maker, because it carries the effect size and the uncertainty together. An interval from -0.01 to +0.85 says the experiment could not rule out zero, but it also could not rule out a lift worth twenty times the cost of shipping. Effect size in A/B testing covers how to compare that interval against the minimum effect you set out to detect, and why a null result with a wide interval is not the same as a null result with a narrow one.

Where the t-test sits in an experimentation stack

A t-test is a fixed-horizon test. It assumes you chose the sample size in advance — the calculation in A/B test sample size and statistical power, or the sample size calculator — collected exactly that many visitors, and computed the statistic once. Looking at the p-value every morning and stopping the first time it dips under 0.05 inflates the false-positive rate several times over; the mechanism and the fix are in sequential testing and the peeking problem.

That is why a platform such as Optimizely does not report a plain t-test on a live experiment. Its Stats Engine uses sequential testing so that every look is valid, and applies false discovery rate control across the metrics and variations on a results page. The t-test remains the right tool for offline analysis of a completed experiment, for a re-analysis of exported data, and for understanding what the platform's numbers mean — the standard error and confidence interval it reports are the same quantities this page derives.

For a conversion rate rather than a numeric metric, the natural test is a two-proportion z-test rather than a t-test, and the reasons are in t-test vs z-test. For an experiment with several variations against one control, running a separate t-test for each pair inflates the error rate, and t-test vs ANOVA explains the alternatives.

Common mistakes

Running one test per metric and reporting whichever is significant. Five metrics at a 5% threshold give a 23% chance of at least one false positive when nothing is happening. Pick a primary metric before launch, and treat the rest as guardrails or apply a multiplicity correction.

Using a paired t-test on independent groups, or the reverse. A paired test needs the same units measured twice, such as the same users before and after. Random assignment produces two independent groups, and that is what the two-sample test is for.

Treating "not significant" as "no effect". A p-value of 0.054 is not evidence of zero lift. Read the confidence interval, and do not reach for a post hoc power calculation to rescue the result — it only restates the p-value.

Trusting the p-value on a heavily skewed metric with a small sample. The t-test relies on the sampling distribution of the mean being approximately normal, and revenue per visitor with a handful of very large orders can violate that at sample sizes that look comfortable. The diagnostics and remedies are on the normality assumption page.

Frequently asked questions

What is the difference between a one-sample and a two-sample t-test?

A one-sample t-test compares one group's mean against a fixed number, such as last quarter's average order value. A two-sample t-test compares two groups' means against each other. An A/B test has two randomised groups, so it uses the two-sample form.

Does the t-test work for conversion rates?

It can — a conversion rate is the mean of a 0/1 variable, and with thousands of visitors the t-test and the two-proportion z-test give nearly identical answers. The z-test is the conventional choice for proportions because the variance of a 0/1 variable is determined by the rate itself. T-test vs z-test walks through when the choice matters.

How many visitors do I need for a two-sample t-test?

There is no fixed minimum; the answer depends on the variance of your metric and the smallest lift you need to detect. Size the test before launch with the sample size calculator, using the standard deviation of the metric from historical data.

Should I use Welch's or Student's t-test?

Welch's. It is correct when the variances are equal and correct when they are not, and the cost of using it when Student's would have been fine is negligible.