T-Test vs Z-Test: Which One an A/B Test Should Use

A t-test and a z-test compute the same ratio — an observed difference divided by its standard error — and differ only in what they assume about the noise term. That one difference decides which test an A/B test should use, and the answer depends less on the "n over 30" rule most people remember than on whether your metric is a conversion rate or a numeric value. This page is about that choice. For the mechanics of the test itself, the formula and a worked example are on the two-sample t-test page. For an experiment with more than two variations, see t-test vs ANOVA; for the case where neither test is valid because the metric is too skewed, see the normality assumption.

The one difference between the two tests

Both tests start from the same statistic:

statistic = (mean_B - mean_A) / standard_error

A z-test treats the standard error as a known quantity and compares the statistic with the standard normal distribution. A t-test acknowledges that the standard error was itself estimated from the sample, and compares the statistic with a t-distribution instead.

The t-distribution has heavier tails than the normal distribution, and the heaviness is controlled by the degrees of freedom, which grow with the sample size. Heavier tails mean a larger statistic is needed to reach the same p-value — the test is being cautious about the fact that its estimate of the noise could itself be off. As the sample grows, the estimate of the standard error gets more precise, the tails thin out, and the t-distribution converges to the normal. By a few hundred observations per group the two are practically identical.

That is the whole story of the "n over 30" rule: it is the point at which the difference between the two reference distributions stops mattering for a 5% threshold. It is not a rule about which test is correct; it is a rule about when the choice stops changing the answer.

When is the standard error actually known?

A z-test is exactly right, not merely approximately right, when the variance of the metric is fixed by something you already know. For a numeric metric such as revenue per visitor, that is never true — the spread of order values is something you can only estimate from the data.

For a conversion rate it is different. A conversion is a 0/1 outcome, and the variance of a 0/1 variable is determined entirely by its rate: p × (1 - p). Once you have an estimate of the rate you have an estimate of the variance for free, and the two-proportion z-test uses that fact directly:

p_pooled = (conversions_A + conversions_B) / (n_A + n_B)
standard_error = sqrt( p_pooled × (1 - p_pooled) × (1/n_A + 1/n_B) )
z = (p_B - p_A) / standard_error

Under the null hypothesis the two groups share one conversion rate, so the variance is pooled from both. This is why the z-test is the conventional test for conversion rates and why every sample-size formula for a conversion metric, including the one in A/B test sample size and statistical power, is built on it.

Strictly, the pooled rate is still an estimate, so the two-proportion z-test is also an approximation. But a conversion metric at experiment scale involves thousands of visitors per group, and at that size the approximation is excellent — the binomial distribution of conversion counts is very close to normal well before then.

The decision rule for A/B tests

Put the two facts together and the rule is short:

The rule of thumb that follows: use a z-test for rates, a t-test for everything else, and do not worry about the distinction once the groups are in the thousands. There is no situation in online experimentation where a z-test on a numeric metric is more correct than a t-test; the most you can say is that at large samples it is not less correct either.

How much the choice changes the answer

The practical difference is the critical value — the statistic needed to reach a two-sided p-value of 0.05:

With ten observations per group, a t-test asks for a statistic 14% larger than a z-test would before calling the result significant. That gap is real: a z-test on such a sample would report significance too often, because it pretends to know a standard error it has only guessed from ten numbers. By a hundred observations the gap is one percent, and at experiment scale it disappears.

This is why the choice is a genuine question for small-sample analyses — a test on a subset of buyers, a B2B experiment with a few hundred accounts, a per-segment cut — and a non-question for a full-traffic conversion test.

Running both tests in Python

statsmodels provides the two-proportion z-test and scipy provides Welch's t-test. Both take raw counts or per-visitor values rather than summary statistics, so the calling code is short.

import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest

# Conversion rate: counts of conversions and visitors per group
conversions = np.array([1080, 1000])   # variation, control
visitors = np.array([20000, 20000])
z, p_z = proportions_ztest(conversions, visitors)
print(f"two-proportion z-test: z = {z:.3f}, p = {p_z:.4f}")

# Revenue per visitor: one value per visitor, zeros included
control = np.loadtxt("control_revenue.csv")
variation = np.loadtxt("variation_revenue.csv")
t = stats.ttest_ind(variation, control, equal_var=False)
print(f"Welch t-test: t = {t.statistic:.3f}, df = {t.df:.0f}, p = {t.pvalue:.4f}")

For the conversion example — 5.4% against 5.0% with 20,000 visitors per group — the pooled rate is 5.2%, the standard error is about 0.00222, and z comes out near 1.80, a two-sided p-value of roughly 0.07. The same data through a Welch t-test on the 0/1 values gives a statistic within a hundredth of that and the same conclusion, which is the convergence described above in action.

What Optimizely does instead

Neither test is what a live experimentation platform reports. Both are fixed-horizon tests: they assume one look at a pre-planned sample size. Optimizely's Stats Engine uses sequential testing so that results can be read at any time without inflating the false-positive rate, and it applies false discovery rate control across the metrics and variations on a results page. The standard errors it works from are the same ones described here — a proportion's variance for conversion metrics, an estimated variance for numeric ones — but the decision rule on top is different.

The t-test-or-z-test question therefore matters most in two places: when you size an experiment before launch, where the conversion-rate formula assumes a z-test and the sample size calculator does that arithmetic for you, and when you re-analyse exported data after an experiment has ended, where you are back to a single fixed-horizon look and the rules above apply. If you are deciding between fixed-horizon, Bayesian and sequential analysis for a specific test, the stats method picker walks through that choice.

The mistakes that actually happen

Applying a z-test to a small numeric sample. The critical-value table shows why: with a few dozen buyers the z-test understates the noise and over-reports significance. If a group has fewer than a hundred or so observations, use the t-test.

Applying a t-test with equal variances assumed. The default in many libraries is Student's pooled version. Use Welch's — equal_var=False in SciPy — which is right whether or not the variances match. The two-sample t-test page explains the difference.

Dropping the zeros from a revenue metric. Revenue per visitor includes every visitor who bought nothing. Removing them changes the metric to revenue per buyer, changes the variance, and changes the answer.

Choosing the test by the p-value it produces. If a z-test says 0.04 and a t-test says 0.06 on the same data, the sample is small and the t-test is the one to believe. The disagreement is the t-test doing its job.

Frequently asked questions

Is a z-test more powerful than a t-test?

Slightly, at small samples, because it uses a narrower reference distribution — but that extra power is borrowed from a false assumption that the standard error is known. At large samples the two have the same power.

Can I use a t-test for a conversion rate?

Yes. A conversion rate is the mean of a 0/1 variable, and Welch's t-test on those 0/1 values gives almost exactly the same result as the two-proportion z-test at experiment sample sizes. The z-test is conventional for proportions, not mandatory.

Does the "n greater than 30" rule mean I need 30 visitors?

No. It describes when the t- and normal distributions stop differing materially, not how many visitors an experiment needs. Sample size is set by the minimum detectable effect and the variance of the metric, which is a separate calculation.

Which test does a chi-square test correspond to?

For a two-group comparison of a conversion rate, a chi-square test of independence on the 2×2 table is mathematically equivalent to the two-proportion z-test: the chi-square statistic is the square of z, and the p-values are identical.