T-Test vs ANOVA: Analyzing an Experiment with More Than Two Variations

An A/B/n experiment — one control and two, three or five variations — raises a question the two-sample t-test was not built to answer: with several groups, which comparisons do you run, and how do you keep the false-positive rate where you planned it? Analysis of variance (ANOVA) is the textbook answer, and this page explains what it does, when it is the right tool, and why an experimentation platform reports something different. It is about the more than two variations problem only. Which of the t-test and z-test to use for a single comparison is on t-test vs z-test; whether either test is valid on a skewed metric is on the normality assumption.

What each test asks

A two-sample t-test asks whether two means differ: is variation B's average different from control's? It returns one statistic, one p-value, one confidence interval, and it is complete in itself.

A one-way ANOVA asks whether at least one of k means differs from the others: across control, B, C and D, is there any difference at all? It returns one statistic, F, and one p-value — and it deliberately does not say which groups differ. It is an omnibus test: a single gate on the whole family of comparisons.

The two are not competitors for the same job. With exactly two groups they are the same test in different clothing: a one-way ANOVA on two groups gives F = t² and an identical p-value. The difference appears at three groups and grows from there.

Why several t-tests inflate the error rate

The instinct with four groups is to run a t-test for each pair. Four groups have six pairs, and each t-test at a 5% threshold carries its own 5% false-positive chance. If nothing is happening in any group, the probability that at least one of the six comparisons comes back "significant" is not 5%:

comparisons    P(at least one false positive at alpha = 0.05)
    1               5.0%
    3              14.3%
    6              26.5%
   10              40.1%
   15              53.7%

This is the multiple comparisons problem, and it is the same arithmetic that false discovery rate control exists to manage across metrics. For an A/B/n test the family is the set of variation-versus-control comparisons rather than the set of metrics, but the inflation is identical. A team that runs six uncorrected t-tests and ships the one that crossed 0.05 has, more than a quarter of the time, shipped noise.

There are two ways out. Correct the individual tests so that the family-wise rate stays at 5% — Bonferroni, Holm, or the FDR procedures. Or run one omnibus test first, so that the family is only examined when there is evidence something differs. ANOVA is the second route.

How one-way ANOVA works

ANOVA splits the total variation in the metric into two parts: variation between the group means, and variation within each group around its own mean. If the treatments do nothing, the group means differ only by sampling noise and the between-group part is about the same size as the within-group part. If a treatment works, the between-group part is larger.

MS_between = sum over groups of  n_i × (mean_i - grand_mean)^2  /  (k - 1)
MS_within  = sum over groups of  (n_i - 1) × s_i^2               /  (N - k)

F = MS_between / MS_within

k is the number of groups, N the total number of visitors, n_i and s_i² the size and variance of group i. MS_between measures how far the group means are scattered, weighted by how many visitors each mean is based on. MS_within is the pooled variance of the metric — the same quantity Student's t-test pools across two groups, extended to k. Their ratio, F, is compared with an F-distribution with k - 1 and N - k degrees of freedom to produce the p-value.

The reading is direct: F near 1 means the groups' means are no more spread out than their internal noise predicts; F well above 1 means at least one group has moved.

ANOVA says something differs, not what

A significant F tells you the null hypothesis of "all means equal" is rejected. It does not say whether B beat control, or whether B and C differ from each other, or whether D is the loser. Those answers come from post-hoc tests, run after the omnibus test and corrected for the number of comparisons they make.

Two post-hoc procedures cover most experiments:

• Dunnett's test compares each variation against a single control and corrects for exactly those k - 1 comparisons. This is the natural post-hoc test for an A/B/n experiment, where the question is "which variations beat the control?" and the variation-versus-variation pairs are not of interest.

• Tukey's HSD compares every pair of groups and corrects for all k(k - 1)/2 of them. It is the right choice when every pair matters, such as a pricing test where the question is whether $19 differs from $24 as much as whether either differs from $29.

Dunnett's correction is less severe than Tukey's because it makes fewer comparisons, so on an A/B/n design it has more power to find the variation that actually won. Using Tukey by default on a control-anchored design throws that power away.

Running it in Python

SciPy provides the omnibus test and both post-hoc procedures.

import numpy as np
from scipy import stats

# one array per group, one value per visitor, zeros included
control = np.loadtxt("control.csv")
var_b = np.loadtxt("variation_b.csv")
var_c = np.loadtxt("variation_c.csv")
var_d = np.loadtxt("variation_d.csv")

# Omnibus test: does any group mean differ?
f, p = stats.f_oneway(control, var_b, var_c, var_d)
print(f"one-way ANOVA: F = {f:.3f}, p = {p:.4f}")

# Each variation vs control, corrected for three comparisons
dunnett = stats.dunnett(var_b, var_c, var_d, control=control)
for name, stat, pval in zip(["B", "C", "D"], dunnett.statistic, dunnett.pvalue):
    print(f"{name} vs control: t = {stat:.3f}, adjusted p = {pval:.4f}")

# Every pair, corrected for all six comparisons
tukey = stats.tukey_hsd(control, var_b, var_c, var_d)
print(tukey)

scipy.stats.dunnett arrived in SciPy 1.11 and tukey_hsd in 1.7; older environments can use statsmodels.stats.multicomp.pairwise_tukeyhsd for the all-pairs case. The input convention is the same as for the t-test: one value per visitor, non-converters included as zeros.

The assumptions ANOVA shares with the t-test

ANOVA inherits the t-test's assumptions and adds one. The observations must be independent, which random assignment provides. The sampling distribution of each group mean must be approximately normal, which is the normality assumption and is governed by the skewness of the metric and the sample size in each group, not by the shape of the raw histogram. And — the addition — the groups should have similar variances, because MS_within pools them.

That last assumption is the same one that makes Student's t-test inferior to Welch's, and it fails for the same reason: a variation that changes the mean of a revenue metric usually changes its spread too. When variances differ, especially with unequal group sizes, classical ANOVA reports the wrong error rate. Welch's ANOVA — the k-group generalisation of Welch's t-test — drops the equal-variance requirement; it is available as statsmodels.stats.oneway.anova_oneway with use_var="unequal", and it is the version to prefer for the same reason Welch's t-test is.

What experimentation platforms do instead

Open the results page for a four-variation experiment in Optimizely and you will not find an F statistic. You will find each variation compared against the control, each with its own lift, confidence interval and significance. That is a Dunnett-shaped question answered by a different method: the platform's Stats Engine runs each variation-versus-control comparison as a sequential test, so the results can be read at any time, and controls the multiplicity across variations and metrics with false discovery rate control rather than with a family-wise correction.

The reason is practical. An omnibus test answers a question nobody on the team is asking — "is something happening somewhere?" — and gates every decision behind a single fixed-horizon look. A control-anchored, always-valid comparison per variation answers the question the experiment was launched to answer and can be read as data accrues. FDR control is the multiplicity method that scales to dozens of metrics without destroying power, which Bonferroni and Tukey do not.

So ANOVA belongs in offline analysis: a re-analysis of exported experiment data at its planned end, an academic-style write-up, or a design — such as a full-factorial multivariate test — where main effects and interaction effects are the point and a factorial ANOVA is the standard way to estimate them. On a live A/B/n experiment, the platform's per-variation comparisons with multiplicity control are the right reading.

Which test to use

Frequently asked questions

Is ANOVA more powerful than a t-test?

With two groups they are identical. With more groups the comparison is not like-for-like: ANOVA tests a broader hypothesis. For the question an A/B/n test actually asks — which variations beat the control — Dunnett's test is more powerful than either an uncorrected ANOVA-then-Tukey pipeline or a set of Bonferroni-corrected t-tests, because it corrects for exactly the comparisons of interest.

Can I skip ANOVA and go straight to Dunnett's test?

Yes. Dunnett's procedure controls the family-wise error rate on its own; the omnibus test is not a prerequisite. Running ANOVA first is a convention from designs where the omnibus question is genuinely interesting.

Does adding variations to an experiment slow it down?

Yes, in two ways. Traffic is divided across more arms, so each arm reaches its required sample size later; and the multiplicity correction raises the bar each comparison must clear. Both are real costs and the second is often forgotten when a fifth variation is added "just to see".

What is two-way ANOVA?

An ANOVA with two factors — say, headline copy and button colour — that estimates the main effect of each factor and their interaction. It is the analysis behind a full-factorial multivariate test. A one-way ANOVA has a single factor with several levels, which is the A/B/n case.