What Is Design of Experiments? Principles, Factorial Designs and A/B Testing

Design of experiments — DOE — is the branch of statistics that decides which observations to collect before any are collected, so that the data you end up with can separate the effects you care about from each other and from noise. It is older than online testing by about a century, and it is a different subject from setting up a split test. If what you are doing is planning one A/B test — picking a hypothesis, a primary metric, a variant and a runtime — experiment design for A/B testing is the page for that, and this one is not a replacement for it. This page is about the statistical discipline itself: where it came from, the three principles every DOE rests on, why changing one factor at a time is a weak way to learn anything, and what a factorial design buys you. The place the two subjects meet is at the end, and it is more direct than most practitioners realise: a multivariate test is a factorial design.

Where design of experiments came from

DOE was assembled by Ronald Fisher at Rothamsted Experimental Station in the 1920s and 1930s, working on agricultural field trials. The setting matters, because it explains the shape of the whole discipline.

A field trial is expensive and slow. You get one growing season, a fixed number of plots, and soil that is not uniform — one end of a field drains better than the other. You cannot rerun it next week if the design was poor, and you cannot hold the soil constant. Fisher's response was to stop treating variation as something to be eliminated and start treating it as something to be arranged: assign treatments to plots in a way that makes the unwanted variation cancel out of the comparison you care about, and that lets you estimate how large the leftover noise is from the experiment itself.

His 1935 book The Design of Experiments is where the field gets its name, and where randomisation is first argued for as the thing that licenses the statistical test rather than as a nicety. Everything since — response-surface methods in the 1950s, Taguchi's robust-design work in Japanese manufacturing from the 1960s — is built on the same frame.

The frame transfers to product work almost unchanged. Traffic is a finite resource, your population is not uniform, and a badly designed test cannot be rescued by clever analysis afterwards.

The three principles: randomisation, replication and blocking

Every design in the discipline is some combination of three ideas.

Randomisation

Assign experimental units to treatments by a random mechanism, not by convenience, order or judgement.

Randomisation does two jobs. It breaks the link between the treatment and every characteristic of the unit you did not think to measure — motivation, device, prior exposure, soil drainage — so those characteristics are distributed across your groups in a way that does not favour one of them systematically. And it is what makes the probability statement at the end legitimate: a p-value describes how often a difference this large would arise from the random assignment alone, which is only a coherent question if assignment really was random.

Failing to randomise is the most common way an experiment becomes uninterpretable. Comparing this month's users to last month's, or a group that opted in against one that did not, produces a difference you cannot attribute to the treatment. Even correct randomisation can go wrong in delivery, which is why an SRM check belongs on every online test.

Replication

Run each treatment on more than one unit.

Replication is what gives you an estimate of variability. With one plot per treatment, a difference between them could be the treatment or could be the plot; there is nothing in the data that distinguishes the two. With many units per treatment, the spread within a treatment tells you how large a difference the noise alone can produce, which is the yardstick every test statistic is measured against.

In online experimentation replication is what sample size means, and how much of it you need is a calculation with an answer — see A/B test sample size and statistical power.

Note the distinction the discipline is strict about: replicating a unit is not the same as measuring one unit repeatedly. Ten users seeing one variant once each is ten replicates; one user seeing it ten times is one.

Blocking

Group units into blocks that are internally similar, and randomise within each block.

If you know a source of variation before the experiment runs — field position, machine, day of week, new versus returning visitor — you can stop it inflating your noise estimate. Randomise within the block, and the block-to-block differences are accounted for separately instead of being dumped into the error term.

Blocking is the principle with the least visible presence in online testing and the most under-used. Stratified assignment by a known segment is blocking. So, in effect, is CUPED, which uses a pre-experiment covariate to remove variation that has nothing to do with the treatment. The slogan is Box's: block what you can, randomise what you cannot.

One factor at a time, and why it misses interactions

The intuitive way to study several factors is to vary one, hold everything else fixed, and repeat. This is one-factor-at-a-time — OFAT — and it is the default approach almost everyone reaches for without naming it.

It has two problems, and the second is the important one.

The first is efficiency. To study three factors at two levels each with OFAT, you run a baseline and then one change per factor, and every one of those runs is informative about a single factor. The same total number of observations, arranged as a factorial design, gives you an estimate of all three factor effects — each one averaged over both levels of the other two, so each estimate uses all the data rather than a slice of it.

The second problem is that OFAT cannot see interactions at all. An interaction is when the effect of one factor depends on the level of another: a new headline that helps on mobile and hurts on desktop, a discount banner that lifts conversion only when the free-shipping badge is absent. Hold factor B fixed at its current level while you vary A, and everything you learn about A is conditional on that one level of B — and the design contains no information about whether the answer would have been different at the other level.

This is the single most useful idea to take from DOE into product work. Real systems interact constantly, and OFAT is structurally blind to it. You do not get a weak signal about an interaction and miss it; you get no signal at all.

Factorial designs

A factorial design runs every combination of the factor levels. With k factors at two levels each, that is 2^k runs — the notation the field writes as a 2-to-the-k design.

Three factors at two levels is eight runs, written as a design matrix with - for the low level and + for the high:

run   A   B   C
 1    -   -   -
 2    +   -   -
 3    -   +   -
 4    +   +   -
 5    -   -   +
 6    +   -   +
 7    -   +   +
 8    +   +   +

Because each factor is at its high level in exactly half the runs and its low level in the other half — and that split is balanced across every other factor — the effect of A is estimated from all eight runs, not two. The same eight runs also estimate the three two-factor interactions and the three-factor interaction. Nothing extra was collected to get them.

flowchart TD
    Q["Several factors to study"] --> OFAT["One factor at a time"]
    Q --> FULL["Full factorial: every combination"]
    OFAT --> O1["Each estimate uses a slice of the data"]
    OFAT --> O2["No information about interactions"]
    FULL --> F1["Each estimate uses every run"]
    FULL --> F2["Interactions estimated directly"]
    FULL --> F3["Run count doubles with each factor added"]
    F3 --> FRAC["Fractional factorial: run a balanced subset"]
    FRAC --> R1["Fewer runs"]
    FRAC --> R2["Effects become aliased with each other"]

Fractional factorials, and what you trade

2^k grows fast. Seven factors is 128 runs, ten is 1,024 — and in online testing a run is not a row in a spreadsheet, it is a cell of traffic that needs enough users in it to say anything.

A fractional factorial runs a carefully chosen subset — a half, a quarter, an eighth — keeping the balance that makes the arithmetic work. What you give up is precise: effects become aliased. Two effects are aliased when the design gives them the same pattern of + and - across the runs it does contain, so the data cannot tell them apart. Whatever the estimate is, it is the sum of both.

Designs are graded by resolution, which says how bad the aliasing is:

• Resolution III — main effects are aliased with two-factor interactions. Cheap, and only safe as a screening exercise to find which factors matter at all.

• Resolution IV — main effects are clear of two-factor interactions, but those interactions are aliased with each other.

• Resolution V — main effects and two-factor interactions are all estimable separately; only higher-order interactions are confused.

The choice is a bet, and the bet is that high-order interactions are small — the sparsity of effects principle, which usually holds and occasionally does not. Fractional designs are honest about this in a way that OFAT is not: the aliasing is written down in the design, so you know exactly which conclusions you have chosen not to be able to draw.

Beyond factorials the discipline holds several other design families — screening designs for many factors, response-surface designs that fit curvature to find an optimum rather than compare two levels, Taguchi's robust-parameter designs aimed at reducing sensitivity to conditions you cannot control, and randomised block and Latin square designs built around blocking. Choosing between them is a question in its own right, and it turns on how many factors you have, whether you are screening or optimising, and how many runs you can afford.

Where DOE meets online experimentation

An A/B test is a one-factor, two-level experiment with randomisation and replication and, usually, no blocking. It is the simplest design in the catalogue.

A multivariate test is a full factorial. Testing three page elements at two variants each is a 2^3 design run on live traffic, and every property above applies to it directly: the interaction estimates are the reason to run it, the run count is why it needs several times the traffic of an A/B test, and the fractional-factorial option is exactly what platforms mean by a "partial factorial" MVT. Multivariate testing vs A/B testing works through when that trade is worth making.

Two differences from the agricultural setting are worth naming, because they change what is practical.

Units arrive over time rather than existing up front. You cannot lay out all your plots before starting, so blocking has to be done on characteristics known at assignment time, and the population drifts underneath a long-running test.

And traffic is the binding constraint on run count in a way that plots rarely were. This is why full factorials past three or four factors are uncommon online, and why the discipline's answer — screen with a fractional design, then study the survivors properly — transfers well.

Where you cannot randomise at all, the design question does not go away; it becomes a different subject, covered in quasi-experimental design.

What DOE gives you that A/B experiment design does not

The two subjects overlap and are not the same, and the difference is worth stating plainly.

A/B experiment design is about running one comparison well: a clear hypothesis, one primary metric, a variant that isolates the change, enough traffic, and a decision rule agreed before the data arrives. Almost all of the practical failure modes in product experimentation live there, which is why that is the more useful page for most readers on most days.

Design of experiments is about the structure of the observations across multiple factors: which combinations to run, what each run buys you, which effects you will be able to separate and which you have accepted you cannot. It is the subject you need the moment the question stops being "does this change help" and becomes "which of these things matter, and do they matter together".

Take from it three things. Randomise, replicate and block, in that order of importance. Distrust any conclusion drawn from varying one factor while holding the others fixed, because it is conditional on values nobody chose deliberately. And when you do run several factors at once, know before you start which effects your design can separate — because that is a property of the design, and no amount of analysis afterwards can add it.