Types of Design of Experiments: Choosing Between Factorial, Fractional, RSM and Taguchi

There is no single design of experiments. There is a catalogue of them, and the useful question is not what they are called but which one your situation calls for. Three inputs decide it almost every time: how many factors you are varying, how many runs you can afford, and whether you are trying to find out which factors matter or trying to find the settings that work best. If you have not met the discipline before, what design of experiments is covers the principles the whole catalogue rests on — randomisation, replication and blocking — and this page assumes them. And if the thing in front of you is a single online split test rather than a multi-factor study, experiment design for A/B testing is the page for planning that well; an A/B test is one design in this catalogue, and the simplest one.

Three questions that pick the design

Answer these before you look at any design name.

How many factors? A factor is something you set deliberately: headline, price, algorithm, temperature, whether the badge shows. One factor is an A/B or A/B/n test. Two to four factors is factorial territory. Five or more and you are screening, which is a different job with different designs.

What is your run budget? In manufacturing a run is a physical batch. Online it is traffic, and the arithmetic is the same: each additional cell needs its own sample, so the number of design points multiplies the sample size a single comparison would have needed. The number itself comes from a power calculation — see A/B test sample size and statistical power.

Screening or optimising? Screening asks which of many factors have any effect at all, and it is happy with a coarse two-level answer. Optimising assumes you already know the factors that matter and asks what values maximise the response, which requires curvature and therefore more than two levels. Designs that are good at one are wasteful at the other.

The flow below is the short version.

flowchart TD
  A[How many factors?] -->|One| B[Completely randomised design: A/B or A/B/n]
  A -->|Two to four| C{Runs affordable?}
  A -->|Five or more| D[Screening: fractional factorial, resolution III or IV]
  C -->|Yes| E[Full factorial: every combination]
  C -->|No| F[Fractional factorial: resolution IV or V]
  D --> G{Known factors, want best settings?}
  E --> G
  F --> G
  G -->|Yes, factors continuous| H[Response-surface design: central composite or Box-Behnken]
  G -->|Yes, goal is robustness to noise| I[Robust-parameter design, Taguchi or a combined array]
  G -->|No, comparison was the goal| J[Stop and analyse]
  B --> K{Known nuisance variable?}
  K -->|Yes| L[Randomised block design: randomise within block]
  K -->|No| J

Completely randomised and randomised block designs

These two are the base of the catalogue, and everything else is built on them.

A completely randomised design assigns every experimental unit to a treatment at random, with no structure imposed. With one factor at two levels it is exactly an online A/B test. It is the right choice when the units are interchangeable enough that you have nothing better to condition on, and it is the design with the fewest ways to go wrong — which is also why the failure that does occur is almost always in delivery rather than in the design, so an SRM check is not optional.

A randomised block design groups units into blocks that are internally similar, then randomises within each block. Agriculturally the block is a strip of field; online it is a segment known at assignment time — device, new versus returning, country, weekday versus weekend. The block-to-block variation is then estimated separately instead of inflating the error term, which raises power without raising traffic.

Use a block design when you can name a variable that (a) you know before assignment, (b) plausibly moves the metric, and (c) you are not interested in for its own sake. If you are interested in it — if the question is whether the treatment works differently on mobile — then it is not a nuisance variable, it is a second factor, and you want a factorial design.

A Latin square blocks in two directions at once with a single run per row-column pair. It is common in agriculture and in physical trials where order and position both matter, and rare online, because it assumes no interactions between the blocking variables and the treatment, which is a strong assumption to buy in exchange for runs you usually have.

Full factorial: the default when you can afford it

A full factorial runs every combination of every factor level. With k factors at two levels, that is 2^k design points: four for two factors, eight for three, sixteen for four.

run   A   B   C
 1    -   -   -
 2    +   -   -
 3    -   +   -
 4    +   +   -
 5    -   -   +
 6    +   -   +
 7    -   +   +
 8    +   +   +

Two properties make this the design to reach for first.

Every main effect is estimated from all the data, not a slice of it. The effect of A is the average response at A = + minus the average at A = -, and both averages use half the runs. Compare that with one-factor-at-a-time, where each factor's estimate rests on a single pair of runs.

And every interaction is estimable. The AB interaction is the difference between A's effect at B = + and its effect at B = -, and the design contains both. This is what OFAT structurally cannot give you: not a weak reading on interactions, but no reading at all.

The cost is that 2^k grows quickly, and online the growth is in traffic. Eight cells need four times the total traffic of a two-cell test to give every cell the same sample size — eight cells at N/2 each is 4N, against N for two cells at N/2 each. The main effects do not cost that much, though: each one is a contrast between four cells and the other four, so it uses the whole sample, and at equal total traffic a 2^3 factorial estimates each of its three main effects about as precisely as a two-cell test estimates its one. The extra traffic buys the interactions and the cell-by-cell comparisons, not the main effects. A multivariate test is exactly this design run on live traffic — multivariate testing vs A/B testing works through when the traffic is worth spending that way.

Full factorial is the right default up to three or four two-level factors. Beyond that, either the run count or the patience runs out first.

Fractional factorial: what the smaller run count costs

A fractional factorial runs a carefully chosen subset — a half, a quarter, an eighth — of the full design. Written 2^(k-p), where p is how many times you have halved it: 2^(4-1) is eight runs of a sixteen-run design.

The subset is not arbitrary. You pick a generator, which assigns an extra factor to an interaction column of a smaller full factorial. For 2^(4-1), take the eight-run 2^3 design above and set D = ABC. The defining relation that follows is I = ABCD, and from it you can read exactly what you have given up.

Aliasing: which effects get confounded with which

Multiply every effect by the defining relation and reduce, using the rule that any letter squared is I. For I = ABCD:

A = BCD        AB = CD
B = ACD        AC = BD
C = ABD        AD = BC
D = ABC

Read that literally. The estimate you compute and label "the effect of A" is really the effect of A plus the ABC-style three-factor interaction BCD; the two are aliased, and no analysis of this data can separate them. Worse for most product work, the column you would call "the AB interaction" is the sum of AB and CD. If your headline-by-price interaction column comes back large, this design cannot tell you whether it is headline-by-price or badge-by-shipping.

That is the whole trade: you did not save runs by getting a rougher answer to the same question, you saved runs by asking a smaller question and agreeing in advance which distinctions you will not be able to draw.

Resolution, and what each level buys

Resolution is the length of the shortest word in the defining relation, written as a Roman numeral, and it summarises the aliasing in one symbol. The rule: in a resolution R design, an effect of order k is aliased with effects of order R - k and above.

• Resolution III (I = ABC, e.g. 2^(3-1)): main effects are aliased with two-factor interactions — A = BC, B = AC, C = AB. Cheap, and only honest when you are screening and genuinely willing to assume interactions are negligible. A large effect here may not be the factor you think it is.

• Resolution IV (I = ABCD, e.g. 2^(4-1)): main effects are clear of two-factor interactions, but two-factor interactions are aliased with each other in pairs. This is the usual screening compromise: you can trust the ranking of the factors, and you can see that an interaction exists without knowing which one it is.

• Resolution V (I = ABCDE, e.g. 2^(5-1), sixteen runs for five factors): main effects are aliased only with four-factor interactions and two-factor interactions only with three-factor ones. In practice that means everything you actually care about is estimable. If you can afford resolution V, you rarely need the full design.

The bet underneath all of this is the sparsity of effects principle: high-order interactions are usually small compared with main effects and two-factor interactions. It usually holds. When it does not, the design has already told you which alias pairs are the suspects, and the standard follow-up is a fold-over — a second block of runs with signs reversed on one or more factors, which breaks the specific aliases you now care about.

The reason to prefer a fractional factorial over an informal "we'll just test a few combinations" is not the run count. It is that the aliasing is written down before the data arrives, so the ambiguity is a known property of the design rather than a surprise in the interpretation.

Response-surface methods: optimising rather than screening

Two-level designs fit a plane. They can tell you that more of A is better, but they cannot tell you how much more, because a straight line has no maximum. Once the goal is a setting rather than a comparison, you need at least three levels of each factor and a model with quadratic terms.

Response-surface methodology (RSM) is the family for that. The workhorse is the central composite design: a two-level factorial core, plus centre points repeated several times, plus axial points set outside the factorial range on each factor's axis. That combination fits a full second-order model — main effects, two-factor interactions and pure quadratic terms — with far fewer runs than a three-level full factorial (3^k grows faster than anyone can pay for). A Box-Behnken design does a similar job in three levels without any corner runs, which matters when the extreme combinations are unsafe or impossible.

The centre points are doing real work: repeated runs at the middle setting estimate pure error, and comparing them with the factorial corners is a formal test of whether curvature exists at all. If there is none, the second-order design is telling you a first-order one would have done.

RSM belongs at the end of a sequence, not the start. The standard progression is screen with a fractional factorial, follow the gradient with a steepest ascent path of single runs until improvement stops, then site a response-surface design around that point and fit the optimum. Reaching for RSM with ten unscreened factors is spending a precision instrument on a job for a shovel.

Online, RSM is genuinely rare, and the reason is worth naming: most product factors are categorical (this layout or that one) rather than continuous, and a quadratic surface over a categorical factor is meaningless. Where it does apply is anything with a real dial — recommendation slate size, discount depth, email frequency, a ranking model's blend weight. Those are the cases where "which of these two" is the wrong question and "how much" is the right one.

Taguchi methods, and why statisticians are divided

Genichi Taguchi's robust-parameter design came out of Japanese manufacturing and reframed the goal. Instead of finding the settings that maximise average performance, find the settings at which performance is least sensitive to the things you cannot control — ambient temperature, component variation, how the customer actually uses the product. The machinery is an inner array of controllable factors crossed with an outer array of noise factors, run over standard orthogonal arrays (L8, L9, L18), with results summarised by a signal-to-noise ratio and interpreted through a quadratic quality loss function that treats any deviation from target as a cost rather than counting parts inside a tolerance band as equally good.

The division among statisticians is specific, and it is worth stating accurately because both halves of it are right.

The engineering contributions are widely accepted. Treating variance reduction as a first-class goal rather than a nuisance, designing for robustness to noise instead of tightening tolerances, and the loss function's argument that being near-but-not-on target has a real cost — these changed industrial practice for the better and are not controversial.

The statistical criticisms, made most forcefully by George Box and by Box and Hunter's successors, are also substantially correct. The signal-to-noise ratios bundle location and dispersion into one number, which can be improved by moving either, so an S/N gain does not tell you what you actually changed; modelling the mean and the variance separately is more informative and no harder. The orthogonal arrays are mostly resolution III, so main effects are aliased with two-factor interactions in a method whose central claim is about the interaction between control factors and noise. And crossing an inner array with an outer array multiplies the run count, when a combined array — one design holding both control and noise factors, analysed with an explicit model containing the control-by-noise interactions — answers the same question in fewer runs and says which interaction produced the robustness.

The practical position most texts settle on: keep Taguchi's goals, use conventional design and analysis to pursue them. If a robustness question turns up in product work — a recommendation model that must not degrade for low-history users, a checkout that must hold up on slow connections — the useful part is the framing. Name the noise factors explicitly, include them in the design, and look at the control-by-noise interactions, which is a combined array by another name.

A design of experiments template you can copy

The selection table first. Find the row that matches your situation.

And the planning template. Fill every line in before the first unit is assigned; the point of writing it down is that the aliasing and the decision rule are properties of the design, and neither can be added afterwards.

Objective          screen | compare | optimise | make robust
Response           the one metric the decision turns on
Factors            name, type (categorical/continuous), levels
Nuisance variables known before assignment -> block on these
Noise factors      not controllable in production -> include if robustness matters
Design             name, e.g. 2^(5-1) resolution V, half fraction
Generators         e.g. E = ABCD
Defining relation  e.g. I = ABCDE
Aliases accepted   which effects you have agreed you cannot separate
Units and blocks   what one run is; how randomisation happens within blocks
Replication        runs or sample per design point, from a power calculation
Centre points      how many, and what they are for (pure error, curvature test)
Randomisation      the mechanism, and who verifies it
Analysis plan      model terms, alpha, and the decision rule, agreed now
Follow-up          fold-over, steepest ascent, or confirmation runs

A design that cannot fill in the "aliases accepted" line is either a full factorial or is not understood yet.

Choosing a design for online experimentation

Traffic is the binding constraint, and it changes which parts of the catalogue are practical.

Most of the time the honest answer is a completely randomised one-factor test, because most sites do not have the traffic to spend on cells. Before designing anything multi-factor, run the numbers: the minimum detectable effect you can reach with your traffic, over a runtime that covers whole business cycles — how long to run an A/B test has the timing argument.

When you do go multi-factor, a full factorial on two or three elements is the design you want, and your platform already offers it as a multivariate test. If the platform offers a "partial factorial" MVT, that is a fractional factorial, and the aliasing above is what it is doing to your results — worth knowing before reading an interaction estimate off it.

Blocking is the most under-used idea here. Stratifying assignment on a segment you know at assignment time costs nothing and buys precision, and CUPED is the same instinct applied to a pre-experiment covariate rather than a category.

Running several unrelated tests at once is not a factorial design and should not be analysed as one — see concurrent A/B tests for what the overlap does and does not do to your results. A factorial design is a deliberate structure in which the combinations are assigned; overlapping tests are separate randomisations that happen to share users.

And when you cannot randomise at all, no design in this catalogue applies. That is a different subject, covered in quasi-experimental design.