False Discovery Rate Control in A/B Testing

Track one metric on an A/B test and a false positive is a 1-in-20 event at the usual 95% confidence threshold. Track twenty metrics and, under the null hypothesis, you should expect one of them to look like a winner purely by chance. Add a few variations and a segment breakdown, and the number of simultaneous comparisons climbs into the dozens. This is the multiple comparisons problem, and left uncontrolled it turns a rich results page into a machine for manufacturing phantom wins. The false discovery rate is the modern statistical answer to it — a way to keep testing many things at once without letting your rate of false claims run away from you.

This guide explains what the false discovery rate is, how it differs from older corrections like family-wise error control and Bonferroni, how the Benjamini-Hochberg procedure works with a fully worked example, and how Optimizely's Stats Engine applies false discovery rate control automatically across every metric and variation on the results page.

What Is the False Discovery Rate?

The false discovery rate (FDR) is the expected proportion of your "discoveries" — the results you flag as statistically significant — that are actually false positives. If you declare 10 winning metrics and your FDR is controlled at 5%, then on average you expect about 0.5 of those 10 to be a mistake. FDR is a property of the set of significant results, not of any single test.

That framing is the whole point. A traditional p-value controls the error rate of one hypothesis in isolation: a 5% significance level means that if this one variation has no real effect, there is a 5% chance you wrongly call it significant. But an experiment rarely tests one hypothesis. It tests a primary metric, several secondaries, a stack of guardrails, maybe two or three variations, and often a few segments. Every one of those cells is a hypothesis test. The false discovery rate asks a more useful question: across everything I flagged as a discovery, what fraction should I expect to be wrong?

Controlling the FDR lets you keep a wide, informative results page — which you want, because catching harm requires watching many metrics — while bounding how often you chase something that was never really there.

The Multiple Comparisons Problem

The mechanism is pure arithmetic. Suppose every metric on your experiment is truly null (no real difference between variation and control), and you test each at a 5% significance level. The probability that a single test does not produce a false positive is 0.95. The probability that none of m independent tests produces one is 0.95^m, so the probability of at least one false positive is:

P(at least one false positive) = 1 - 0.95^m

m = 1   metric   -->  5.0%
m = 5   metrics  -->  22.6%
m = 10  metrics  -->  40.1%
m = 20  metrics  -->  64.2%
m = 50  metrics  -->  92.3%

At twenty metrics you are more likely than not to see at least one "significant" result even when nothing is happening. This is why a naive results page that runs every metric at a flat 95% threshold is untrustworthy the moment it carries more than a handful of metrics: the headline significance numbers no longer mean what they appear to mean. The problem compounds with variations and segments, because each comparison is another draw from the same lottery.

The instinct to fix this by simply "watching fewer metrics" is the wrong trade. You need guardrail metrics to catch changes that win on conversion but quietly raise refunds or slow the page — that is a deliberately wide net. The right fix is not to look at less, but to adjust the significance bar to account for how much you are looking at.

Family-Wise Error Rate vs False Discovery Rate

There are two different error rates you might try to control, and the choice shapes everything downstream.

The family-wise error rate (FWER) is the probability of making even one false positive across the entire family of tests. Controlling FWER at 5% means there is at most a 5% chance that any of your significant results is wrong. This is a strict, conservative standard — appropriate when a single false claim is catastrophic, such as a drug-approval trial.

The false discovery rate (FDR) is the expected proportion of false positives among the results you called significant. Controlling FDR at 5% means you accept that some individual discoveries may be wrong, as long as, on average, no more than 5% of them are.

For online experimentation, FDR is almost always the better target. You run many tests, most effects are small, and the cost of missing a real winner (low power) is a genuine business cost. FDR control gives you dramatically more statistical power than FWER control while still keeping false claims on a leash. That trade — accept a known, bounded fraction of errors in exchange for detecting many more true effects — is why FDR has become the default for large-scale testing.

The Benjamini-Hochberg Procedure

The Benjamini-Hochberg (BH) procedure, introduced by Yoav Benjamini and Yosef Hochberg in 1995, is the standard method for controlling the false discovery rate. It is refreshingly mechanical.

Given m p-values and a target FDR level q (say 0.10):

1. Sort the p-values in ascending order, so p(1) <= p(2) <= ... <= p(m).

2. For each rank k, compute the BH critical value (k / m) * q.

3. Find the largest rank k for which p(k) <= (k / m) * q.

4. Declare significant every hypothesis from rank 1 up to and including that k — even any whose raw p-value sits above the naive threshold.

The key idea is the sloped threshold. Instead of comparing every p-value to one fixed cutoff, BH compares each ranked p-value to a line that rises with its rank. A p-value only needs to beat (k / m) * q, which is more forgiving for higher ranks when there are many small p-values supporting them. The procedure adapts: when the data contain many strong effects, the effective cutoff loosens; when almost nothing is real, it tightens toward the strict Bonferroni bound.

flowchart TD
    A[Collect m p-values from every metric and variation] --> B[Sort ascending: p1 to pm]
    B --> C[For each rank k compute BH critical value k over m times q]
    C --> D{Find largest k where p_k is at or below its BH critical value}
    D --> E[Reject ranks 1 through k: declare significant]
    D --> F[Ranks above k: not significant]
    E --> G[Expected false positives among rejections stays at or below q]

A Worked Example

Take 10 metrics on an experiment and a target FDR of q = 0.10. Sort the p-values ascending, compute each BH critical value (k / 10) * 0.10 = k * 0.01, and compare:

Rank k   p-value    BH critical (k/m)*q    p <= critical?
  1      0.001         0.010                yes
  2      0.008         0.020                yes
  3      0.012         0.030                yes
  4      0.021         0.040                yes
  5      0.030         0.050                yes
  6      0.041         0.060                yes
  7      0.060         0.070                yes   <-- largest passing rank
  8      0.200         0.080                no
  9      0.350         0.090                no
 10      0.600         0.100                no

Largest k with p(k) <= (k/m)*q  is  k = 7.
Benjamini-Hochberg rejects ranks 1 through 7  ->  7 discoveries.

Notice rank 7: its p-value is 0.060, which is above the naive 0.05 threshold, yet BH still declares it significant. That is the procedure working as designed — the weight of six even-smaller p-values below it earns rank 7 a place. Now compare the same data under a strict Bonferroni correction, which would test every metric against 0.10 / 10 = 0.01: only ranks 1 (0.001) and 2 (0.008) clear that bar, for 2 discoveries instead of 7. Same data, same 10% error budget — BH finds more than three times as many real effects because it spends that budget across the whole set instead of on each test in isolation.

FDR Control vs Bonferroni Correction

The Bonferroni correction is the best-known multiple-comparisons fix: divide your significance level by the number of tests, so with 20 metrics at 5% each metric must clear 0.05 / 20 = 0.0025. It controls the family-wise error rate, and it is trivially simple. It is also badly suited to A/B testing.

Bonferroni's flaw in this setting is that it treats every additional metric as equally threatening and pays for it with a cutoff that shrinks toward zero. With 50 metrics, each must beat 0.001 — so real, moderate effects get thrown out alongside the noise, and your false-negative rate soars. FDR control accepts a small, known fraction of false positives in exchange for keeping the power to detect the effects that matter. For an experimentation program that lives or dies by finding true wins across a broad metric set, that is the correct trade.

How Optimizely's Stats Engine Controls the False Discovery Rate

Optimizely's Stats Engine applies false discovery rate control automatically. Per Optimizely's documentation, the Stats Engine "provides a data-rich view of visitor interactions, includes confidence intervals, and applies false discovery rate control" across the metrics on the Experiment Results page. You do not configure it, sort p-values, or apply a correction by hand — it is built into how significance is reported. For the broader mechanics of how significance and always-valid inference are computed, see How the Optimizely Stats Engine Works.

Two documented properties are worth internalizing:

• Testing more does not raise your false-positive risk. Optimizely states that the "Stats Engine adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." That is the FDR guarantee in plain language — the correction scales with how many comparisons the results page carries, so adding a guardrail does not silently inflate your chance of a phantom winner.

• The primary metric is protected. On the results page, "the primary metric's statistical significance is calculated independently from all other metrics and monitoring goals." The tiering matters: your headline result reaches significance as fast as possible, while secondary and monitoring metrics absorb the multiple-comparisons adjustment. Adding many low-improvement secondary metrics slows their time to significance, but never the primary's.

This is why the Change the statistical significance setting documentation can promise that carrying a wide metric set — including a full complement of guardrail metrics — does not cost you in false alarms. FDR control is what makes a broad, honest results page statistically safe.

Do Not Change Metrics Mid-Experiment

There is one operational rule that follows directly from how FDR control is computed. Optimizely controls the false discovery rate over "metrics currently appearing on the Experiment Results page." Its Edit a metric documentation warns that changing the metrics during a running experiment "alters the number and nature of the hypotheses being tested, which can affect the false discovery rate control calculations," and that "excessive metric changes after an experiment or campaign has started may invalidate Optimizely's guarantees regarding false detection."

The reason is exactly the BH mechanism above: the critical values depend on m, the number of hypotheses. Add or swap metrics after data has started arriving and you retroactively change m, which shifts every threshold and reopens the door to the cherry-picking FDR control was meant to close. Decide your metric set before launch and freeze it — the same discipline that protects you from the winner's curse and from sequential testing peeking traps.

Practical Implications for Your Experiments

Understanding FDR control changes a few concrete habits:

• Keep the metrics you need, not fewer. Because the Stats Engine controls FDR across the results page, you can carry a healthy set of guardrails and secondaries without inflating false positives. Trim metrics for clarity, not out of statistical fear.

• Lock the metric set at launch. Write your primary, secondary, and guardrail metrics into the experiment brief before traffic starts, and do not edit them mid-flight. Changing m mid-experiment undermines the FDR guarantee.

• Read the primary metric first. It is computed independently and reaches significance fastest. Treat secondary and monitoring metrics as supporting evidence whose significance the FDR adjustment deliberately makes more conservative.

• Power the experiment for the metrics that matter. FDR control makes false positives rare, but it cannot rescue an underpowered test from false negatives. Size the experiment properly — see A/B test sample size and statistical power — so you can actually detect the effects you care about.

• Validate the traffic split. No error-rate control matters if the experiment is broken at the instrumentation level. Check for sample ratio mismatch before you trust any significance number on the page.

Frequently Asked Questions

What is a good false discovery rate to target?

For most online experimentation, controlling the FDR at 5% to 10% is standard — the same range as conventional significance levels. A 10% FDR means you accept that, on average, up to one in ten of your flagged discoveries may be a false positive, in exchange for substantially more power to detect real effects. Optimizely ties this to your project-wide statistical significance setting rather than asking you to specify an FDR directly.

Is the false discovery rate the same as the false positive rate?

No, and the distinction is important. The false positive rate (or Type I error rate) is the probability of a false positive among all truly null tests — it looks backward from the ground truth. The false discovery rate is the expected fraction of false positives among the results you declared significant — it looks forward from your discoveries. FDR is the more actionable quantity because it describes the reliability of the conclusions you actually act on.

Why not just use Bonferroni correction?

Bonferroni controls the family-wise error rate by dividing your significance level across all tests, which makes the threshold shrink rapidly as you add metrics. With many metrics it becomes so conservative that it discards real effects along with the noise, driving up false negatives. Benjamini-Hochberg controls the false discovery rate instead, adapting the threshold to the data and preserving far more power — a better fit for the many-metric reality of A/B testing.

Does tracking more metrics in Optimizely increase my false positive risk?

No. Optimizely's documentation states that the Stats Engine "adjusts statistical significance for every metric and variation, so testing more does not raise the risk of false positives." The false discovery rate control scales with the number of comparisons on the results page. Adding metrics can lengthen the time for secondary metrics to reach significance, but it does not inflate your overall false-positive risk.

Why does changing metrics mid-experiment break FDR control?

Benjamini-Hochberg critical values depend on the number of hypotheses being tested. When you add, remove, or edit metrics on a running experiment, you change that number retroactively, which shifts the thresholds the Stats Engine already applied and can invalidate Optimizely's guarantees against false detection. Decide your metric set before launch and keep it fixed for the life of the experiment.