One-Tailed vs Two-Tailed A/B Tests: Which Should You Run

Every hypothesis makes a prediction about direction — "this will increase revenue," not "this will change revenue somehow." That prediction has a statistical consequence most teams never examine: whether the test behind it should be one-tailed or two-tailed. Get this choice wrong, or worse, make it after the data arrives, and you can turn a 5% false-positive rate into something closer to 10% without changing a single line of test code. This guide covers what each test actually claims, why Optimizely's Stats Engine defaults to two-tailed, the narrow cases where one-tailed is legitimate, and the much more common case where it is p-hacking wearing a statistics textbook's clothes.

What a two-tailed test actually claims

A two-tailed test's null hypothesis is that the variation and the control produce the same result. Its alternative hypothesis is simply that they differ — in either direction. The test's rejection region is split across both tails of the distribution: half the alpha budget covers "variation beats control," half covers "control beats variation." At a 95% confidence level, that means roughly 2.5% of the probability mass sits in each tail.

This is the conservative, symmetric default, and it is conservative on purpose. It treats "the button color change made things worse" as just as real and just as reportable a finding as "the button color change made things better." A two-tailed test does not presuppose which way the world will move; it only asks whether it moved at all.

What a one-tailed test actually claims

A one-tailed test narrows the alternative hypothesis to a single direction — "the variation is better than control," full stop. The rejection region lives entirely in one tail, so the same 5% alpha budget that was split across two tails in the two-tailed case is now concentrated in one. Practically, that means a one-tailed test reaches significance with a smaller observed effect than a two-tailed test would need, for the identical data and the identical alpha.

That extra sensitivity is the entire appeal, and it is also the entire danger. A one-tailed test is a bet: you are declaring, before you see any data, that a result in the "wrong" direction is not just unlikely but uninteresting — that you would take no action on it either way. If that premise is false, the test is miscalibrated, because it is silently discarding evidence of harm as if it could not exist.

Why Optimizely's Stats Engine defaults to two-tailed

Optimizely's Stats Engine runs two-tailed tests by design, and that is not an arbitrary default — it is required for the false discovery rate control the engine is built around. FDR control needs symmetric tails to keep its error-rate guarantee honest across every metric in a test, including the ones you did not expect to move. A one-tailed override would quietly break that guarantee for the metric it was applied to.

There is a practical reason to like the default even if you ignore the FDR mechanics: most experiments are run precisely because you do not know which way a change will move the metric. If you already knew the direction with certainty, you would not need to test it. Two-tailed testing keeps the platform honest about that uncertainty instead of assuming it away.

When a one-tailed test is legitimate

There is a narrow, defensible case for a one-tailed test, and it has nothing to do with statistical power. It applies only when a result in the "wrong" direction is genuinely inactionable — when you would ship the same decision regardless of whether the metric moved down or stayed flat, because either way the answer is "do not ship."

A migration or performance-safety check is the clearest example: you are testing whether a new checkout implementation is at least as fast as the old one before a full rollout you already intend to do for other reasons. A result showing it is dramatically faster changes nothing about the decision; a result showing it is the same speed changes nothing either. Only "meaningfully slower" would change what you do next, so the interesting region really is one-sided.

Outside cases like that, a one-tailed test is difficult to justify honestly, because most product and marketing decisions are exactly the kind where a surprising result in the "wrong" direction would change what you do.

The failure mode: switching tails after you've seen the data

The far more common way a one-tailed test shows up in practice is not principled design — it is a rescue attempt. A two-tailed test comes back with a p-value of 0.08, just short of significance. Someone notices the effect is in the "expected" direction and proposes re-running the analysis as one-tailed, which roughly halves the p-value and pushes it under 0.05.

This is p-hacking with a statistical vocabulary. The test's design was chosen after seeing the outcome it was meant to be judged against, which means the "5% error rate" claimed for the result was never actually in effect — the true error rate for a decision rule of "run two-tailed, and switch to one-tailed if it doesn't reach significance" is close to double the stated alpha, because you get two chances to declare a win from the same data. The fix is procedural, not statistical: decide the tail before the test launches, write it into the hypothesis, and do not revisit the decision once results are in.

Worked example: the same data, two verdicts

The table below shows why the distinction is not academic. Both rows use the identical observed data — a 4.2% relative lift with a two-sided p-value of 0.062 — and differ only in which test was declared before the experiment ran.

Declared before launch:  Two-tailed
Observed lift:           +4.2%
Two-sided p-value:       0.062
Verdict at alpha=0.05:   Not significant

Declared before launch:  One-tailed ("variation > control")
Observed lift:           +4.2%
One-sided p-value:       0.031
Verdict at alpha=0.05:   Significant

Nothing about the underlying data changed between the two rows — only the pre-registered claim did. That is exactly why the decision has to be made before launch and recorded somewhere a reviewer can check it, rather than chosen after the fact to produce the answer you wanted.

How this interacts with your stopping rule

Tail choice and stopping rule are two separate decisions that compound if you get both wrong. A one-tailed test that is also peeked at repeatedly during the run — checked daily, stopped the moment it crosses significance — stacks a directional shortcut on top of the optional-stopping problem, and the two inflate the same underlying quantity: how often you declare a winner that is not real. Fix the stopping rule with a proper sequential method or a fixed horizon, and fix the tail by writing it into the hypothesis before you launch. Neither substitutes for the other.

Common mistakes

• Deciding the tail after seeing a "close" result. This is the single most common misuse, and it is invisible in a results dashboard — the platform cannot tell the difference between a principled one-tailed test and a rescued one, because the giveaway is when the decision was made, not what number came out of it.

• Treating "one-tailed" as a way to get more power for free. It is not free. You are trading the ability to detect and act on harm in the opposite direction for a smaller required effect size. That trade is sometimes worth making, but it is a trade, not a discount.

• Applying a one-tailed override to only the primary metric. If a guardrail metric moves sharply in the "wrong" direction, a one-tailed primary-metric test gives you no formal signal to catch it — you are relying entirely on eyeballing the guardrail dashboard.

• Assuming Stats Engine has a one-tailed mode to switch on. It does not, by design; the FDR guarantee depends on the two-tailed default holding for every metric in the experiment.

Key takeaways

• A two-tailed test asks whether variation and control differ at all; a one-tailed test asks only whether variation is better, and gives up the ability to detect "worse" as a formal result.

• Optimizely's Stats Engine runs two-tailed tests by design — it is a requirement of the false discovery rate control the engine provides, not an arbitrary setting.

• A one-tailed test is only legitimate when a result in the unexpected direction is genuinely inactionable, decided and documented before the experiment launches.

• Choosing the tail after seeing the data — even informally, even just noticing the direction matches your prediction — silently doubles your real false-positive rate relative to the one you're reporting.

Frequently asked questions

Can I run a one-tailed test in Optimizely Web Experimentation or Feature Experimentation?

Stats Engine's significance calculation is two-tailed by design across both products, as part of its false discovery rate guarantee. You can still write a directional hypothesis and interpret a two-tailed result through that lens at the analysis stage — you are simply not changing the underlying test statistic to do it.

Is a one-tailed test ever more "correct" than a two-tailed test?

Neither is more correct in the abstract; they answer different questions. A one-tailed test is the right tool only when a result in the unexpected direction is truly inactionable and that has been decided in advance. Outside that narrow case, a two-tailed test is the safer default because it does not assume away the possibility of harm.

Why does a one-tailed test need a smaller effect to reach significance?

Because the same alpha budget is concentrated in one tail of the distribution instead of split across two, so a smaller observed deviation is enough to cross the threshold. That is the source of the extra sensitivity — and also the source of the risk, since it comes from narrowing what counts as evidence, not from collecting more of it.