Bayesian vs Frequentist vs Sequential A/B Testing in Optimizely

Optimizely's new A/B Results page now lets you analyze the same manual A/B test with three different statistical engines — Frequentist (Fixed Horizon), Bayesian, and Sequential (Stats Engine) — and the choice changes what you are allowed to do while the test runs. The wrong instinct is to pick a method by ideology ("Bayesian is more intuitive," "frequentist is more rigorous"). The right instinct is to pick the method whose operating rules your team can actually follow: how you plan the sample, whether you will peek at the dashboard, and how you will phrase the result to a stakeholder. This guide compares the three methods on those practical decisions, maps them to the current Optimizely UI, and gives you a pre-launch checklist so you never end up choosing a method after you have already seen which one declares a winner.

The Short Answer: Choose the Operating Model You Can Actually Follow

Before any statistics, decide how the test will be operated — that single answer usually picks the method for you.

• Choose Frequentist (Fixed Horizon) when you can commit to a prespecified sample size and a single look at the end. Calculate the sample and minimum duration before launch, do not act on interim significance, and analyze once at the planned horizon. It is efficient and transparent — but only if nobody peeks.

• Choose Bayesian when stakeholders need to hear a direct probability ("there is a 92% chance the variation beats control") rather than a p-value. You get flexible duration and intuitive language, but you must set your decision threshold before launch and treat that flexibility as a protocol, not permission to move the goalposts.

• Choose Sequential (Stats Engine) when the team will realistically watch the dashboard throughout the run. Sequential analysis is built for repeated looks: it adjusts the math so continuous monitoring does not inflate your false-positive rate. Optimizely's Stats Engine is the sequential option and the default.

Note the honest constraint before you get attached to Bayesian or Fixed Horizon: at publication time, Optimizely documents both Frequentist (Fixed Horizon) and Bayesian as beta (contact your Customer Success Manager for access). Sequential/Stats Engine is the generally available default.

flowchart TD
    A[How will the team operate the test?] --> B{One prespecified final look?}
    B -->|Yes| C[Frequentist Fixed Horizon]
    B -->|No| D{Need direct probability statements?}
    D -->|Yes| E[Bayesian]
    D -->|No, dashboard monitored repeatedly| F[Sequential / Stats Engine]
    C --> G[Precompute sample and do not peek]
    E --> H[Prespecify chance-to-beat and loss rules]
    F --> I[Use sequentially adjusted decisions]
    G --> J[Apply practical significance and data-quality gates]
    H --> J
    I --> J

Separate Statistical Philosophy From Product Workflow

Most method debates conflate two different things: the underlying statistics and the product workflow they enable. Keep them separate.

Frequentist (Fixed Horizon) treats probability as long-run frequency. It reports p-values, confidence intervals, and statistical significance, and it rests on a strict contract: define the sample size and analysis plan before the experiment starts, then evaluate only after all the data arrives. A p-value is the probability of seeing data this extreme if there were no true effect — it is not "the probability the variation wins." That distinction is the single most common misreading, and it does not go away just because the dashboard is easy to read.

Bayesian treats probability as a degree of belief that updates as data arrives. Optimizely's implementation uses a Beta-Binomial model for binary metrics and a Normal-Normal model for numeric metrics, and it reports a point estimate (the posterior mean of the relative improvement), a credible interval, a chance to beat baseline, and — for multi-arm tests — a probability of being the best arm. A 90% credible interval genuinely means "a 90% chance the true effect is in this range," which is why stakeholders find it easier to act on. Two cautions: the current version uses uninformed priors, so you cannot feed in a custom historical prior; and "more intuitive" does not mean "smaller samples" or "no false positives."

Sequential is not a third philosophy — it is frequentist. Stats Engine is a repeated-monitoring design with built-in error control, not a Bayesian method and not a license to check an ordinary fixed-horizon p-value every morning. The difference between "sequential testing" and "peeking at a fixed-horizon test daily" is exactly the error-rate math that makes the former valid and the latter a false-positive generator.

Compare the Methods on the Decisions You Actually Make

Ideology is irrelevant at the desk. Compare on the four decisions every experiment owner makes.

Planning and sample size. Fixed Horizon requires the planned sample before launch. Bayesian and Sequential relax the hard stopping rule, but neither eliminates planning: you still need a minimum-effect, power, and runtime plan, or you will stop on noise. Flexible duration is not the same as no plan.

Peeking, early stopping, and inconclusive tests. This is where the methods genuinely diverge:

Whatever the method, prespecify the same guardrails before launch: a success rule, a harm rule, a futility rule, a minimum runtime (run at least one full business cycle), and data-quality gates. Method choice never replaces a decision contract.

Result language. Report in the vocabulary of the method you chose. Fixed Horizon: p-value, confidence interval, "statistically significant." Bayesian: posterior mean, credible interval, chance to beat, probability best arm. Sequential: significance and confidence interval adjusted for repeated monitoring. Optimizely's own rule holds across all three — a result is significant precisely when the improvement interval moves entirely off zero.

Multiple metrics and variations. Switching methods does not remove multiplicity, metric-selection bias, or the winner's curse. Test enough metrics or variations and something will look significant by chance under any engine. The primary-metric contract — decide the one metric your decision hinges on, in advance — matters just as much in Bayesian as in frequentist analysis.

How Optimizely's New A/B Results Page Maps to Each Method

The three engines live on Optimizely's new A/B Results page, available for Web Experimentation and Feature Experimentation with Opti ID. It covers manual A/B tests only — Stats Accelerator experiments and multivariate tests are not supported — and you choose the engine during experiment setup.

• Configure a Bayesian test by selecting Bayesian for the Stats Configuration in place of the default Sequential. Set the chance-to-beat threshold before you start; Optimizely requires it to be at least 70%, and you should document the business loss you are willing to tolerate behind whatever number you pick. Remember the priors are uninformed.

• Configure a Fixed Horizon test by calculating and recording your target sample, minimum duration, MDE, alpha, and power up front. Then hold the line: an interim unadjusted result crossing a threshold is not a reason to stop — that is the exact behavior fixed-horizon math assumes you will not do.

• Operate Stats Engine as your continuous-monitoring default. For its full treatment of false-discovery-rate control and sequentially adjusted intervals, see how the Optimizely Stats Engine works, and still define a minimum runtime and a practical-significance check on top of the product status.

Across every method, planning starts with a defensible effect size — use how to choose a minimum detectable effect — and turns into a concrete duration with the A/B test duration calculator.

Ratio Metrics Need Care in Every Method

Ratio metrics like revenue per visitor are not a simple mean of independent rows: the numerator (revenue) and denominator (visitors or sessions) are related, so the analysis unit and aggregation determine the estimand. Optimizely handles this explicitly — for Bayesian ratio metrics it generates the posterior by sampling from two independent posterior distributions, one for the numerator metric and one for the denominator. The practical takeaway is not to reach for a generic t-test recipe on a ratio; use the engine's documented handling and be precise about what "per visitor" means in your instrumentation.

A Decision Matrix for Real Teams

The method usually falls out of who is running the test and how:

• High-governance research team that can preregister and take a single final look → Fixed Horizon. It is efficient and maximally transparent when the no-peeking contract is genuinely realistic.

• Product team that checks dashboards daily → Sequential. It matches actual behavior and protects repeated monitoring, instead of asking people to pretend they are not looking.

• Stakeholders who need direct probability statements → Bayesian, provided the chance-to-beat threshold and loss tolerance are agreed in advance.

And two rules that override all of the above. Never choose or switch the method after seeing which one declares a win — that turns method selection into the largest uncontrolled source of false positives in your program. And do not treat method disagreement as proof one is broken: they answer and operationalize different questions, so a Bayesian chance-to-beat and a frequentist p-value are not supposed to be identical.

Pre-Launch Method-Selection Checklist

Settle all of this before the experiment starts:

• Decision contract — primary metric, practical (not just statistical) effect size, harm rule, futility rule, minimum runtime, maximum runtime, and a segment policy that does not let you fish for a winning slice after the fact.

• Data-quality gates — instrumentation QA, a sample ratio mismatch check before you trust any read, a clear exposure definition, and a novelty/primacy review for early effects.

• Method chosen from constraints, then supported — run your operating answer through the stats method picker, then plan the design and read the deep guide for whichever engine you landed on.

Pick the method you can operate honestly, write down the rules before launch, and hold to them. The teams that get burned are rarely the ones who chose "the wrong" statistics — they are the ones who chose the method, or moved the threshold, after the numbers were already on the screen.