Stats Accelerator vs MAB vs Contextual Bandit: Which to Use

You have a test ready to ship and Optimizely is asking you to pick an allocation method: a standard A/B test, Stats Accelerator, a multi-armed bandit, or a contextual bandit. The menu looks like a set of interchangeable options, but it is not — each method optimizes for a fundamentally different objective, and choosing the wrong one either wastes traffic or answers a question you never asked. This guide gives you a decisive framework: decide what you are optimizing for first, and the method chooses itself.

The one decision that changes everything: learn, earn, or personalize

Optimizely's own documentation splits these methods into two camps by their goal. A/B tests, multivariate tests, and Stats Accelerator are built for learning and long-term insight — they run the Stats Engine and produce statistical significance. Multi-armed bandit (MAB) and contextual bandit are built for immediate impact — they dynamically reallocate traffic to maximize a metric and deliberately do not calculate statistical significance.

That single distinction is the fork in the road. Before you compare features, answer one question: what does this test need to produce?

• A trustworthy verdict you will act on for months — which hero image, checkout flow, or pricing page genuinely converts better. You are optimizing for learning.

• The most conversions during the test itself, because the window is short and the "insight" has no shelf life — a weekend sale, a holiday banner, a headline for a news story that is stale in 48 hours. You are optimizing for earning.

• The best experience per visitor, because the right answer differs by segment — travel offers for frequent flyers, refinancing for high-interest borrowers. You are optimizing for personalized impact.

Those three objectives map cleanly onto Stats Accelerator, multi-armed bandit, and contextual bandit. Everything else in this article is detail. Hold the objective in your head and the rest follows.

Stats Accelerator: optimize for learning speed

Stats Accelerator answers the same question as a standard A/B test — which variation is genuinely better? — but it tries to get you there faster. It uses machine learning to shift more traffic toward variations that are on track to reach a conclusion, shortening the time to statistical significance while still producing a valid winner.

How Stats Accelerator works

Under the hood, Stats Accelerator runs a multi-armed bandit model (Optimizely uses an Upper Confidence Bound strategy) to decide, in real time, how to distribute traffic so the experiment reaches significance sooner. The critical detail — the reason this is a learning tool and not an earning tool — is how it protects your statistics.

Dynamically reallocating traffic normally biases your results: variations that happen to look good early get more traffic, which distorts the observed rates. Stats Accelerator corrects for this with a technique Optimizely calls weighted improvement. It breaks the experiment into short segments (epochs), holds allocation constant within each epoch, computes results per epoch, and combines them in a way that filters out the allocation bias. On top of that, the Stats Engine still decides when a variation is significant, exactly as it does for a manual A/B test, and it only operates on always-valid p-values — the same sequential statistics that make it safe to peek at results as they accumulate. If you want the mechanics of that, see how Optimizely's Stats Engine works and the deeper treatment in sequential testing and the peeking problem.

The net effect: you still get a statistically defensible winner and a real improvement figure, you just spend less traffic getting there. The results page shows the improvement over baseline, the confidence interval, and the significance level — the same scorecard you would read on any A/B test.

When Stats Accelerator is the wrong choice

Stats Accelerator earns its keep when you have several variations and want to find the best one without paying the full traffic cost of an even split. It is a poor fit in a few specific situations:

• Two-arm tests. Stats Accelerator requires at least three variations — a baseline plus two test variations. With a single variation against control, there is nothing to reallocate intelligently; run a standard A/B test.

• A/A tests. Do not use Stats Accelerator for traffic distribution in an A/A (calibration) test. There is no real difference to accelerate toward, and the reallocation logic has nothing meaningful to optimize.

• Delayed conversions. If your primary metric converts long after the visitor is bucketed, the epoch-based weighted improvement can be skewed, and pausing variations makes it worse. Metrics with a long lag between exposure and conversion undercut the whole premise of reallocating quickly.

• Very low traffic. If you barely have enough visitors to reach significance on an even split, the algorithm has too little signal to reallocate usefully. Estimate your runway first with the traffic estimator.

Multi-armed bandit: optimize for earning

A multi-armed bandit inverts the goal. It does not care about proving which variation is best with statistical certainty — it cares about maximizing your primary metric while the test is running. In academic terms, it minimizes regret: the cumulative conversions you lose by showing sub-optimal variations to visitors. Optimizely frames this as "earn while you learn" — you extract value from the leading variation throughout the campaign instead of paying the opportunity cost of an even split until a winner is declared.

Because that is the goal, a bandit does not report statistical significance. It reports the improvement in your primary metric — the lift the bandit captured versus what an equal (baseline) allocation would have delivered. If a stakeholder asks "is this result significant?", a bandit is the wrong tool; it was never trying to answer that.

The algorithms behind Optimizely's MAB

Optimizely uses two different bandit algorithms depending on your primary metric type. This matters because it changes how the bandit behaves early in a test:

• Binary (Bernoulli) metrics — a conversion that is either 0 or 1, such as a click or a signup — use Thompson Sampling. For each variation, Optimizely characterizes a beta distribution from the observed unique conversions and visitors, samples those distributions many times, and allocates traffic according to how often each variation "wins" the sample. Because early data is uncertain, Thompson Sampling naturally explores more at the start and concentrates traffic as confidence grows.

• Numeric metrics — such as revenue or events per visitor — use Epsilon-Greedy. A small fraction of traffic (epsilon) is spread uniformly across all variations to keep exploring, and the large remainder is sent to the variation with the highest observed mean. Epsilon-Greedy leans harder toward exploitation of the current leader.

Here is an illustrative sketch of the Thompson Sampling idea — this is a teaching example of beta-distribution sampling, not Optimizely's internal implementation:

# Illustrative only — demonstrates the Thompson Sampling concept,
# NOT Optimizely's production code.
import numpy as np

# Observed results per variation: (conversions, visitors)
data = {"A": (40, 1000), "B": (55, 1000), "C": (48, 1000)}

def sample_win_rates(data, draws=10_000):
    samples = {}
    for name, (conv, visitors) in data.items():
        # Beta(successes + 1, failures + 1) posterior over conversion rate
        samples[name] = np.random.beta(conv + 1, visitors - conv + 1, draws)
    stacked = np.vstack([samples[k] for k in data])
    winners = np.argmax(stacked, axis=0)          # which variation wins each draw
    keys = list(data)
    return {keys[i]: (winners == i).mean() for i in range(len(keys))}

# Allocation ∝ probability each variation is best
print(sample_win_rates(data))

The point of the snippet is only to make the intuition concrete: traffic flows toward variations in proportion to the probability that they are best, and that probability is re-estimated as data arrives.

When a bandit is the wrong choice

Reach for a bandit when the value is in the outcome during the test, not in a durable learning:

• Good fits: promotions and limited-time offers, seasonal or holiday banners, headline and creative testing for time-sensitive content, and any experience you will retire before a classic test could conclude.

• Poor fits: decisions you need to defend or roll out permanently. If you will cite this test in six months to justify a redesign, you want significance, not regret minimization — use Stats Accelerator or a standard A/B test. A bandit optimizes the moment; it does not give you a verdict you can generalize.

One more caution: a bandit chasing a fast, proxy metric (a click) can quietly sacrifice a slower, more important one (revenue, retention). Pick a primary metric that actually represents the value you want to capture.

Contextual bandit: optimize for personalized impact

A multi-armed bandit looks for the single best variation for everyone. A contextual bandit looks for the best variation for each visitor, based on their attributes. It still optimizes toward your primary metric and still forgoes statistical significance, but it adds a personalization layer: rather than converging on one global winner, it learns which variation works best for which kind of user and serves accordingly.

How contextual bandits differ from MAB

The mechanics are meaningfully different from a plain bandit:

• It uses context. You attach user attributes — device, location, membership status, lifetime value, spending categories — and the model factors them into every allocation decision. A frequent flyer and a small-business owner can be routed to different offers in the same campaign.

• It runs on machine-learning models server-side. Optimizely uses tree-based models (binary classification and regression) that live behind a prediction API, not in the browser snippet. The snippet sends identifiers, and the prediction service returns the variation to show.

• It has no sticky bucketing. Unlike an A/B test, a returning visitor is not locked to one variation; the model serves the best fit for the current session as behavior and context change.

• You tune exploration vs exploitation via a distribution goal.Automated starts at full exploration and gradually shifts toward exploitation as the model learns; Maximize Personalization fixes a 10% exploration / 90% exploitation split; Manual lets you set the rates (exploration is capped between 5% and 50%, because a model that mostly serves random variations is not personalizing).

• It has practical limits. Optimizely recommends keeping a contextual bandit to roughly 20 variations and 50 attributes, and it is configured as a Distribution Mode inside an Optimizely Personalization campaign. The attributes must be enriched with real user data — empty or sparse attributes give the model nothing to learn from.

Use a contextual bandit only when you genuinely believe the best variation differs by segment and you have the attribute data to express that. If one variation is simply better for everyone, a contextual bandit adds cost and complexity for no gain — a plain MAB or an A/B test is the right call.

Side-by-side comparison

A useful mental shortcut: Stats Accelerator and A/B tests answer a question; bandits capture value. If you need an answer you will reuse, stay in the learning column. If the value is in the here-and-now, move to the earning column — and add context only when the answer varies by person.

A decision framework for picking a method

Work top-down through the objective, not the feature list. The method falls out of two questions: do I need statistical proof? and does the best answer differ by visitor?

flowchart TD
    A[You have a test to run] --> B{Do you need a<br/>statistically valid winner<br/>to act on later?}
    B -->|Yes| C{How many variations,<br/>and is traffic steady?}
    C -->|2 variations| D[Standard A/B test]
    C -->|3+ variations,<br/>steady traffic| E[Stats Accelerator]
    C -->|Delayed conversions<br/>or very low traffic| D
    B -->|No, I want to<br/>maximize a metric now| F{Does the best variation<br/>differ by user segment?}
    F -->|No, one global winner| G[Multi-armed bandit]
    F -->|Yes, and I have<br/>quality attribute data| H[Contextual bandit]
    F -->|Yes, but no<br/>reliable attributes| G

Read it as a sequence of commitments:

1. Need proof? If yes, you are in the learning column. Two variations → run a standard A/B test. Three or more with steady traffic → Stats Accelerator. If conversions are delayed or traffic is thin, drop back to a standard A/B test so the reallocation logic does not mislead you.

2. Don't need proof, want conversions now? You are in the earning column. If one variation is best for everyone, use a multi-armed bandit.

3. Best answer varies by person? Only then step up to a contextual bandit — and only if you actually have enriched attribute data. Without good attributes, a contextual bandit degrades to a slower, more expensive MAB.

Common mistakes when choosing an allocation method

• Using a bandit to "run an A/B test faster." A bandit does not produce significance. If a colleague later asks whether the result was real, you will have no answer. When speed-to-insight is the goal, Stats Accelerator is the accelerator — not a bandit.

• Running Stats Accelerator on two arms or an A/A test. It needs at least three variations to reallocate meaningfully and should never drive traffic in a calibration test.

• Pointing a bandit at a shallow proxy metric. Optimizing clicks can cost you revenue. Choose the primary metric that represents real value before you let an algorithm chase it.

• Reaching for a contextual bandit without attribute data. Personalization is only as good as the signal you feed it. Sparse or unenriched attributes mean the model has nothing to personalize on.

• Forgetting that bandits skip validity safeguards you rely on elsewhere. Techniques such as CUPED variance reduction and the significance machinery on the results page are built around learning-mode experiments. If you switch to earning mode, you are trading that rigor for real-time optimization — make that trade deliberately.

Matching the method to the moment

There is no "best" allocation method, only the method that matches your objective. Stats Accelerator is for when you need to learn the winner and want to get there faster without breaking your statistics. A multi-armed bandit is for when you need to earn — to capture conversions during a window too short for a verdict to matter. A contextual bandit is for when the right answer depends on who is asking, and you have the data to tell people apart. Decide which of those three you are doing before you open the setup screen, and the choice stops being a guess. When in doubt, default to a learning-mode experiment: a verdict you can trust and reuse is almost always worth more than a marginal conversion you captured this week.