Multi-Armed Bandit Testing: How It Works and When to Use

A multi-armed bandit is an algorithm that decides how to split traffic across several variations while a test is still running, shifting more visitors toward whichever option is performing best. Instead of holding an even split until a fixed sample size is reached, a bandit continuously reallocates traffic to earn conversions during the test itself. This guide explains how multi-armed bandits work, the core algorithms behind them (epsilon-greedy, Thompson sampling, and Upper Confidence Bound), the explore-exploit tradeoff that governs every bandit, when a bandit beats a fixed A/B test, and how Optimizely implements bandit optimization.

What Is a Multi-Armed Bandit?

The name comes from slot machines. A single slot machine is nicknamed a "one-armed bandit" because it has one lever and it tends to take your money. Now imagine a row of slot machines, each with a different unknown payout rate. You have a fixed number of pulls and one goal: win as much as possible. Every pull forces a dilemma. Do you keep pulling the machine that has paid out best so far, or do you try the others in case one of them is secretly better? That is the multi-armed bandit problem, and it maps directly onto experimentation. Each "arm" is a variation, each "pull" is a visitor, and the "payout" is a conversion.

A fixed A/B test answers this dilemma by refusing to answer it during the test. It sends an equal share of traffic to every variation until enough data accumulates to declare a winner with statistical confidence, then you deploy the winner afterward. A multi-armed bandit answers it continuously. As evidence builds that one variation converts better, the bandit routes more traffic to it in real time, so fewer visitors are sent to the underperforming options while the test runs.

This changes the objective. A fixed A/B test optimizes for learning: it wants a precise, unbiased estimate of how each variation performs so you can make a confident decision. A bandit optimizes for earning: it wants to maximize total conversions across the whole population of visitors, even at the cost of a cleaner statistical readout. Understanding that tradeoff is the key to knowing when to reach for a bandit.

The Explore-Exploit Tradeoff

Every bandit balances two competing impulses on every decision:

• Exploit – Serve the variation that currently looks best, to capture conversions now.

• Explore – Serve other variations, to gather more data in case the current leader is not actually the best.

Pure exploitation is dangerous. If an inferior variation happens to get lucky in its first few hundred visitors, a greedy algorithm that only exploits will lock onto it and never discover its mistake. Pure exploration is wasteful: it is just a fixed A/B test that keeps sending traffic to known losers. A good bandit algorithm blends the two, exploring enough to stay confident about which arm is best while exploiting enough to bank conversions along the way.

The metric that formalizes this is regret: the cumulative difference between the conversions you would have earned if you had always served the truly best variation and the conversions you actually earned. A bandit's job is to minimize regret. Early on, regret accrues quickly because the algorithm does not yet know which arm is best. As evidence accumulates and the bandit concentrates traffic on the leader, the rate of regret slows. The whole design of a bandit algorithm is a strategy for driving regret as low as possible, as fast as possible.

The allocation loop below runs continuously for the life of a bandit experiment:

flowchart LR
  A[Split traffic evenly] --> B[Serve a variation]
  B --> C[Observe conversion outcome]
  C --> D[Update value estimate per arm]
  D --> E{Explore or exploit?}
  E -->|Explore| F[Sample under-tested arms]
  E -->|Exploit| G[Send more traffic to the leader]
  F --> B
  G --> B

Because the loop never stops updating, a bandit adapts if conditions change mid-test. If a variation that was losing starts winning (for example, because a new audience arrives), the value estimates shift and traffic follows. A fixed A/B test cannot do this: its allocation is frozen for the duration.

Bandit Algorithms: Epsilon-Greedy, Thompson Sampling, and UCB

"Multi-armed bandit" names the problem, not a single algorithm. Several algorithms solve it, and they differ mainly in how they decide to explore versus exploit. Three are worth knowing.

Epsilon-Greedy

Epsilon-greedy is the simplest bandit. You pick a small exploration rate, epsilon (for example, 0.1). On each visitor, the algorithm flips a biased coin:

• With probability epsilon, it explores by serving a random variation.

• With probability 1 minus epsilon, it exploits by serving the variation with the highest observed conversion rate so far.

With epsilon set to 0.1, roughly one visitor in ten sees a random variation and the other nine see the current leader. Epsilon-greedy is easy to reason about and easy to implement, but it has two weaknesses. It explores blindly, wasting some of its exploration budget on variations that are clearly bad, and a fixed epsilon keeps exploring at the same rate forever even after the best arm is obvious. A common fix is epsilon-decreasing, where epsilon shrinks over time so the algorithm explores heavily at first and exploits more as confidence grows.

Thompson Sampling

Thompson sampling is a Bayesian approach and the one most modern platforms favor. Instead of tracking a single conversion-rate estimate per variation, it maintains a full probability distribution over each variation's true conversion rate. For a binary conversion metric, that distribution is a Beta distribution updated from the counts of conversions and non-conversions each arm has seen.

On each decision, Thompson sampling draws one random sample from every arm's distribution and serves whichever arm produced the highest sample. The elegance is that exploration falls out automatically. An arm with little data has a wide distribution, so it sometimes produces a high sample and gets explored; an arm with lots of data has a narrow distribution tightly centered on its true rate. Over many decisions, each arm receives traffic roughly in proportion to the probability that it is the best arm. No epsilon to tune, and exploration self-adjusts as certainty grows.

# Illustrative only - demonstrates the Thompson Sampling concept,
# not any specific platform's production implementation.
import random

# (conversions, non_conversions) observed per variation
arms = {
    "A": [12, 388],   # 12 conversions in 400 visitors
    "B": [30, 370],   # 30 conversions in 400 visitors
    "C": [18, 382],
}

def choose_variation(arms):
    best_arm, best_sample = None, -1.0
    for name, (conv, non_conv) in arms.items():
        # Sample the arm's plausible conversion rate from its Beta posterior
        sample = random.betavariate(conv + 1, non_conv + 1)
        if sample > best_sample:
            best_arm, best_sample = name, sample
    return best_arm

# Arm B wins most draws, but A and C are still explored
# whenever their sampled rate happens to come out on top.

Upper Confidence Bound (UCB)

UCB follows the principle of "optimism in the face of uncertainty." For each arm it computes the observed conversion rate plus a bonus that grows with how uncertain that estimate is. An arm that has been shown to few visitors carries a large uncertainty bonus, so UCB is optimistic about it and explores it; an arm shown to many visitors carries a small bonus, so it is judged mostly on its actual performance. On each decision, UCB simply serves the arm with the highest upper confidence bound. As every arm accumulates data, the bonuses shrink and the algorithm converges on exploiting the genuinely best arm. Unlike Thompson sampling, UCB is deterministic given the same data, which makes its behavior easy to audit.

Bandit vs Fixed A/B Test

A bandit and a fixed-horizon A/B test are built for different jobs. The table below summarizes the practical differences.

The core trade is precision for opportunity cost. A fixed A/B test pays a known price - it deliberately sends half its traffic to a variation that may be worse - in exchange for a clean, trustworthy measurement of exactly how much better the winner is. A bandit reduces that opportunity cost by steering traffic to the leader early, but it sacrifices the clean measurement: because allocation is entangled with performance, you cannot read an unbiased effect size off a bandit the way you can off a controlled A/B test. If you plan test duration with a sample-size and statistical power calculation and care about the minimum detectable effect, you are thinking in fixed-A/B-test terms, and that is usually the right default.

When a Multi-Armed Bandit Beats a Fixed A/B Test

A bandit is the better tool when maximizing conversions during the test matters more than measuring the exact difference between variations. Reach for a bandit when:

• The opportunity is short-lived. For a weekend sale, a holiday banner, or a news-driven promotion, there is no "afterward" in which to deploy the winner. Whatever conversions you earn, you earn during the campaign. A bandit captures them; a fixed A/B test spends the whole window collecting evidence you will never act on.

• You are choosing among many low-stakes options. Testing ten subject lines or fifteen hero images as a full A/B test needs enormous traffic to power every comparison. A bandit concentrates traffic on the front-runners and quietly starves the losers, earning more conversions without waiting for every arm to reach significance.

• You do not need a defensible effect size. If the decision is "serve the best headline," not "prove headline B lifts conversions by 4.2% for the quarterly readout," the bandit's biased estimates are an acceptable price.

Stick with a fixed A/B test when the opposite is true: when you need a trustworthy, quantified result for a durable decision, when the change is high-stakes or hard to reverse, when you want a clean learning to generalize to future work, or when peeking and early stopping are the real concern - in which case sequential testing is the tool designed for valid early looks, not a bandit. The decision comes down to a single question.

flowchart TD
  A[New test] --> B{Need a precise, defensible<br/>effect size?}
  B -->|Yes| C[Fixed A/B test]
  B -->|No| D{Is the window short<br/>or are there many options?}
  D -->|Yes| E[Multi-armed bandit]
  D -->|No| C

How Optimizely Implements Multi-Armed Bandits

Optimizely groups its experiment types by objective. A/B tests, multivariate tests, and Stats Accelerator are built for learning and incorporate statistical significance, while multi-armed bandit and contextual bandit optimizations are built for immediate impact and optimize traffic allocation without performing significance analysis. Optimizely states plainly that a multi-armed bandit "does not rely on a fixed sample size or equal traffic allocation," and that bandits "do not achieve statistical significance because their goal is to optimize traffic allocation dynamically to maximize conversions." (Optimizely: Contextual bandits FAQ)

Multi-armed bandit optimization

In a standard Optimizely multi-armed bandit, you choose a primary metric and the platform displays variations to visitors, then routes more traffic toward whichever variation has the greatest impact on that metric. Optimizely's own example is capitalizing on conversions "during a short cycle like a weekend sale" - exactly the short-window case where a bandit shines. (Optimizely: Contextual bandits) Because a MAB optimizes for impact rather than significance, the Results page shows an Improvement figure for the variation over the baseline, but it does not declare a statistically significant winner the way a manual A/B test does. (Optimizely: Experiment Results page) Bandits also do not use the sticky bucketing that keeps an A/B-test visitor in the same variation across sessions, since a bandit is continuously reallocating traffic.

Exploration and exploitation rates

Optimizely exposes the explore-exploit tradeoff directly as two rates. The exploration rate is the probability the system serves a random variation (Optimizely calls this the "learning rate"), and the exploitation rate is the probability it serves the best-performing variation. When set to automatic, the exploration rate starts high and decreases as the model gathers more events - the same epsilon-decreasing idea described above, managed for you. (Optimizely: Contextual bandits FAQ)

Stats Accelerator and contextual bandits

Optimizely offers two neighboring tools that are easy to confuse with a plain bandit. Stats Accelerator reallocates traffic to help an experiment reach a statistically significant result faster - it optimizes for learning speed, not pure earning, and it keeps significance in play. A contextual bandit (CMAB) goes the other direction: rather than finding one best variation for everyone, it personalizes the variation per visitor using user attributes, running server-side behind a prediction API. Like a plain MAB, a contextual bandit does not report statistical significance. (Optimizely: Contextual bandits)

Choosing an Allocation Method

The right choice depends on whether you are optimizing for learning, earning, or personalization. Because Optimizely gives you a manual A/B test, Stats Accelerator, a multi-armed bandit, and a contextual bandit, picking among them is its own decision. For a full side-by-side of those four allocation methods and a decision framework for choosing between them, see the companion guide: Stats Accelerator vs MAB vs contextual bandit. If your reason for reaching past a fixed A/B test is speed to a trustworthy result rather than raw conversions, two complementary techniques are worth combining with your A/B tests instead of a bandit: CUPED reduces variance so tests reach significance faster, and Optimizely's Stats Engine provides always-valid significance so you can act on results sooner without inflating false positives.

Frequently Asked Questions

Does a multi-armed bandit reach statistical significance?

No. A multi-armed bandit is designed to maximize conversions by dynamically reallocating traffic, not to test a hypothesis, so it does not rely on a fixed sample size or equal allocation and does not report statistical significance. If you need a defensible, quantified effect size, run a fixed A/B test instead.

Is a multi-armed bandit the same as an A/B test?

No. Both compare variations, but an A/B test holds an even split to measure each variation precisely, while a bandit continuously shifts traffic toward the leader to earn more conversions during the test. An A/B test optimizes for learning; a bandit optimizes for earning.

What is the difference between epsilon-greedy and Thompson sampling?

Epsilon-greedy explores at a fixed rate by serving a random variation a set fraction of the time, exploiting the current best otherwise. Thompson sampling explores adaptively by sampling from each variation's probability distribution, so uncertain variations get explored more and exploration shrinks automatically as data accumulates. Thompson sampling generally produces lower regret without a rate to tune.

When should I not use a multi-armed bandit?

Avoid a bandit when you need a precise effect size for a durable or high-stakes decision, when you want a clean learning to generalize to future tests, or when your primary concern is valid early stopping - sequential testing is the correct tool for peeking, not a bandit.

Does Optimizely support multi-armed bandits?

Yes. Optimizely offers multi-armed bandit optimization that reallocates traffic to the highest-impact variation for a chosen primary metric, plus contextual bandits that personalize the variation per visitor using user attributes. Both optimize for impact and do not report statistical significance.

Key Takeaways

A multi-armed bandit reframes experimentation from "which variation is best?" to "how do I earn the most conversions while I find out?" It answers the explore-exploit tradeoff continuously, shifting traffic toward the leader to minimize regret rather than holding an even split to measure each option cleanly. Epsilon-greedy, Thompson sampling, and UCB are three strategies for making that explore-exploit decision, with Thompson sampling the modern default because its exploration self-adjusts as certainty grows. Reach for a bandit when the opportunity is short-lived or you are screening many low-stakes options and do not need a defensible effect size; stick with a fixed A/B test when you need a trustworthy, quantified result. Optimizely implements bandit optimization directly, exposing exploration and exploitation rates and offering both standard and contextual bandits, neither of which reports statistical significance - because for a bandit, conversions earned, not significance reached, is the goal.