How Long Should You Run an A/B Test? A Practical Guide
TL;DR
Every practitioner asks the same question the moment a test goes live: how long do I have to wait? And close behind it comes the more tempting one: can I stop now that it finally looks significant? Both questions have real answers, but neither answer is a single number. Duration is not something you pick; it is something you derive from how big an effect you are trying to detect, how much traffic you have, and the statistical method your platform uses to decide a winner. Get the derivation right and you avoid the two expensive failure modes of experimentation: calling a false winner because you stopped too early, and burning weeks of traffic on a test that was never going to resolve. This guide walks through how to calculate a defensible runtime, why a minimum calendar window matters regardless of the math, and exactly when you are allowed to stop early.
What actually determines test duration
Runtime is the product of two things: the total sample size your test needs to detect the effect you care about, and the rate at which your traffic delivers that sample. A test that needs 100,000 visitors resolves in a week at 15,000 visitors a day and takes over a month at 3,000. Neither number is arbitrary. The sample size comes from statistics; the delivery rate comes from your site.
On top of that math sits a business-calendar constraint that the statistics alone will not capture. Human behavior runs on weekly cycles: the people who buy on a Tuesday afternoon are not the same people who browse on a Sunday night, and a test that only saw one of those groups is measuring a slice of reality, not the whole. So the honest framework has four steps, in order: calculate the sample size, convert it to days from your real traffic, enforce a minimum full business cycle, and only then apply a stop rule that matches your statistical method. Skip any step and you get a number that looks precise and is quietly wrong.
Calculate the required sample size
The finish line for a test is a sample size, and you can estimate it before you launch. This is the single most useful thing you can do to set expectations, because it converts "let's see how it goes" into "this test needs roughly N visitors, which at our traffic is about X weeks." Optimizely's Sample Size Calculator does this arithmetic for you, but understanding the inputs is what lets you make good tradeoffs.
The four inputs that set sample size
Four parameters determine how many visitors each variation needs:
Baseline conversion rate. The current rate of the metric you are trying to move. Lower baselines need dramatically more traffic.
Minimum detectable effect (MDE). The smallest relative improvement you want to be able to detect. A 10% MDE on a 5% baseline means you care about detecting a lift to 5.5%. Smaller MDEs need far more traffic.
Statistical significance. Your confidence threshold, commonly 90% or 95%. Higher confidence needs more data.
Statistical power. The probability of detecting a real effect if one exists (typically 80%). Underpowered tests miss real winners.
How each input moves the number
Two relationships dominate, and both push in the direction of "more traffic than you expected." First, the smaller your baseline, the larger the sample needed to detect the same relative change. Second, the smaller your MDE, the larger the sample: chasing tiny effects is expensive.
The following figures are Optimizely's own illustrative examples, useful for calibrating intuition rather than as guarantees, since exact numbers depend on the calculator's assumptions:
At 95% significance, holding MDE at 10% relative:
baseline 15% -> 7,271 visitors per variation
baseline 10% -> 12,243 visitors per variation
baseline 3% -> 51,141 visitors per variation
At 95% significance, holding baseline at 10%:
MDE 10% -> 12,243 visitors per variation
MDE 5% -> 59,401 visitors per variation
MDE 3% -> 185,661 visitors per variation
The practical takeaway: if a test is going to take too long, your levers are to accept a larger MDE (detect only bigger effects), pick a higher-frequency metric with a larger baseline, or reduce the number of variations so each gets more traffic. Variance-reduction techniques like CUPED can also reach significance with less data by removing pre-experiment noise. What you should not do is quietly lower your significance threshold to make the finish line closer.
Convert sample size into days
A sample size is only half the answer. To turn it into a runtime, divide by how fast your eligible traffic arrives:
weeks to run = total sample size / unique visitors per week
Suppose your test needs 60,000 visitors in total and the page you are testing gets 40,000 unique visitors per week:
60,000 / 40,000 = 1.5 weeks -> round up to 2 weeks
Two details matter here. First, use the traffic that actually reaches the test, not sitewide traffic: URL targeting, audience conditions, and the percentage of visitors you allocate all shrink the real rate. Splitting across more variations shrinks per-variation traffic further; a four-variation test at 8,000 visitors per variation and 10,000 visitors per week takes roughly 3.2 weeks, not one. Second, always round up to whole weeks. Because business metrics move on weekly rhythms, a runtime of "1.5 weeks" that ends mid-week hands you an unbalanced mix of weekdays and weekends. Rounding up to a clean multiple of seven days is not just tidy; it is what keeps the sample representative. Our traffic estimator and velocity calculator do this conversion for your actual numbers.
Enforce a minimum business cycle
Here is the rule that overrides the math when the math comes back too small: run every test for at least one full business cycle, which for most sites means a minimum of seven days, even if your calculated sample size arrives sooner. High-traffic sites can hit their target sample in a day or two, and stopping there is a mistake. A one-day test measures one day's worth of buyer behavior and calls it the truth.
Day-of-week effects
Traffic composition changes systematically through the week. B2B tools see engaged users Monday through Friday and a trickle on weekends; retail often sees the opposite. Visitors arriving from a payday, a weekly email send, or a weekend browsing session convert at different rates and buy different things. If your test only spans part of a week, you have confounded your variation with the day-of-week mix. Covering at least one complete cycle, ideally two, averages those effects out so you are comparing variations, not comparing Tuesday to Sunday.
Novelty and primacy effects
There is a second, subtler reason not to trust a short test, and it applies specifically to changes that returning users notice. A novelty effect occurs when a change is initially intriguing purely because it is new: users click the shiny new button more at first, and that lift decays as the novelty wears off. Its mirror image, the primacy effect, occurs when returning users are momentarily thrown by a change they were used to, so early engagement dips before recovering as they adapt. In both cases the first few days misrepresent the steady-state effect. A test stopped early during a novelty spike overstates the lift, and the "winner" quietly regresses after you ship it. Running long enough for the initial reaction to settle, and watching whether the effect holds across the full window rather than only at the start, is how you avoid shipping a mirage. This is well documented in the online-experiments literature, notably in Kohavi, Tang, and Xu's work on trustworthy controlled experiments.
Can you stop when it looks significant?
This is the question that trips up most teams, and the correct answer is: it depends entirely on which statistical method your test is using. Modern platforms, including Optimizely, let you choose between a fixed-horizon (frequentist) engine, a sequential engine, and a Bayesian engine, and the stop rule is completely different for each.
Fixed-horizon tests: commit to the horizon
A fixed-horizon test is the classic frequentist A/B test. You calculate the required sample size in advance, and the significance guarantee only holds if you look once, at the end, after that sample is collected. Checking results midway and stopping the moment you see significance is called peeking, and it inflates your false-positive rate badly. The intuition is a coin: flip a fair coin 100 times to test fairness, but peek after 10 flips and you might see 7 heads and wrongly declare it biased. Early data is noisy, and if you give yourself many chances to stop on a lucky reading, you will eventually take one. Under a fixed-horizon design the discipline is non-negotiable: commit to the sample size and the minimum duration up front, and do not act on partial results. Optimizely's Frequentist (Fixed Horizon) engine, currently in beta, enforces exactly this by withholding significance and confidence intervals until the sample size and any minimum duration are met.
Sequential testing: peeking is allowed by design
Optimizely's default Stats Engine is a sequential method, and it is built precisely so that peeking is safe. It uses always-valid statistics (a mixture sequential probability ratio test, developed with Stanford researchers) that hold at every point in time, not just at a predetermined endpoint. That means the 90% or 95% significance you see is genuinely valid whenever you look, and you can act on a significant result as soon as it appears without inflating your false-positive rate. This is a real advantage: strong effects can be called early, and inconclusive ones can run longer without penalty. For the mechanics of why this works, see our deep dive on sequential testing and the peeking problem.
But "you may stop early" is not "you should stop the instant the number turns green." Two guardrails still apply. First, the business-cycle minimum does not go away: even with a valid early significance signal, a two-day-old test has not seen a representative week, and novelty effects may not have decayed. Cover at least one full cycle before you trust the read. Second, confirm the result is not just significant but precise enough to act on. A significant result with a very wide confidence interval, or one riding an obvious traffic spike, is telling you the direction but not the magnitude. Let it run until the interval tightens.
A stop-or-continue decision framework
Putting it together, here is the sequence to walk every time you consider ending a test.
flowchart TD
A[Test is running] --> B{Has it run at least<br/>one full business cycle?}
B -- No --> W[Keep running.<br/>Do not decide yet]
B -- Yes --> C{Which engine?}
C -- Fixed-horizon --> D{Sample size and<br/>min duration reached?}
D -- No --> W
D -- Yes --> E{Significant?}
C -- Sequential / Bayesian --> F{Significant now?}
E -- Yes --> G[Ship the winner]
E -- No --> H{Conclusive null,<br/>or out of time/traffic?}
F -- Yes --> I{Interval tight enough<br/>and no novelty/spike doubt?}
F -- No --> H
I -- Yes --> G
I -- No --> W
H -- Yes --> J[Conclude: no detectable<br/>effect at this MDE]
H -- No --> WThe branch that catches people is the sequential path: reaching significance sends you not straight to "ship" but to a precision check first, and only then to a decision.
When to extend, and when to stop
Sometimes the right call is to run longer than planned. Extend when external events, promotions, or a traffic spike may have skewed the window; when a significant result still has an uncomfortably wide confidence interval; or when a change is the kind that returning users notice, so you want the novelty effect to fully decay before trusting the number. Extending is cheap insurance against shipping a result that will not replicate.
But do not run indefinitely, and this is the mistake in the opposite direction. External validity decays over time: the longer a test runs, the more seasonality, cookie churn, and changing traffic sources erode the assumption that your two groups differ only by the variation. A test that drags on for months is no longer measuring a clean comparison. If a test has covered a full cycle, collected close to its target sample, and still sits far from significance, that is itself an answer: the effect, if any, is smaller than your MDE. Conclude it, record the null result, and move traffic to a more promising hypothesis. Deciding your acceptable MDE range in advance is what lets you make that call without agonizing.
Mistakes that waste traffic
A few failure patterns account for most wasted experiments:
Stopping a fixed-horizon test at first significance. The single most common cause of unreplicable "wins." If you want to stop early, use a sequential engine designed for it.
Ignoring the business cycle on a high-traffic site. Hitting your sample in 36 hours does not mean the test is done; it means you must still wait out a full week.
Sizing on sitewide traffic. Targeting and allocation shrink real throughput; size on the traffic that actually enters the test, per variation.
Chasing a tiny MDE on a low baseline. This is how a test balloons to months. Either accept a coarser MDE or move the metric to a higher-frequency proxy.
Not checking for sample ratio mismatch. If your traffic split is off, no amount of runtime saves the result; validate it first (see sample ratio mismatch).
The short version
How long should you run an A/B test? Long enough to collect the sample size your MDE and baseline demand, delivered at your real traffic rate, and never shorter than one full business cycle, with two weeks a sensible default for most tests. Whether you can stop the moment it looks significant depends on your engine: under a fixed-horizon design, no, wait for the horizon; under Optimizely's sequential Stats Engine, yes, provided you have covered a representative cycle and the result is precise. When you are ready to read the outcome, our guide to the Optimizely results page walks through interpreting significance, confidence intervals, and improvement so you decide with confidence rather than hope.