Data Platform
Most A/B tests do not fail because the variation was wrong. They fail because the experiment could never have answered the question: the sample was too small to detect the effect anybody cared about, the results were read before they settled, the traffic split was uneven and nobody checked, or twenty metrics were compared and the one that moved got reported. Every one of those is a measurement problem, and every one of them is invisible if you only look at the winner.
This section is about the statistics underneath an experimentation programme, written for people running tests rather than for people studying them. It covers how many users a test needs before it can see anything, what stopping early actually costs, how to reduce variance so a test finishes sooner, when a result is real and when it is regression to the mean, and how Optimizely computes the numbers it shows you.
The articles below go from the decisions you make before a test starts to the ones you make when the numbers come back. Several pair with a free calculator, so the method is not just explained but runnable. Start with whatever is about to bite you.
Articles in this section
Event Properties vs User Attributes in Optimizely: Essential Guide for Data Collection
Understand event properties vs user attributes in Optimizely and learn when to use each for accurate data collection across experimentation and analytics.
How to Read the Optimizely Results Page Correctly
How to read the Optimizely results page: statistical significance, confidence intervals, and the common gotchas that lead experimenters to the wrong conclusion.
How to Set Up Cross Product Events in Optimizely Experimentation (step-by-step)
Step-by-step guide to setting up cross-product events in Optimizely, so one user action is tracked consistently across Web and Feature Experimentation.
Optimizely MAUs: What Counts as a Monthly Active User & How to Reduce Overages
What counts as a Monthly Active User (MAU) in Optimizely, how MAUs are counted, and step-by-step fixes to diagnose and reduce MAU overages on your bill.
Overriding Variation Assignments in Optimizely: Complete Guide for Web and Feature Experimentation
Override variation assignments in Optimizely Web and Feature Experimentation for QA, debugging, and demos, using the correct method for each product.
Prevent Optimizely from being blocked by ad-blockers using AWS
Stop ad-blockers from blocking Optimizely by proxying the snippet through AWS CloudFront, so your experiments run reliably for every visitor on your site.
Send Events from Optimizely Full Stack to Optimizely Web
Send conversion events from Optimizely Full Stack to Optimizely Web so you trigger each event once and record it in both products at the same time.
How the Optimizely Stats Engine Works
Understand how Optimizely's Stats Engine uses sequential testing and false discovery rate control so you can interpret and trust your experiment results.
Reach Significance Faster: CUPED Variance Reduction
CUPED cuts experiment variance using pre-experiment data as a covariate, so tests reach significance faster. The math, Optimizely’s method, and when it fails.
Sequential Testing in A/B Tests: When Peeking Is Safe
Sequential testing lets you check A/B tests early without inflating false positives. How always-valid stats and Optimizely's Stats Engine make peeking safe.
Stats Accelerator vs MAB vs Contextual Bandit: Which to Use
When to use Optimizely Stats Accelerator, multi-armed bandit, or contextual bandit — a decision framework for choosing the right allocation method.
Sample Ratio Mismatch: Is Your A/B Test Broken?
Your traffic split looks off and results feel wrong. Learn to detect sample ratio mismatch with a chi-square test, find the cause, and fix it.
How Long Should You Run an A/B Test? A Practical Guide
How long should you run an A/B test? Learn to size the sample, convert it to days, cover a full business cycle, and when you can stop early.
Effect Size in A/B Testing: Absolute vs Relative Lift
Effect size is the true magnitude of an A/B test result. Learn absolute vs relative lift, Cohen's h, and how effect size drives sample size and reporting.
Guardrail Metrics in A/B Testing: Catch Tests That Win but Hurt
Guardrail metrics catch A/B tests that win on the primary metric but quietly harm the business. How to choose, threshold, and implement them in Optimizely.
How to Choose a Minimum Detectable Effect (MDE)
Learn how to pick a minimum detectable effect for your A/B test sample-size calculator — one that is worth shipping and your traffic can actually power.
False Discovery Rate Control in A/B Testing
The false discovery rate controls how many of your significant A/B test results are false positives. How Benjamini-Hochberg and Optimizely’s Stats Engine keep multiple-metric testing honest.
How to Calculate A/B Test Sample Size and Statistical Power
Calculate the visitors an A/B test needs from baseline rate, minimum detectable effect, significance, and power. Includes the formula, worked examples, and duration.
Winner's Curse and Regression to the Mean in A/B Testing
Why a winning A/B test variation’s measured lift is usually exaggerated and shrinks after launch — the winner’s curse, regression to the mean, and how to guard against it.
A/A Testing: When to Run One and How to Read It
An A/A test validates your experiment setup with two identical pages. When to run one in Optimizely, how to size it, and how to read a significant result.
Multi-Armed Bandit Testing: How It Works and When to Use
A multi-armed bandit shifts traffic to winning variations as data arrives. How epsilon-greedy, Thompson sampling, and UCB work, and when a bandit beats a fixed A/B test.
Multivariate Testing vs A/B Testing: When MVT Wins
Multivariate testing (MVT) vs A/B testing: full-factorial vs fractional designs, interaction effects, the traffic cost of combinations, and Optimizely MVT support.
Novelty and Primacy Effects in A/B Testing: When Early Lift Lies
Early A/B test lift can be novelty that fades or a primacy cost that recovers. Diagnose time-varying treatment effects and analyze them in Optimizely.
A/B Test Segmentation and Heterogeneous Treatment Effects
A/B test segmentation finds effects the average hides. Significant-here-not-there proves nothing: test the interaction and control multiplicity in Optimizely.
Quasi-Experimental Design: How to Measure Impact When You Cannot Randomize
Use quasi-experimental design when you cannot randomize: choose difference-in-differences, regression discontinuity, time series, or synthetic control.
Bayesian vs Frequentist vs Sequential A/B Testing in Optimizely
Choose between Frequentist Fixed Horizon, Bayesian, and Sequential analysis in Optimizely by the operating model your team can actually follow, not ideology.
A/B Testing Metrics Framework and OEC in Optimizely
A reusable A/B testing metrics framework: choose an OEC, build a primary/secondary/monitoring hierarchy, and map it to Optimizely scorecards.
Cohort Analysis for Experimentation Teams
Build cohort tables that mean something, read an Optimizely experiment by cohort without inventing a result, and know where cohort analysis stops and randomisation has to start.
How to Choose a North Star Metric
Pick a north star metric that measures delivered value, keep it separate from the OEC an experiment is judged on, and instrument the proxy chain in Optimizely underneath it.
Incrementality Testing: Measuring What Would Not Have Happened Anyway
Measure the causal contribution of a channel or feature with a randomised holdout, size the test honestly, and read incremental cost per acquisition instead of the attributed figure.
Subscribe
Practical Optimizely tips, monthly. No fluff.