A/B Testing Metrics Framework and OEC in Optimizely
TL;DR
Every experiment eventually forces the same question: which number decides whether this shipped? Teams that answer it after seeing results are metric-shopping; teams that answer it before, with a deliberate hierarchy, run an honest program. A metrics framework is that hierarchy — one Overall Evaluation Criterion at the top, a single primary metric that decides the test, a short list of secondary metrics for context, and monitoring goals that guard against hidden harm. This guide defines the taxonomy, shows how to pick each tier by its distance from the change, and maps it onto Optimizely's actual metrics model — where the ranking you choose literally changes the statistics.
Start With the Business Question, Not the Dashboard
The Overall Evaluation Criterion (OEC) is the business outcome your program is really trying to move — the thing you'd defend to a CFO, not the click your test happens to fire. Define it first, and keep it distinct from any single test's implementation metric. "Increase net revenue per session without raising refunds" is an OEC; "clicks on the new button" is an implementation metric that may or may not serve it.
One number is never enough, though. A lone success metric can't tell you whether a "win" quietly cost something elsewhere — more sign-ups that never activate, more add-to-carts that raise returns. That's why the OEC sits above a hierarchy, not alone.
And the OEC is frequently too far downstream to be the decision metric for a given test. Revenue, retention, and lifetime value are the language executives use, but they arrive slowly and need enormous traffic to detect. The practical move is to decide with a closer, truthful metric and confirm with the downstream ones — which is exactly the structure the hierarchy provides.
Build a Metric Hierarchy
Four tiers, each with a distinct job:
Primary metric — the single metric that decides success. It should be close enough to the change to move quickly and clearly. You get exactly one.
Secondary metrics — supporting metrics that show downstream or adjacent effects and explain the shape of the change. Use them to understand what happened, not to casually overrule the primary metric.
Monitoring goals / guardrails — diagnostic safety signals that warn you when the change hurts another part of the product. Optimizely's own glossary defines guardrails as secondary metrics used to monitor possible negative impact. Treat them as an alarm system, not a second success criterion.
Leading indicators / local metrics — useful when the OEC is too far downstream, but each must still connect to the hypothesis or it becomes a vanity metric.
The discipline is in the restraint: the more metrics you elevate, the harder every one becomes to call — for real statistical reasons, as the Optimizely mapping shows next.
Map the Hierarchy to Optimizely
In Optimizely, metric ranking is not just documentation — Stats Engine treats each tier differently through a tiered Benjamini-Hochberg false-discovery-rate procedure:
The first-ranked metric is your primary. Its FDR control is computed independently of the others, so it "achieves statistical significance at full speed, regardless of any other goals" — adding metrics never slows your headline result.
Metrics ranked two through five are secondary. Their significance threshold is adjusted for the number of metrics and variations, so the more secondary metrics you add, the longer each takes to reach significance (the primary is unaffected). Keep the list short and deliberate.
Anything ranked beyond five is a monitoring metric, evaluated with a fractional weight of
1/n(n = number of monitoring metrics), so it has minimal impact on secondary metrics and none on the primary.
Two consequences worth internalizing. First, rank by how much you're willing to act on the metric, not by curiosity — the primary should be a high-signal goal (a behavior directly affected by your change, like add-to-cart for a product-page value prop), because a low-signal primary (revenue at the end of checkout) drags out time to significance. Second, FDR control is not maintained once you segment — the deeper you slice, the more false positives you invite, so use segments for exploration, not decisions. This taxonomy is the operating contract behind guardrail metrics and the numbers you read on the Optimizely results page.
Choose Metrics by Distance From the Change
Rank metrics along one axis: how directly the change touches them.
Close metrics give fast feedback. Clicks, form starts, add-to-cart, and step completions usually move faster and reach significance sooner than revenue.
Distant metrics are truthful but slow. Revenue, retention, and order confirmation matter, but they need more traffic and time, and small UI changes rarely move them detectably.
The rule: make the closest truthful metric your primary, and use downstream metrics as secondary confirmation. "Closest truthful" is the key phrase — a metric that's close but doesn't actually capture the mechanism (a click that doesn't imply value) is worse than useless. Because metric choice drives detectability, it directly drives runtime; size it honestly with how long to run an A/B test, and let the stats method picker align your metric contract with the analysis method you'll use.
flowchart TD
A[Business goal] --> B[Define OEC]
B --> C[Choose primary metric]
C --> D[Choose 2-5 secondary metrics]
D --> E[Add monitoring goals / guardrails]
E --> F[Build Optimizely scorecard]
F --> G[Launch experiment]
G --> H[Read result with effect size and uncertainty]
H --> I[Update program learnings]Special Cases That Need Explicit Rules
Some metrics need decisions made in advance or they'll mislead you:
Ratio metrics (revenue per visitor, completion rate) need a defined denominator and unit of analysis — "per visitor" versus "per session" is a different estimand. Optimizely's scorecard supports a dedicated Ratio metric type alongside Conversion, Numeric aggregation, Funnel, Bounce, and Exit; pick the type that matches the question rather than forcing a rate into a conversion metric.
Revenue metrics are the executive language but usually the wrong primary for a small UI change — too distant and noisy. Keep revenue as a secondary or monitoring metric there, and reserve it as primary for changes big enough to plausibly move it.
Segmented metrics and guardrails must not let a segment-specific harm hide inside a healthy aggregate. Watch guardrails at the segment level for known-risk groups — while remembering that segmenting suspends FDR control, so treat segment findings as signals to investigate, and confirm a real allocation problem with a sample ratio mismatch check before trusting any read.
Governance for Metric Selection
The framework only works if the hierarchy is fixed before the data arrives:
Preregister the hierarchy. Write down the OEC, the primary metric, the two-to-five secondary metrics, and the guardrails before launch — ideally in the experiment's own hypothesis and test-plan record so it travels with the test.
Do not metric-shop. Never promote a favorable secondary metric to primary after seeing the result. Optimizely even makes the cost concrete: if you change the primary mid-experiment, Stats Engine recalculates as if the new metric were always primary and pools the old primary into the secondaries, where it now takes longer to reach significance. The tool will let you; your credibility won't.
Report effect size and uncertainty, not just significance. A scorecard should tell a decision story — how big the effect is and how confident you are — not merely whether a threshold was crossed. Pair every "significant" with the interval, and let tests run at least one full business cycle before you read them.
Metric-Review Checklist
Before a scorecard goes live, confirm it has:
One OEC the whole program aligns to.
One primary metric, high-signal and close to the change.
Two to five secondary metrics — no more.
Explicit monitoring goals / guardrails for known risks.
A clear denominator and unit of analysis for every ratio.
A written reason for each metric's placement, fixed before launch.
Get the hierarchy right and the scorecard reads itself: one number decides, a few explain, the rest stand guard. That structure — chosen in advance and mapped to how Optimizely actually ranks metrics — is what separates a program that compounds learning from one that just accumulates dashboards.