A/B Testing Hypothesis Template and Examples for Optimizely
TL;DR
Most experiment backlogs are full of ideas ("redesign the pricing page") and empty of hypotheses. The difference matters: an idea is a change you might make, but a hypothesis is a falsifiable causal claim about what that change will do and why. A good hypothesis forces you to name the mechanism, the audience, and the metric before launch — which is exactly what stops teams from running tests they can't learn from and from inventing explanations after the result is in. This guide gives you a reusable hypothesis template, side-by-side examples from vague to strong, and a direct mapping into Optimizely's native Hypothesis and Test plan fields so the reasoning lives with the experiment instead of in a forgotten doc.
A Hypothesis Is a Causal Claim, Not a Wish
Keep three things distinct, because teams routinely collapse them:
An idea is a potential change ("shorten the signup form").
A hypothesis predicts the direction and mechanism of impact ("shortening the form will raise completion because it lowers the initial commitment cost").
A test plan describes how the rule will actually run — audience, metrics, duration, rollout.
A hypothesis worth testing states three things: what should change, why it should change (the mechanism), and for whom or where the claim applies. And it must be falsifiable — tied to a primary metric that will move in the predicted direction if the claim is true, and that is sensitive enough to detect the mechanism you're proposing. If nothing you could observe would prove the hypothesis wrong, it isn't a hypothesis yet.
The Copyable Hypothesis Template
Start from a template and fill every bracket. The basic form captures the causal claim:
If we change [experience] for [audience],
then [primary metric] will [increase/decrease]
because [mechanism].
The better form adds the expected size and the guardrails that protect you from a "win" that quietly costs something elsewhere:
If we change [experience] for [audience],
then [primary metric] will [move by expected direction/range]
because [reason/mechanism],
and we will watch [guardrail metrics]
to make sure we do not harm [risk].
Then record the operating assumptions alongside it — the details that turn a claim into a testable plan:
Audience or segment the claim applies to.
Baseline and expected direction (and a rough range if you have one).
Primary metric and the guardrail metrics you'll monitor.
Time window / duration or stop rule.
Size the "expected range" honestly with how to choose a minimum detectable effect, and turn the duration assumption into real numbers with the A/B test duration calculator — a hypothesis that needs a 0.2% lift on low traffic is not testable this quarter, and it's better to learn that before you launch.
What Weak vs Strong Hypotheses Look Like
The fastest way to calibrate is to see the same idea at three levels of rigor.
Vague — not a hypothesis:
"Make the page better."
No change, no metric, no mechanism. Nothing here can be tested or falsified.
Better, but still incomplete:
"A clearer CTA will improve conversions."
Now there's a direction, but no audience, no mechanism, and no guardrail. "Clearer" isn't a specific change, and "conversions" isn't a named metric.
Strong — testable:
"If we shorten the onboarding form and move the CTA above the fold for new visitors, then form-completion rate will increase because the initial commitment cost is lower, while support tickets and bounce rate remain stable."
One change, one audience (new visitors), one primary metric (form-completion rate), one mechanism (lower commitment cost), and explicit guardrails (support tickets, bounce rate). You can run it and you can be wrong — which is the point.
Translate the Hypothesis Into Optimizely's Fields
As of July 2, 2026, Optimizely Feature Experimentation added optional Hypothesis and Test plan fields directly on rules, so the rationale and plan live with the experiment instead of in a separate tool. Use them for what each is designed to hold — and don't confuse the two.
Hypothesis field — the causal claim. On a rule, click Add hypothesis and enter the expected outcome and the reasoning. On the New A/B test page it's the same Add Hypothesis step (reasoning, expected outcome, and why you expect it). Keep it short enough that a collaborator can scan it in the Flags dashboard — this is the claim, not the strategy deck. Edits are tracked in Change History.
Test plan field — how the rule runs. Click Add test plan and record duration, success criteria, and rollout strategy. It accepts either a URL and reference label (link up to 2,000 characters, label up to 100) or a file upload (.pdf, .doc, .docx, or .md, up to 10 MB), it applies to every rule type, and it's optional. Treat it as the operational contract, not a place for background theory.
Two practical payoffs: you can filter and sort rules by their hypothesis or test-plan content right from the Flags dashboard, and — if your team needs rule owners, Jira links, or a test phase enforced at creation — rule-level custom fields can require that metadata so the reasoning never goes missing. When your organization plans in Collaboration, link the experiment to its hypothesis there for the fuller ideation and workflow record; just don't duplicate an entire strategy document into the native rule fields.
Examples by Experiment Type
The template adapts across contexts — the constant is naming a mechanism and a falsifiable primary metric.
E-commerce. "If we add a persistent add-to-cart bar on product pages for mobile visitors, then revenue per visitor will increase because we reduce the friction of returning to the cart, while cart-abandonment rate does not rise."
SaaS. "If we show a simplified two-tier pricing page to first-time trial visitors, then trial-start rate will increase because reduced choice overload speeds the decision, while 30-day activation stays flat (we don't want more trials that never activate)."
Content / media. "If we make the newsletter CTA sticky after 50% scroll depth on article pages, then subscription conversion will increase because we reach engaged readers at peak intent, while scroll-through and time-on-page remain stable."
Note the pattern: each names the audience, predicts one primary metric with a mechanism, and pins a guardrail so a shallow win can't hide a deeper loss.
flowchart TD
A[Idea / backlog item] --> B[Write causal hypothesis]
B --> C[Define primary metric]
C --> D[Add guardrails and duration]
D --> E[Populate Optimizely Hypothesis field]
D --> F[Populate Optimizely Test plan field]
E --> G[Run experiment]
F --> G
G --> H[Read result and decide]
H --> I[Store learning for the next test]Write the Hypothesis Before You Prioritize
The hypothesis is not paperwork you fill in after deciding to run a test — it's the filter that decides whether to run it.
Writing it first exposes missing evidence: if you can't articulate why the metric should move, the idea isn't ready, and no amount of prioritization scoring fixes an idea with no mechanism. Once the mechanism is clear, feed the hypothesis into your scoring framework — ICE, PIE, or PXL — to weigh expected impact against effort and your confidence in the mechanism. That ordering (hypothesis → prioritize → run) is what keeps the hypothesis from degrading into a post-hoc story. Write it before launch; if you find yourself "explaining" a result after the fact, you've skipped the step this whole practice exists to protect. And confirm the change is actually testable in the real build with the experiment QA checklist before you commit traffic to it — a hypothesis that's really about broken instrumentation is better caught with an A/A test than mistaken for a treatment effect.
Review Checklist for a Good Hypothesis
Before it goes into the Hypothesis field, a strong hypothesis has exactly:
One change being tested.
One audience it applies to.
One primary metric that decides the outcome.
One mechanism explaining why the metric should move.
One set of guardrails protecting against hidden harm.
One duration or stop rule.
No vague filler — no "better," "improve," or "optimize" standing in for a specific, falsifiable claim.
Get those seven right and the test is honest before it starts: you've committed to what you expect, why, and how you'll know you were wrong. That is the whole difference between running experiments and just shipping changes with a dashboard attached.