Landing Page A/B Testing with Optimizely Web Experimentation

A landing page is the cleanest surface in experimentation. One page, one job, one conversion, and traffic you can usually control the volume of. That makes landing page A/B testing the place most teams start, and also the place where a run of inconclusive tests most often ends the programme before it gets going.

This guide covers what a landing page test can realistically move, how to work out whether you have the traffic before building anything, and how to run the test in Optimizely Web Experimentation without the variation flashing on screen or the ad platform quietly undoing the assignment.

What a landing page test can and cannot move

Landing page tests move the conversion rate of the page. They do not move the quality of the traffic arriving at it, and that distinction decides which ideas are worth testing.

Changes that reliably produce measurable effects are the ones that alter what the visitor has to do or understand: the number of fields in the form, whether the offer is above the fold, whether the headline matches the ad the visitor clicked, whether the primary action is obvious within a second. Changes that reliably produce nothing measurable are the ones that alter how the page feels without altering the task: a different shade of button, a stock photograph swapped for another, a paragraph rewritten with the same meaning.

The reason is arithmetic rather than taste. A page converting at 3% needs a large sample to resolve a 5% relative change, and most cosmetic edits produce far less than that. Test the ideas whose plausible effect is large enough for your traffic to see.

Traffic maths comes first

Do this before designing the variation, not after the test has been running for three weeks.

You need four numbers: current conversion rate, weekly visitors to the page, the smallest relative improvement that would change what you do, and how many variations you plan to run. Put them into the sample size calculator and read the required duration.

The answer is often uncomfortable. A page with 2,000 weekly visitors converting at 3%, testing for a 10% relative lift with two variations, needs several months. Three responses are honest; a fourth is not.

• Test a bigger change. Redesign the page rather than adjust it, so the effect to detect is larger.

• Consolidate traffic. Point several campaigns at one page instead of five near-identical ones.

• Raise the threshold. Decide that only a 20% improvement would change anything, and size for that.

• Run it anyway and call a 3% difference a win. This is the one that ends programmes, because those wins do not reproduce.

The guide to minimum detectable effect covers the trade-off directly, and how long to run an A/B test covers why the duration should also cover whole weeks — weekday and weekend traffic on a landing page often convert at quite different rates.

Same-page variation or redirect test

Two mechanisms, and the choice is driven by how much the variation differs.

flowchart TD
  A{How different is<br/>the variation?} -->|A few elements| B[Same-page variation<br/>DOM changes on one URL]
  A -->|Whole page or<br/>separate template| C[Redirect / split URL test]
  B --> D[Watch for flicker.<br/>Snippet must load first]
  C --> E[Both URLs need the<br/>same tracking and canonical]
  D --> F[One canonical URL,<br/>ad links unchanged]
  E --> G[Preserve query strings<br/>through the redirect]

Same-page variation. Optimizely applies changes to the DOM on the existing URL. Best for anything short of a full redesign: headline, form, layout order, offer placement. One URL keeps the ad links, the analytics and the canonical tag simple.

Redirect test. Visitors assigned to the variation are sent to a different URL that serves the alternative page. Necessary when the variation is a separate template, built by a different team, or too large to express as DOM edits. It costs a redirect in load time and introduces two failure modes worth knowing about — query parameters that do not survive the hop, and a second URL that search engines can index independently. The published guide to redirect and split URL experiments covers both.

Building the variation without flicker

Flicker — the original page appearing for a moment before the variation replaces it — is the defect that most often invalidates a landing page test. It is not only ugly. It changes behaviour, and it changes it only in the treated arm, which is precisely the shape of a bias that cannot be corrected afterwards.

Three rules prevent nearly all of it.

Load the snippet synchronously in the head, before anything else. Not through a tag manager, not with async, not after the stylesheets. A tag manager loads asynchronously by design, so the snippet inside it cannot run before the page paints.

<!-- First script in <head>. Synchronous and self-hosted-adjacent: nothing
     above it, and no tag manager between it and the browser. -->
<head>
  <meta charset="utf-8" />
  <script src="https://cdn.optimizely.com/js/YOUR_PROJECT_ID.js"></script>
  <link rel="stylesheet" href="/styles/landing.css" />
</head>

Keep the variation code cheap. Changes that run before first paint must not wait on a network call, a font, or an element that has not been rendered yet. Write the change against a selector that exists in the initial HTML.

Do not hide the whole page while you decide. The anti-flicker snippet that blanks the body until the variation applies trades a flash for a delay, and on a slow connection the delay costs more conversions than the flash did. Hide the specific element being changed, and set a short timeout after which it is shown regardless.

The published guide to the flicker effect in Optimizely covers the diagnosis in detail, including how to tell a flicker problem from a slow-render problem.

Paid traffic, ad platforms and tracking parameters

Landing pages usually sit behind paid campaigns, and three things about that arrangement break experiments quietly.

Ad platform optimisation is itself an assignment mechanism. A campaign optimising toward conversions will change who it sends over the course of the test. That does not break randomisation — visitors are still assigned randomly once they arrive — but it does mean the population drifts, so a test that runs long enough for the algorithm to relearn is measuring the variation on a different audience at the end than at the start. Keep campaign settings frozen for the duration.

Query parameters carry the campaign identity. A redirect test that drops utm_ or gclid parameters destroys attribution for the variation arm and makes the two arms non-comparable in the ad platform's own reporting. Preserve the full query string across the redirect.

Consent gating changes the denominator. If the snippet is blocked until a consent banner is accepted, the experiment only sees consenting visitors, and consent rates differ by region and device. That is workable, but the population needs to be stated in the readout rather than discovered later.

Verify all three before launch rather than after. A short pre-launch check on a staging URL — assignment fires, parameters survive, no flash, conversion event records — costs an hour and saves a rerun. The experiment QA checklist covers the full list.

Reading the result

Report the primary conversion with its interval, and read three companion numbers next to it.

{
  "experiment": "landing_hero_form_2026q3",
  "primaryMetric": "demo_request_submitted",
  "control": { "visitors": 18420, "conversions": 553, "rate": 0.030 },
  "variation": { "visitors": 18360, "conversions": 662, "rate": 0.036 },
  "relativeLift": 0.201,
  "confidenceInterval95": [0.062, 0.358],
  "guardrails": {
    "bounce_rate_delta": -0.004,
    "lead_quality_rate_delta": -0.031,
    "page_load_ms_delta": 12
  }
}

The guardrail on the third line is the one that matters most on a landing page. A form that converts better by asking for less will frequently produce leads that qualify worse, and a test judged on submissions alone will report a win the sales team experiences as a loss. Pick a downstream metric — qualified leads, activated accounts, first purchase — and watch it even when it is too slow to be the primary. The guide to guardrail metrics covers how to set the thresholds.

Check the split of visitors between arms as well. A meaningful imbalance means something interfered with assignment — a cache serving one variation preferentially, a bot filter applied unevenly, a redirect failing for some browsers. The note on sample ratio mismatch covers what to do about it, and the answer is always to fix the cause rather than to reweight the result.

Frequently asked questions

How much traffic does a landing page A/B test need?

It depends on the conversion rate and the size of the effect you want to detect, not on a fixed number of visitors. Run the numbers first: a page converting at 3% typically needs tens of thousands of visitors per arm to resolve a 10% relative change.

Should I use a redirect test or a same-page variation?

Same-page for anything short of a full redesign, because it avoids the redirect delay and keeps one URL. Redirect when the variation is a genuinely different page or template.

Can I run a landing page test through Google Tag Manager?

You can, but not without risking flicker. A tag manager loads asynchronously, so the snippet inside it may run after the page has painted. Put the snippet directly in the head, above everything else.

How many variations can I test at once?

As many as the traffic supports. Each additional arm divides the sample and adds a comparison, so two or three is usually the practical limit on a landing page unless the traffic is large.

What if the test wins on conversions but the leads are worse?

Trust the downstream metric. A form that converts better by asking less will often produce weaker leads, and the submission count is a proxy for value rather than the value itself. Decide before launch which number wins that argument.