Run A/B Tests on Shopify with Optimizely Web Experimentation

Loading...·8 min read

TL;DR

A Shopify storefront is one of the easier places to run Optimizely Web Experimentation and one of the easiest places to get it subtly wrong. The easy part is that a Shopify theme gives you a single template that renders every page, so the snippet goes in one file and covers the whole catalogue. The hard part is that Shopify owns the two things an experiment cares about most: the checkout, which you cannot inject script into on most plans, and the event stream, which since the Web Pixels API runs inside a sandboxed iframe rather than on your page.

That split decides the whole implementation. Variations are yours to make on the storefront; conversions have to be handed over from Shopify's sandbox. This guide covers both halves — installing the snippet so the storefront does not flicker, and forwarding Web Pixels API events into Optimizely as custom events — and it assumes you already know what the Optimizely Results page is telling you. If you have not sized the test yet, do that first with the sample size calculator, because a store doing a few hundred orders a week will not finish a checkout-rate test in the fortnight most people budget for it.

How Optimizely Web Experimentation Runs on Shopify

The Optimizely snippet is a synchronous script in the theme's <head>. It decides which variation the visitor is in before the page paints, applies the variation's changes, and exposes a global command queue that anything on the page can push events onto. Shopify's own event stream is separate: a custom web pixel runs in a sandbox with no access to your page's DOM or globals, and talks to the parent frame through a documented browser API.

flowchart TD
    A[Visitor loads a storefront page] --> B[theme.liquid head: Optimizely snippet]
    B --> C[Variation applied before paint]
    C --> D[Visitor adds to cart / checks out]
    D --> E[Shopify Web Pixel sandbox: analytics.subscribe]
    E --> F{Event in the allowlist?}
    F -->|yes| G[Push onto the Optimizely queue]
    F -->|no| H[Dropped, silently]
    G --> I[Custom event recorded against the visitor's variation]

What Shopify Controls and What You Control

You control theme.liquid and every section and snippet beneath it, which is where variations run. Shopify controls the checkout, the thank-you page and the order-status page. On Shopify Plus you can add scripts to checkout through checkout extensibility; on every other plan you cannot, and an experiment that needs to change the checkout UI is simply out of scope for Web Experimentation.

What is not out of scope is measuring the checkout. Web pixels run on checkout pages even when your scripts do not, which is why the forwarding half of this integration matters more here than on a self-hosted store.

Why the Checkout Is Different

A custom web pixel executes in a sandboxed iframe. It has no document from your storefront, no jQuery, no theme globals — and, importantly, no direct access to window.optimizely. What it does have is the standard events Shopify emits, and the ability to relay them. The practical consequence is that your forwarder is two pieces: a subscriber inside the pixel, and a small receiver on the storefront page that pushes onto the Optimizely queue.

Prerequisites

  • The Optimizely Web Experimentation snippet, with your project's snippet URL to hand.

  • Theme editing access (Online Store → Themes → Edit code), or a development theme you can publish from.

  • Custom events created in Optimizely for every Shopify event you intend to forward, with their API names noted. An event pushed under a name that does not exist in the Optimizely UI is discarded without an error.

  • Event properties created in Optimizely for any field you want to filter metrics by — five predefined names plus ten custom slots, fifteen in total per event.

  • Permission to add a custom pixel (Settings → Customer events → Add custom pixel).

Step 1: Install the Snippet Without Flicker

Flicker on Shopify has one dominant cause: the snippet is loaded after a render-blocking app script, or loaded asynchronously, so the original page paints before the variation applies. Shopify themes accumulate app scripts quickly, and every app that injects into content_for_header sits above whatever you add later.

Adding the Snippet to theme.liquid

Open layout/theme.liquid and place the snippet as high in <head> as it can go — above {{ content_for_header }} if your app stack tolerates it, and immediately below the charset and viewport meta tags otherwise.

<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width,initial-scale=1">

  <script src="https://cdn.optimizely.com/js/YOUR_PROJECT_ID.js"></script>

  {{ content_for_header }}
  ...
</head>

Do not add async or defer. Both turn a synchronous decision into a race with the first paint, and the visible result is the original content showing for a beat before the variation replaces it. If page-speed budgets make a synchronous script unacceptable, the honest answer is that this test belongs in a server-side implementation rather than a client-side one — the flicker article covers that trade in full.

Keeping It Out of the Checkout

Do not attempt to inject the snippet into checkout with an app or a script tag. On non-Plus plans it will not run, and on Plus it will run in an environment where the DOM you targeted does not exist. Scope experiments to the storefront and measure checkout completion through the pixel instead.

Step 2: Forward Web Pixel Events as Optimizely Custom Events

Shopify emits a fixed set of standard events — page_viewed, product_added_to_cart, search_submitted, checkout_started, checkout_completed and others. Forwarding all of them is a mistake: every forwarded name has to exist as a custom event in Optimizely, and page_viewed alone would multiply your event volume for no analytical gain. Forward the ones a metric will actually be built on.

The Event Map

Write the mapping down in one place. Shopify's names are snake case and fixed by the platform; Optimizely's API names are yours, and frozen the moment a metric depends on one.

{
  "events": {
    "checkout_completed": "purchase_completed",
    "product_added_to_cart": "add_to_cart",
    "search_submitted": "search_submitted"
  },
  "currencyProperty": "totalPrice",
  "numericProperty": "itemCount",
  "properties": [
    { "from": "productType", "to": "Category" },
    { "from": "variantSku", "to": "SKU" },
    { "from": "currency", "to": "Currency" }
  ]
}

Two reserved tags do the numeric work. revenue must be an integer number of cents — Optimizely discards a non-integer revenue tag, so an order total of 54.99 sent unconverted produces a revenue metric that reports nothing at all rather than something obviously wrong. value is a plain number and is the right home for item counts.

The Custom Pixel

A custom pixel cannot touch window.optimizely. What the sandbox does give you is a browser object whose localStorage accessor proxies to the top frame's storage — which is the supported way to hand something to the storefront page.

Add this under Settings → Customer events → Add custom pixel.

const FORWARDED_EVENTS = [
  'checkout_completed',
  'product_added_to_cart',
  'search_submitted',
];

FORWARDED_EVENTS.forEach(function (shopifyEvent) {
  analytics.subscribe(shopifyEvent, function (event) {
    const checkout = event.data.checkout || {};
    const cartLine = event.data.cartLine || {};
    const merchandise = cartLine.merchandise || {};

    browser.localStorage.setItem('optimizely_pending_event', JSON.stringify({
      name: event.name,
      data: {
        totalPrice: checkout.totalPrice ? checkout.totalPrice.amount : undefined,
        itemCount: cartLine.quantity,
        productType: merchandise.product ? merchandise.product.type : undefined,
        variantSku: merchandise.sku,
        currency: checkout.currencyCode,
      },
    }));
  });
});

The storefront script listens for the storage event, applies the map, and pushes onto the Optimizely queue:

function forwardToOptimizely(event) {
  const eventName = FORWARDED_EVENTS[event.name];
  if (!eventName) {
    return;
  }

  const data = event.data || {};
  const tags = {};

  if (typeof data.totalPrice === 'number') {
    // Optimizely records revenue in cents, as an integer.
    tags.revenue = Math.round(data.totalPrice * 100);
  }
  if (typeof data.itemCount === 'number') {
    tags.value = data.itemCount;
  }

  window['optimizely'] = window['optimizely'] || [];
  window['optimizely'].push({
    type: 'event',
    eventName: eventName,
    tags: tags,
    properties: {
      Category: data.productType,
      SKU: data.variantSku,
      Currency: data.currency,
    },
  });
}

window.addEventListener('storage', function (event) {
  if (event.key !== 'optimizely_pending_event' || !event.newValue) {
    return;
  }
  forwardToOptimizely(JSON.parse(event.newValue));
});

There is one case this cannot cover, and it should be stated rather than discovered. checkout_completed fires when the visitor is on Shopify's checkout, where your storefront page — and therefore the Optimizely snippet — does not exist. The pixel writes the pending event, and nothing on the page reads it. Two honest options: read the pending value on the next storefront page the visitor loads, which works for returning visitors and loses everyone who closes the tab, or send the conversion from your own server to the Optimizely Event API using the visitor id you captured earlier. Anything that claims to record a checkout completion in the browser at the moment it happens, on a non-Plus plan, is claiming something Shopify does not allow.

What the Payload Looks Like

A completed checkout for a £54.99 order of two items produces exactly this:

{
  "type": "event",
  "eventName": "purchase_completed",
  "tags": { "revenue": 5499, "value": 2 },
  "properties": { "Category": "Footwear", "SKU": "RUN-114-BLK", "Currency": "GBP" }
}

You can watch that mapping run, event by event, in the Shopify integration demo — including the case where an event is outside the allowlist and nothing is sent.

Gotchas

A page_viewed event that never arrives. Events outside the allowlist are dropped without a warning, which is correct behaviour and indistinguishable from a broken forwarder. If a metric is flat, check the map before you check the code.

Currency mixing. Shopify multi-currency stores emit totalPrice in the presentment currency. Summing revenue across currencies produces a number that means nothing. Either forward only the shop currency, or forward the presentment currency as a property and segment on it.

Theme updates overwrite theme.liquid. A theme update from the Shopify admin replaces the layout file. Keep the snippet insertion in version control, and re-check theme.liquid after every theme update.

App blocks that move the head. Some apps inject above your snippet regardless of where you place it. If flicker appears after installing an app, that is the first thing to look at.

Preview mode and bots. Shopify's own preview and Google's crawler both load storefront pages. Optimizely excludes known bots, but theme-preview traffic is real browser traffic and will bucket. Exclude ?preview_theme_id from targeting.

Verifying the Integration

Open a storefront page with ?optimizely_x=1 appended and confirm in the console that window.optimizely is defined before your first app script runs. Add a product to the cart and check that a push with eventName: "add_to_cart" appears — the Optimizely browser extension shows the queue directly. Then complete a test order and confirm purchase_completed arrives with an integer revenue tag.

The last check is the one people skip: leave the experiment running in draft for a day and compare Optimizely's conversion count against Shopify's own order count for the same window. They will not match exactly — Optimizely counts events from bucketed visitors only — but they should move together. A ratio that drifts is usually a sample ratio mismatch rather than a forwarding bug, and it is worth telling the two apart before you change any code.