Run A/B Tests on Adobe Commerce with Optimizely Web Experimentation

Loading...·7 min read

TL;DR

Adobe Commerce, which most of its users still call Magento, is the hardest common platform to run client-side experiments on. Not because of anything Optimizely does, but because of two Magento architecture decisions that are individually sensible and jointly hostile to DOM manipulation: aggressive full-page caching, and a cart and header rendered by Knockout from a customer-data API after the page has already arrived.

The result is a storefront where a variation targeting a product tile works perfectly and the identical variation targeting the mini-cart works on the first render and then vanishes. This guide covers where to put the snippet, how to write variations Knockout will not undo, and how to forward Adobe Client Data Layer events into Optimizely as custom events. Size the test before building it — the sample size calculator will tell you quickly whether a checkout-step test on your order volume is a two-week job or a two-quarter one.

How Optimizely Web Experimentation Runs on Adobe Commerce

The Optimizely snippet sits in the page head, decides the visitor's variation before paint, and applies DOM changes. Magento serves most of that HTML from its full-page cache, then hydrates the parts that differ per customer.

flowchart TD
    A[Request for a category page] --> B[Full-page cache serves stored HTML]
    B --> C[Optimizely snippet decides the variation before paint]
    C --> D[Variation applies to cached markup]
    B --> E[Knockout requests /customer/section/load]
    E --> F[Mini-cart, header and prices re-render]
    F --> G{Did the variation touch a Knockout region?}
    G -->|yes| H[Change is overwritten unless re-applied]
    G -->|no| I[Change persists]
    F --> J[adobeDataLayer push]
    J --> K{Event in the allowlist?}
    K -->|yes| L[Push onto the Optimizely queue]
    K -->|no| M[Dropped, silently]

Full-Page Cache Is Not the Problem

It is tempting to blame the cache, and it is almost always innocent. The variation decision happens in the browser, after the cached HTML has arrived, so a cached page and an uncached page bucket identically. Varnish does not need a new VCL rule and the cache does not need to be split by variation. Anyone proposing to vary the cache key by Optimizely cookie is about to multiply your cache footprint by the number of variations for no benefit.

Knockout Rendering Is

The mini-cart, the customer greeting, the price block on configurable products and several checkout regions are Knockout components driven by customer-data sections. They render after the page loads, and they re-render whenever the section data changes — which happens on every add-to-cart, on login, and on a timer. Any DOM change Optimizely made inside one of those regions is destroyed by the next render, with no error and no obvious trigger.

Prerequisites

  • The Optimizely Web Experimentation snippet URL for your project.

  • Access to a custom theme or module, so the snippet can be added through layout XML rather than pasted into the admin's "Scripts and Style Sheets" field, which loads late.

  • The Adobe Client Data Layer, present by default in Adobe Commerce 2.4.5 and later, or added by the Data Connection extension.

  • Custom events created in Optimizely for each event you forward, with API names noted. An event pushed under a name that does not exist in Optimizely is discarded without an error.

  • Event properties created in Optimizely for anything you want to segment by — fifteen per event, five predefined and ten custom.

Step 1: Add the Snippet to the Layout XML

Add the script to default_head_blocks.xml in your theme, so it is emitted with the head rather than appended after it.

<page xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:noNamespaceSchemaLocation="urn:magento:framework:View/Layout/etc/page_configuration.xsd">
    <head>
        <script src="https://cdn.optimizely.com/js/YOUR_PROJECT_ID.js" src_type="url" defer="false"/>
    </head>
</page>

defer="false" is the important attribute. Magento defers head scripts by default in several themes, and a deferred snippet paints the original page first — the flicker article covers what that costs. Then flush the cache: bin/magento cache:flush, and re-deploy static content if you are in production mode.

Do not use the admin field under Content → Design → Configuration → HTML Head → Scripts and Style Sheets. It works, and it places the script below the theme's own requirejs bundle, which is exactly the position you are trying to avoid.

Two deployment details catch people out on Adobe Commerce Cloud specifically. Layout XML lives in the theme, so a snippet added to a theme that is not the active one for a given store view does nothing at all — and multi-store installations very often have more themes than anyone remembers. And if a CDN sits in front of Varnish, its own HTML optimisation features can rewrite the head; Fastly's "streaming miss" and script-deferring options are both capable of turning a synchronous script into an asynchronous one without anything in Magento changing. Check the served HTML on production rather than the template.

Step 2: Write Variations That Survive Knockout

For anything outside a Knockout region, write ordinary variation code. For anything inside one, the change must be re-applied after each render and must be safe to apply twice.

var utils = window['optimizely'].get('utils');

function applyMiniCartCopy() {
  var button = document.querySelector('.minicart-wrapper .action.viewcart');
  if (!button || button.getAttribute('data-variation-applied') === 'true') {
    return;
  }
  button.textContent = 'Review your basket';
  button.setAttribute('data-variation-applied', 'true');
}

utils.waitForElement('.minicart-wrapper').then(function (wrapper) {
  applyMiniCartCopy();
  new MutationObserver(applyMiniCartCopy).observe(wrapper, {
    childList: true,
    subtree: true,
  });
});

The guard attribute is not decoration. Without it, the observer sees its own mutation, re-applies, and mutates again — an infinite loop that pins a CPU core and is usually diagnosed as "the site got slow after we started testing".

There is a limit to how far this should be pushed. If the variation needs to change what the customer-data endpoint returns rather than what Knockout renders from it, that is a server-side change and belongs in a server-side experiment, not a DOM variation.

Step 3: Forward Adobe Client Data Layer Events

The Adobe Client Data Layer is an event bus with a documented listener API, which makes it a better forwarding source than scraping the DOM for a success message.

The Event Map

{
  "events": {
    "place-order": "purchase_completed",
    "add-to-cart": "add_to_cart",
    "sign-in": "signed_in"
  },
  "currencyProperty": "grandTotal",
  "numericProperty": "qty",
  "properties": [
    { "from": "categoryName", "to": "Category" },
    { "from": "sku", "to": "SKU" },
    { "from": "customerGroup", "to": "Customer Group" }
  ]
}

revenue is a reserved Optimizely tag and must be an integer number of cents. A grandTotal of 54.99 forwarded unconverted is discarded, and the revenue metric then reports zero — a failure that looks like "the variation earned nothing" rather than like a bug. value is a plain number and holds the quantity.

The Data Layer Bridge

const FORWARDED_EVENTS = {
  'place-order': 'purchase_completed',
  'add-to-cart': 'add_to_cart',
  'sign-in': 'signed_in',
};

window.adobeDataLayer = window.adobeDataLayer || [];

window.adobeDataLayer.push(function (dataLayer) {
  Object.keys(FORWARDED_EVENTS).forEach(function (magentoEvent) {
    dataLayer.addEventListener(magentoEvent, function (event) {
      const properties = event.eventInfo || {};
      const tags = {};

      if (typeof properties.grandTotal === 'number') {
        tags.revenue = Math.round(properties.grandTotal * 100);
      }
      if (typeof properties.qty === 'number') {
        tags.value = properties.qty;
      }

      window['optimizely'] = window['optimizely'] || [];
      window['optimizely'].push({
        type: 'event',
        eventName: FORWARDED_EVENTS[magentoEvent],
        tags: tags,
        properties: {
          Category: properties.categoryName,
          SKU: properties.sku,
          'Customer Group': properties.customerGroup,
        },
      });
    });
  });
});

Pushing a function onto adobeDataLayer rather than calling addEventListener directly is what makes this safe to load at any point: the data layer invokes queued functions once it is ready, and replays events that were pushed before the listener existed.

What the Payload Looks Like

An order of two items totalling £54.99 produces this:

{
  "type": "event",
  "eventName": "purchase_completed",
  "tags": { "revenue": 5499, "value": 2 },
  "properties": { "Category": "Footwear", "SKU": "RUN-114-BLK", "Customer Group": "Wholesale" }
}

The Adobe Commerce integration demo runs the same map over data-layer-shaped events, including a page-view that is outside the allowlist and is dropped.

Gotchas

Static content deployment. In production mode, changing layout XML without running setup:static-content:deploy leaves the old head in place. The snippet appears to have been ignored.

Customer groups change prices. Wholesale and retail customers see different prices on the same URL. A revenue metric that mixes them is not wrong, but it is high-variance and will take far longer to resolve than the same test on a single group.

The checkout is a single-page application. Magento's checkout does not do full page loads between steps, so a metric configured as a pageview of a checkout URL will fire once and never again. Forward data-layer events instead.

Multi-store views. One Magento installation can serve several storefronts on different domains. Confirm the snippet is emitted for every store view whose traffic you intend to include, or one arm quietly under-collects — the kind of imbalance the Results page surfaces as a sample ratio warning rather than as a missing snippet.

Verifying the Integration

Load a category page with the network tab open and confirm the Optimizely script is requested before the requirejs bundle. Then add a product to the cart and confirm two things at once: that a push with eventName: "add_to_cart" reaches the Optimizely queue, and that any mini-cart variation is still applied after the mini-cart re-renders.

Place a test order and confirm the purchase push carries an integer revenue tag. Finally, log in as a customer in a different customer group and confirm the property comes through — a mapping that only ever reports Customer Group: "General" usually means the data layer is reading a cached section rather than the live one.