Are you looking to make it easier to preview and QA Optimizely experiments? Download the OptiPilot Companion to easily see active experiments, preview events & more!

Optimizely Google Analytics 4 Data Discrepancies

Experiencing a Google Analytics data discrepancy with Optimizely? Here’s how you can fix it

by

Integrating two tools is often troublesome, especially when it comes to aligning data.

Google Analytics & Optimizely track users differently, and therefore when trying to integrate you may see that one tool shows more visitors than the other.

We often agree that at maximum a 10% discrepancy is acceptable, which accounts for differences in how both tools operate.

So if you are seeing that one of your experiments shows a discrepancy higher than 10%, then something is up.

In this article, we’ll take a look at how you can fix your Optimizely/Google Analytics discrepancy.

The Approach

To diagnose a discrepancy, we will take the following approach:

  • Create a new custom analytics integration
  • This analytics integration will include additional Optimizely custom events that will track the integration’s success (ie. that measure how many times the integration runs properly and sends data back to GA4)
  • We’ll use the Optimizely results to calculate a success conversion rate

What you need to start

Step 1: adding the new GA4 integration to your account

To start, you’ll need to add a new custom analytics integration inside your Optimizely project. This new integration is designed to help you find the root cause of your Google Analytics discrepancy.

You can do this by going to the Settings tab in the left navigation menu, then Integrations in the sub header menu, and then clicking on the blue Create New Analytics Integration… and then Using JSON…

New Custom Analytics Integration

Here’s the code you’ll want to copy and paste after clicking on Using JSON…:

{"plugin_type":"analytics_integration","name":"OptiPilot GA4 Analytics Integration","form_schema":[{"options":{"choices":[{"value":"on","label":"On"},{"value":"off","label":"Off"}]},"label":"Debug Mode","default_value":"on","name":"debug","field_type":"dropdown"},{"options":{"choices":[{"value":"5","label":"5 seconds"},{"value":"10","label":"10 seconds (default)"},{"value":"20","label":"20 seconds"}]},"label":"Polling Time","default_value":"5","name":"pollingTime","field_type":"dropdown"}],"description":"This integration allows you to send Optimizely experiment & variation names into GA4. You'll need to create a custom dimension whose event parameter is set to optimizely_experiment_web. ","options":{"track_layer_decision":"// Debug switch\nvar debugEvent = extension.debug;\n\n// Define event log\nvar eventLogX = function(eventName) {\n    if(debugEvent === \"on\") {\n      window[\"optimizely\"].push({\n        \"type\": \"event\",\n        \"eventName\": eventName,\n      });\n    }\n};\n\neventLogX(\"step1_ga4_init\");\n\nfunction fireOptlyGATracking() {\n\n  // getDecisionString Returns a string when bucketed or null when not bucketed.\n \tvar dimensionValue = window.optimizely.get(\"state\").getDecisionString({campaignId: campaignId});\n \n  //If bucketed into anything, set dimension value to slot and send to GA\n  if (dimensionValue) {\n    eventLogX(\"step3_ga4_success\");\n     gtag(\"event\", \"optimizely_experiment_web\", {\n\t\t\t optimizely_experiment: dimensionValue,\n\t\t });\n  }\n}\n\n/** \n**  Converting polling time to account for the 200ms polling interval.\n**  E.g for 1 second polling time. \n**  1 * 5 = 5\n**  5 * 200ms = 1000ms = 1sec\n**/\nvar pollFor = parseInt(extension.pollingTime)*5;\n\n// Initiate polling for GA object.\nvar count = 0;\nvar poll = setInterval(function(){ \n    count++;\n    if (count >= pollFor)\n    {\n        clearInterval(poll);\n        eventLogX(\"step4_ga4_timeout\");\n    }\n    if (typeof gtag !== 'undefined')\n        {\n            fireOptlyGATracking();\n            clearInterval(poll);\n            eventLogX(\"step2_ga4_ready\");\n        }\n}, 200);\n\n"}}

Once imported, a new row will be added in your Integrations list:


Finally, make sure to turn the integration on by clicking on the On button on the right hand side to enable the integration in your account.

Step 2: adding the required Optimizely custom events

For the diagnostics metrics to work, we’ll need to create 4 Optimizely custom events with the following names & API names:

  • step1_ga4_init
  • step2_ga4_ready
  • step3_ga4_success
  • step4_ga4_timeout

These custom events basically measure the success rate of different steps of the integration.

For example, the step1_ga4_init custom event records every time the integration properly initializes. step3_ga4_success measures how many times the integration runs successfully.

The idea is that if we see a lower than 100% conversion for any of these metrics, we’ll know that something is up and we’ll know exactly where to look – nearby the Optimizely custom event where the conversion rate drops happens.

Alright now that you know what these events are for, let’s create them in your Optimizely account.

An Optimizely custom event can be created by going to Implementation in the left navigation menu and then by clicking on the Events section of the sub header navigation menu.

You’ll then see a Create New Event blue button, click it and pick Custom in the list:


Once you see the Create New Event modal, enter the names displayed in the earlier bullet list as Name and API Name as such:


Repeat this creation process for all 4 events in the list.

Now that you have the necessary custom events created and the integration added to your account we are ready to start diagnosing your integration.

An A/A test to diagnose this issue

Alright, now that you have everything in your account configured, it’s time to start collecting data.

For this step, you’ll want to create a simple experiment, with no changes in any of the variations.

A few important bits:

  1. Make sure the experiments runs site-wide
  2. Make sure the OptiPilot GA4 Integration is turned on inside the Integrations section of your experiment as such:


We recommend to run the A/A until you get at least 500 unique visitors in the experiment. This gives enough data to work with, and preserves your overall Monthly Active Users (MAU) quota.

Taking action based on the results

After reaching these 500 visitors, have a look at your results page, you should see something like this:


What you see on this screenshot is that we’ve been tracking how healthy the integration is, using a funnel representing each critical step of the integration.

For any of the 3 steps, if you see a % conversion rate below 90%, then something is up. This goes as well for the last timeout event, which shows how many times Optimizely didn’t find Google Analytics on the page.

Let’s dive into each event:

  • step1_ga4_init: measures how many times the integration’s code is initialised on your page. This should always be in the 95%+ range.
  • step2_ga4_ready: measures how many times Optimizely is able to find the Google Analytics 4 instance on the page. This should be at least 90%+.
  • step3_ga4_success: measures how many times Optimizely successfully sent the experiment names and Ids to Google Analytics. This should be always in the 90%+ range.
  • step4_ga4_timeout: measures how many times Optimizely didn’t find Google Analytics after the polling interval (ie. default set to 10 seconds). This should always be a 0%, but we would only consider the integration as working if this metric has a conversion rate that is below 10%.

What to do if you see the numbers continue to be misaligned

If you continue to see a larger than 10% discrepancy, this means that any of the 4 Optimizely custom events we created will have a conversion rate that will be over the accepted range.

If you see this, you now have an idea of which step of the integration is failing on your website.

Here’s the diagnosis:

If step1_ga4_init is below 95%

This means that:

  1. You have not turned on the integration inside the experiment’s Integrations tab. Fix: re-complete the integration installation steps above.
  2. You’ve made a change to the integration code. Fix: delete the GA4 integration you have in your account and add it back following the integration installation steps.

If step2_ga4_ready is below 90% or if step4_ga4_timeout is higher than 10%

This means that Google Analytics is loading too late on your page. We can fix this issue by telling Optimizely to wait longer for Google Analytics to be initialised.

You can do so by going to your experiment, in the Integrations section, select a higher value for Polling Mode:

Once you’ve saved, reset the results of your experiment and run it again. You should now see an acceptable value for step2_ga4_ready.

If step3_ga4_success is below 90%

This means that there’s an issue with the communication from Optimizely to your Google Analytics instance on page. I’d recommend checking that your Google Analytics tracking (gtag.js) code is properly implemented.

In a nutshell

Google Analytics & Optimizely discrepancy are often quite difficult to diagnose. This integration should help to identify the root cause and provide a fix.

Leave a Comment