Integrate Sentry with Optimizely Feature Experimentation

Sentry is an error and performance monitoring platform: it captures exceptions, traces, and session data from your application and groups them into issues you can search, alert on, and assign. Optimizely Feature Experimentation decides which variation of a flag each user receives, and those decisions can happen in the browser, on your server, or both. Connecting the two answers a question that is otherwise almost impossible to answer from an error report alone: is this crash happening to everybody, or only to the users who got the new variation?

That question matters more than it sounds. A feature flag is a deployment mechanism, and a variation that raises the error rate is a deployment that needs rolling back — not an experiment that needs another week of traffic. Without the variation attached to the error, a regression introduced by a 50% rollout looks like a background doubling of noise across your whole error volume, which is exactly the shape that gets dismissed as flaky.

The bridge is a decision notification listener: a callback registered on the Optimizely client that fires every time decide() resolves a flag. Inside that callback you write the variation onto Sentry's scope, so every event Sentry captures afterwards carries it. Feature Experimentation has no Custom Analytics Integration UI — unlike Web Experimentation, there is no dropdown to configure this — so the listener is the whole integration, and where it runs decides which Sentry SDK you call.

How the Integration Works

The listener receives the decision type, the user ID, the user attributes, and a decisionInfo object carrying the flag key, enabled state, variation key, and rule key. You take those fields and attach them to the current Sentry scope. From that moment on, any error, message, or transaction Sentry sends inherits them.

flowchart LR
    A["user.decide('checkout_redesign')"] --> B[DECISION notification fires]
    B --> C{Where does the listener run?}
    C -->|Browser| D["Sentry.setTag + Sentry.setContext"]
    C -->|Server| E["Sentry.withIsolationScope per request"]
    D --> F[Scope carries the variation]
    E --> F
    F --> G[Error captured]
    G --> H[Issue searchable by variation tag]

Decision Notification Data

The DECISION listener's decision info object contains these fields for a flag-type decision. The browser and Node SDKs expose camelCase keys; the Python SDK exposes snake_case keys.

Choosing a Sentry Primitive

Sentry offers three places to put the variation, and they are not interchangeable. Pick deliberately.

The integration below writes all three, because each answers a different question. The tag is what you search on. The context is what you read once you have opened the issue. The breadcrumb is what tells you whether the decision preceded the crash or followed it.

Prerequisites

Before starting:

• Optimizely Feature Experimentation SDK installed for your platform — the JavaScript SDK v6 or later, the Node.js SDK, or the Python SDK.

• A Sentry SDK initialized for the same platform: @sentry/browser or a framework wrapper such as @sentry/react in the browser, @sentry/node on the server, or sentry-sdk for Python. Sentry must be initialized before the Optimizely listener fires, or the first decisions are written to a scope that does not exist yet.

• A short, stable naming convention for the tag key. This article uses flag.<flagKey>, which keeps every flag in its own tag and makes them easy to spot in the tag list. Sentry indexes tag keys and truncates long values, so keep flag keys short and never put a user email or free-form string in a tag value.

• An understanding of scope lifetime on your platform. In the browser there is one user and one global scope, so a tag set once stays set. On a server, one process handles many users concurrently, and a tag written to the global scope leaks from one request into the next. This is the single most common way to get this integration wrong, and it is covered in detail below.

Browser Implementation

In the browser the Optimizely client and Sentry share a page and a user, so the listener can write straight to the global scope.

import { createInstance, enums } from '@optimizely/optimizely-sdk';
import * as Sentry from '@sentry/browser';

Sentry.init({
  dsn: '<YOUR_SENTRY_DSN>',
  environment: 'production',
});

const optimizely = createInstance({ sdkKey: '<YOUR_SDK_KEY>' });

optimizely.onReady().then(() => {
  // Register the listener BEFORE any decide() call, or the first decision is lost
  optimizely.notificationCenter.addNotificationListener(
    enums.NOTIFICATION_TYPES.DECISION,
    ({ type, userId, decisionInfo }) => {
      if (type !== 'flag') return;

      const { flagKey, enabled, variationKey, ruleKey, decisionEventDispatched } = decisionInfo;

      // Indexed and searchable: this is what you filter issues by
      Sentry.setTag(`flag.${flagKey}`, enabled ? variationKey : 'off');

      // Full payload, readable once an issue is open
      Sentry.setContext(`optimizely.${flagKey}`, {
        flagKey,
        enabled,
        variationKey,
        ruleKey,
        decisionEventDispatched,
        userId,
      });

      // Timeline entry: proves the decision happened before the error
      Sentry.addBreadcrumb({
        category: 'optimizely',
        message: `${flagKey} -> ${variationKey}`,
        level: 'info',
        data: { flagKey, variationKey, ruleKey, enabled },
      });
    },
  );

  const user = optimizely.createUserContext('user_123', { plan: 'pro' });
  const decision = user.decide('checkout_redesign');
  console.log('Variation:', decision.variationKey);
});

Two details are worth pausing on. enabled ? variationKey : 'off' keeps the tag meaningful when a flag is off: variationKey can be null for a disabled flag, and a null tag value is dropped silently, leaving you unable to tell "not in the experiment" apart from "integration broken". And decisionEventDispatched distinguishes a real experiment from a targeted delivery — an A/B test sets it to true, while a rollout returns false because no impression is dispatched. If your Sentry counts by variation ever need to reconcile against Optimizely's results, that flag is how you filter out the decisions Optimizely never counted.

Server-Side Node.js Implementation

The server version looks almost identical and is almost always wrong the first time, because of scope lifetime. A Node process serves many users at once. Sentry.setTag writes to whichever scope is current, and if that is the global scope, user B's error is reported carrying user A's variation.

The fix is to give each request its own isolation scope and register the decision inside it.

const express = require('express');
const Sentry = require('@sentry/node');
const optimizelySdk = require('@optimizely/optimizely-sdk');

Sentry.init({ dsn: process.env.SENTRY_DSN });

const optimizely = optimizelySdk.createInstance({ sdkKey: process.env.OPTIMIZELY_SDK_KEY });

const app = express();

app.get('/checkout', (req, res, next) => {
  // One isolation scope per request: tags written inside cannot leak out
  Sentry.withIsolationScope((scope) => {
    const userId = req.user.id;
    scope.setUser({ id: userId });

    const context = optimizely.createUserContext(userId, { plan: req.user.plan });
    const decision = context.decide('checkout_redesign');

    // Write the decision onto THIS request's scope, not the global one
    scope.setTag(`flag.${decision.flagKey}`, decision.enabled ? decision.variationKey : 'off');
    scope.setContext(`optimizely.${decision.flagKey}`, {
      flagKey: decision.flagKey,
      enabled: decision.enabled,
      variationKey: decision.variationKey,
      ruleKey: decision.ruleKey,
    });

    try {
      res.json(renderCheckout(decision.variationKey));
    } catch (error) {
      Sentry.captureException(error);
      next(error);
    }
  });
});

Note what changed: the decision is read from the value decide() returns rather than from a notification listener. On the server this is usually the better shape, because the decision and the scope it belongs to are visible in the same block of code — there is no question about which request a callback belongs to.

If you prefer a listener on the server — for instance because decisions are made in several places and you do not want to repeat the tagging — register it once and have it write to the current scope, which inside an isolation scope is that request's scope:

const { enums } = require('@optimizely/optimizely-sdk');

optimizely.notificationCenter.addNotificationListener(
  enums.NOTIFICATION_TYPES.DECISION,
  ({ type, decisionInfo }) => {
    if (type !== 'flag') return;
    const { flagKey, enabled, variationKey } = decisionInfo;
    Sentry.getCurrentScope().setTag(`flag.${flagKey}`, enabled ? variationKey : 'off');
  },
);

This is correct only when every decide() call happens inside an isolation scope. If any decision runs outside one — at startup, in a cron job, in a background worker — it writes to the global scope and contaminates every subsequent event in the process. Prefer the explicit form above unless you have checked that every call site is covered.

Python Implementation

The Python SDK's DECISION callback takes four positional arguments, and decision_info uses snake_case keys. This differs from the browser and Node SDKs, and mixing them up is the most common Python-side bug: decision_info.get('flagKey') returns None silently rather than raising, so the integration appears to work and every tag comes out empty.

import os

import sentry_sdk
from optimizely import optimizely
from optimizely.helpers import enums

sentry_sdk.init(dsn=os.environ['SENTRY_DSN'], environment='production')


def on_decision(decision_type, user_id, attributes, decision_info):
    if decision_type != 'flag':
        return

    flag_key = decision_info.get('flag_key')
    enabled = decision_info.get('enabled')
    variation_key = decision_info.get('variation_key')
    rule_key = decision_info.get('rule_key')
    dispatched = decision_info.get('decision_event_dispatched')

    sentry_sdk.set_tag(f'flag.{flag_key}', variation_key if enabled else 'off')
    sentry_sdk.set_context(
        f'optimizely.{flag_key}',
        {
            'flag_key': flag_key,
            'enabled': enabled,
            'variation_key': variation_key,
            'rule_key': rule_key,
            'decision_event_dispatched': dispatched,
            'user_id': user_id,
        },
    )
    sentry_sdk.add_breadcrumb(
        category='optimizely',
        message=f'{flag_key} -> {variation_key}',
        level='info',
        data={'flag_key': flag_key, 'variation_key': variation_key, 'rule_key': rule_key},
    )


optimizely_client = optimizely.Optimizely(sdk_key=os.environ['OPTIMIZELY_SDK_KEY'])

# Register BEFORE any decide() call
optimizely_client.notification_center.add_notification_listener(
    enums.NotificationTypes.DECISION, on_decision
)

user = optimizely_client.create_user_context('user_123', {'plan': 'pro'})
decision = user.decide('checkout_redesign')
print('Variation:', decision.variation_key)

Python web servers have the same scope-lifetime problem as Node. Sentry's Python SDK isolates scopes per request automatically for supported frameworks such as Django, Flask, and FastAPI, so a tag set during a request stays inside it. In a worker, a Celery task, or a plain script, wrap the work in with sentry_sdk.isolation_scope(): so decisions from one job do not follow the process into the next.

Verifying the Integration

Do not wait for a real error to find out whether this works. Force one.

1. Add a temporary route or button that calls decide() and then throws.

2. Trigger it, and open the resulting issue in Sentry.

3. Confirm the Tags section shows flag.checkout_redesign with the variation key you expected.

4. Confirm the Contexts section shows the full decision payload, including ruleKey.

5. Confirm the Breadcrumbs timeline shows the optimizely entry above the error — if it appears below, the listener is firing after the failure and the tag on this event came from a previous decision.

6. In Sentry's issue search, run flag.checkout_redesign:treatment and confirm your test event is returned. A tag that does not filter is a tag that was rejected for length or type.

Repeat the check on the server path with two concurrent requests for two different users bucketed into different variations. If both errors carry the same variation, your isolation scope is missing.

Analyzing Experiments in Sentry

Once the tag is flowing, three views become available.

Filter any issue stream by variation. Search flag.checkout_redesign:treatment to see only the errors from users in the new variation. The comparison you want is against flag.checkout_redesign:control over the same window — absolute error counts mean little, but a fivefold difference between arms is a rollback signal.

Read the tag breakdown on an existing issue. Open any issue and Sentry shows the distribution of each tag across its events. If an issue is 97% treatment on a flag running at a 50/50 split, that issue is caused by the variation, and you have found it without writing a query.

Alert on a variation. Create an alert rule filtered to the treatment tag so a regression in a rollout pages someone while the flag is still small, rather than after it reaches 100%.

The same tag is worth adding to your performance monitoring: transactions inherit scope tags, so you can compare p95 latency between arms in Sentry rather than reasoning about it from the experiment results alone.

Troubleshooting

Related Reading

• Integrate Datadog and New Relic with Optimizely Feature Experimentation — the same decision listener pointed at APM instead of error monitoring.

• Integrate FullStory with Optimizely Feature Experimentation — for the session that produced the error, not just the stack trace.

• Integrate PostHog with Optimizely Feature Experimentation — the product analytics side of the same pattern.