Feature Flag Best Practices for Production Systems

Loading...·11 min read

TL;DR

Your flag count has quietly crossed a threshold. What started as a clean way to ship a risky feature behind a toggle is now a few hundred conditionals scattered across services, half of them permanently on, nobody quite sure which are safe to delete. Someone flips the wrong one during an incident. A test passes locally and fails in CI because a flag defaulted differently. This is the predictable midlife of feature flagging: the mechanism is trivial, but the discipline around it is what separates teams who ship faster from teams who have simply moved their risk somewhere less visible.

The good news is that the disciplines are well understood and largely independent of any one vendor. This article lays them out, then shows how they map onto Optimizely Feature Experimentation specifically, so that the practices have a concrete home in tooling rather than living only in a wiki page nobody reads.

Classify every flag by type and lifespan

The single most useful idea in feature flagging is that not all flags are the same thing. Martin Fowler's taxonomy splits them into four categories, and the reason it matters is that each has a different owner, a different lifespan, and a different failure mode when you ignore it (Fowler, Feature Toggles).

  • Release toggles hide in-progress work so it can merge to trunk and ship dormant. They are the shortest-lived flags you own, meant to survive days or weeks, and they should be deleted the moment the feature is fully rolled out.

  • Experiment toggles split traffic between variations to measure an outcome. They live exactly as long as the test needs to reach a decision, then collapse to the winning variation. This is the layer beneath A/B testing.

  • Ops toggles give operators control over system behavior: degrade an expensive feature under load, disable a flaky dependency, or trip a kill switch during an incident. Some are short-lived, but a genuine kill switch may live indefinitely as a safety control.

  • Permissioning toggles decide who gets access to what, such as gating a feature to premium accounts. These can legitimately live for years and are effectively part of your product, not temporary scaffolding.

Fowler frames these along two axes: longevity (how long the flag should exist) and dynamism (whether the decision is fixed per deployment or must be re-evaluated per request). The categories fall at different points, and conflating them is the root of most flag debt. A release toggle that gets treated like a permanent ops switch never gets cleaned up, because nobody remembers it was supposed to be temporary.

The practical takeaway is to make the type explicit at creation. Decide, before you write the flag, whether it is a release, experiment, ops, or permissioning flag, and record that alongside an owner and an expected removal or review date. In Optimizely, flag rules make the distinction concrete: an A/B test rule is an experiment flag, while a targeted delivery is a release or ops flag. If you are weighing a fixed split against an adaptive one for an experiment flag, the trade-offs are covered in Stats Accelerator vs multi-armed bandit vs contextual bandit.

flowchart TD
    A[New flag] --> B{What is its job?}
    B -->|Hide in-progress work| C[Release toggle<br/>lifespan: days-weeks<br/>delete after rollout]
    B -->|Split traffic to measure| D[Experiment toggle<br/>lifespan: the test<br/>collapse to winner]
    B -->|Operator control / kill switch| E[Ops toggle<br/>lifespan: short, or<br/>indefinite for safety]
    B -->|Gate access by plan/role| F[Permissioning toggle<br/>lifespan: years<br/>part of the product]
    C --> G[Book its removal now]
    D --> G
    E --> H[Owner + review date]
    F --> H

Name flags so they stay searchable at scale

At ten flags, naming is a matter of taste. At three hundred, it is the difference between a flag you can find and a flag you are afraid to touch. A good flag key is greppable across your codebase, self-describing, and encodes enough context that someone who did not create it can guess its purpose and owner.

A convention worth adopting encodes, in a consistent order, the owning team or domain, the flag's intent, and where it applies. Something like checkout_new-address-form_release or search_semantic-ranking_experiment reads cleanly, sorts sensibly, and tells a reader the type without opening a dashboard. Keep to one delimiter, avoid spaces and punctuation, and never reuse a retired key for a new purpose. OptiPilot's free feature flag naming convention generator turns a flag's type, team, and feature into a consistent key across dot, kebab, and snake styles, so a whole team names flags the same way.

Naming discipline pays off precisely because flag keys are the join between your code and your configuration. Optimizely makes this contract literal: a rule's key cannot be changed after creation, and if the key referenced in your decide call does not match a key in the platform, no traffic is served to that rule (Manage rules). A typo is not a loud error; it is a silent no-op that quietly serves the fallback. Consistent, reviewed names make that class of mistake far easier to catch in code review.

Roll out gradually and keep a kill switch

The reason flags earn their keep in production is that they decouple deploy from release. You ship code dark, then turn it on for a slice of traffic, watch, and widen the slice, or reverse instantly if something breaks. Two properties make this safe: a gradual ramp and a fast reversal.

A gradual ramp means starting small. Optimizely's own guidance for feature rollouts recommends 10% as a starting allocation before widening (Run Feature Rollouts). You can also target the ramp: deliver to internal users first, then beta customers, then the general population, using audience conditions rather than a blunt percentage. Optimizely's targeted delivery supports exactly this, letting you roll out to a percentage of a specific audience and increase it over time (Flag delivery overview).

Fast reversal is the kill switch. Because flag changes take effect through configuration rather than a code deploy, you can roll a feature back in seconds. In Optimizely, deliveries take effect without a code deployment, and you can toggle a flag on and off without re-bucketing the users already assigned, as long as you do not change the traffic allocation at the same moment (Run flag deliveries). That last caveat matters operationally: an incident kill switch should flip the flag off, not simultaneously re-slice traffic, or you will disturb bucketing for everyone else.

There is a subtle trap in ramping that shows up most in experiment flags. Bucketing is deterministic: Optimizely hashes the user ID to assign a variation, so the same user gets the same experience as long as the distribution has not changed (Core concepts). Change the traffic allocation on a running experiment and users can be re-bucketed, which both disturbs their experience and can skew results. If you need to change allocation mid-flight, use a user profile service to keep bucketing sticky, and watch for sample ratio mismatch, the canary that tells you your split is not landing the way you configured it.

One decision to make deliberately: whether a given flag needs measurement. Optimizely separates a plain targeted delivery (free, no analytics, dispatches no decision events) from a feature rollout (full analytics, consumes monthly active users and impressions). Use a targeted delivery for a simple release or kill switch where you only need on/off control, and a rollout or A/B test when you actually want to measure impact. Reaching for the analytics-bearing rule on a flag you will never analyze is a common and needless cost.

Environments and the datafile: where flags actually live

Flags are configuration, and configuration needs somewhere to live per stage of your pipeline. In Optimizely, a project contains one or more environments, typically staging and production, and each environment holds its own ruleset for a flag. You can run a flag at 100% in development and 10% in production, or configure entirely different rules per environment, with separate permissions for who can change each (Core concepts).

At runtime the SDK does not call back to Optimizely on every decision. It downloads a datafile, a JSON snapshot of your flags and rules for one environment, and evaluates decisions locally against it. When you edit a rule, the datafile updates within a few seconds, and how quickly that reaches production depends on how often your app fetches it; webhooks let you push updates in near real time (Manage rules). This architecture is why flag evaluation is fast and why it keeps working during a brief network blip: decisions are computed from an in-memory config, not a synchronous network round trip.

It also defines your failure mode, which is the next thing to get right.

Test safely with flags and sensible fallbacks

Flags multiply the states your system can be in, and untended they make tests flaky and behavior unpredictable. Three habits keep that under control.

First, always have a safe default. Design so that the flag being off, or the configuration being unavailable, yields a working, conservative experience. Optimizely's decide method returns an OptimizelyDecision object whose enabled field is false until a rule turns the flag on, and on a critical error, such as the SDK not being ready or an unknown flag key, it returns a nullvariationKey and populates reasons (Decide methods). Your code should treat that state as "off" and fall back to a value you control, rather than assuming a variable will be present. For resilience against the SDK not having a datafile yet, you can also bundle a datafile with the application so the client can initialize synchronously from a known-good snapshot.

Second, decouple the decision from the decision point. Fowler's advice is to avoid scattering raw flag checks throughout the codebase, because changing the logic then means hunting down every call site. Instead, funnel decisions through a single abstraction, a FeatureDecisions object or equivalent, so the toggle point (where behavior branches) is separated from the toggle router (how the decision is made). This keeps business logic testable without the flagging system present, and confines flag removal to one place.

Third, test the states you actually ship. At minimum, exercise the flag-on and flag-off paths, since both are real production states. Wrap decisions so tests can inject a decision directly rather than standing up the SDK, which removes network dependence and the flakiness that comes with it. For the human side of pre-launch verification, a structured pass like the experiment QA checklist catches the targeting, bucketing, and event-tracking mistakes that unit tests miss.

Pay down flag debt before it compounds

Every flag is inventory, and inventory has a carrying cost. Fowler's framing is blunt: "savvy teams view their feature toggles in their codebase as inventory which comes with a carrying cost and seek to keep that inventory as low as possible." Each live flag is an extra branch to reason about, an extra combination to test, and an extra thing that can be misconfigured. Stale flags are the single most common feature-flag problem, and they accumulate because deleting a flag is nobody's urgent job.

The fix is to make cleanup a scheduled, owned activity rather than a good intention:

  • Book the removal when you create the flag. For every release toggle, add a removal task to the backlog at the moment it is introduced. The work of retiring it is then already tracked, not rediscovered months later.

  • Give short-lived flags an expiry. Record an expected removal date on release and experiment flags. Some teams go further and build "time bombs" that fail a test, or even refuse to start the application, once a flag outlives its expiration date, converting silent debt into a loud, actionable failure.

  • Cap the inventory. A Lean-style limit on the total number of live flags forces a retirement before a new flag can be added, keeping the pile bounded.

  • Review the whole set on a cadence. Optimizely's flags dashboard gives you the raw material: every flag with its status, creator, and last-modified date, filterable by environment and audience (Custom Flags Dashboard). A monthly sweep for flags that are fully rolled out, concluded, or untouched for a quarter turns "someone should clean these up" into a concrete list.

When a flag is genuinely done, retire it in both places. Remove the code branch and the flag definition together, so you never leave a dangling key or an orphaned conditional. In Optimizely, archiving a flag removes it from the datafile and requires that it not be active in your highest-priority environment, usually production, which is a useful guardrail against archiving something still serving traffic; the flag's data is retained so you can unarchive later if needed (Custom Flags Dashboard).

How Optimizely maps flags to your code

Pulling the pieces together, here is the shape of the integration. A flag is, in Optimizely's own words, "a place in code where a decision is made." You create a user context from a user ID and any targeting attributes, then call decide with the flag key. The returned decision tells you whether the flag is enabled and carries any configuration variables and the variation key (Key concepts).

The following is illustrative JavaScript, showing the decision routed through a single wrapper with an explicit fallback. Method names are verified against current SDK docs, but exact syntax varies by language and SDK major version.

// Illustrative: one place that owns the decision, with a safe default.
const FALLBACK_SORT = 'alphabetical'; // conservative, always-works behavior

async function getProductSort(optimizely, userId, attributes) {
  // Wait until the SDK has a datafile; on failure, fall back.
  try {
    await optimizely.onReady();
  } catch (err) {
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  const user = optimizely.createUserContext(userId, attributes);
  const decision = user.decide('product_sort');

  // Critical error: variationKey is null and reasons are populated.
  // Treat as "off" and use our own fallback rather than trusting variables.
  if (decision.variationKey === null) {
    console.warn('flag decision failed:', decision.reasons);
    return { enabled: false, sortMethod: FALLBACK_SORT };
  }

  return {
    enabled: decision.enabled,
    sortMethod: decision.enabled
      ? decision.variables['sort_method']
      : FALLBACK_SORT,
  };
}

Two SDK details are worth knowing for production hygiene. First, decide dispatches a decision (impression) event by default so the exposure shows up in results; when you only want to read a flag's state without recording an exposure, the OptimizelyDecideOption.DISABLE_DECISION_EVENT option suppresses it (Decide methods). This is the right tool when you evaluate a flag early to pre-render content and will record the real exposure later at the point of interaction. Second, remember the silent-failure rule from earlier: a flag key in your code with no matching, running rule serves no traffic, so keep code and configuration in lockstep.

The through-line across all of this is modest but powerful. Treat each flag as a small, owned, dated commitment rather than a permanent fixture. Name it so it can be found, ramp it so a mistake is cheap, keep a switch that reverses it fast, and schedule its removal the day you create it. Do that consistently and feature flags go back to being what they were supposed to be: a mechanism for shipping with confidence, not a second codebase you are quietly maintaining by accident.