Cohort Analysis for Experimentation Teams

Cohort analysis is the habit of grouping users by something they share, then following each group forward in time instead of averaging them together. It answers questions an overall number hides: whether the product got better for people who joined this month, whether a change helped new users and hurt returning ones, and whether a lift that looked real in week one is still there in week six.

It is also one of the easiest analyses to misread. A cohort table is observational. Two cohorts differ by when they arrived and by everything that was different about the world when they arrived, so a gap between them is a description, not a cause. This article covers what cohort analysis is good at, where it stops, and how to use it on an Optimizely experiment without turning an honest readout into a story.

What a cohort actually is

A cohort is a set of users pinned to a shared starting event and then measured on the same clock. Two properties matter, and both are easy to lose.

The first is a fixed membership rule. Once a user is in the January cohort, they stay in it. If a user who signed up in January is quietly re-counted in March because they came back, the table stops measuring anything: every row is contaminated by every later row.

The second is aligned time. Cohorts are compared on relative time — week 0, week 1, week 2 from their own start — not on calendar time. This is what makes a young cohort comparable to an old one. Plotted on a calendar axis, a cohort that started later simply looks worse because it has had less time; plotted on relative weeks, the two can be read side by side.

Everything else about cohort analysis follows from those two rules. A cohort table with a moving membership rule or a calendar x-axis will produce confident-looking numbers that mean nothing.

Acquisition cohorts and behavioural cohorts

Two shapes cover almost every real use, and they answer different questions.

Acquisition cohorts

Members are grouped by when they first arrived: the signup week, the first purchase month, the release they installed on. This is the shape people usually mean by "cohort analysis", and it answers a product-health question — is the thing we ship getting better or worse for each new intake?

Acquisition cohorts are the right tool for retention curves, payback periods and the slow effects that a two-week test cannot see. They are the wrong tool for attributing a change, because the cohort boundary is a date and dates carry everything: a pricing change, a seasonal peak, a competitor's outage, and the feature you actually care about.

Behavioural cohorts

Members are grouped by an action they took: users who completed onboarding, users who used search at least twice, users who connected an integration. The question is different — among people who did this, what happened next?

Behavioural cohorts are more useful and more dangerous. More useful because the grouping is closer to the mechanism you care about. More dangerous because the behaviour is itself an outcome. "Users who connected an integration retain at 70%" is almost always a statement about which users bother to connect integrations, not about what connecting one does.

Why cohort analysis is not a substitute for an experiment

The gap between two cohorts mixes three things that a table cannot separate: the change you made, everything else that changed at the same time, and the fact that the two groups are made of different people.

Consider a redesigned onboarding flow shipped on 1 March. The March cohort retains four points better than the February one. Three explanations fit that table equally well. The onboarding flow works. March traffic came from a different mix of channels, and those users were always going to retain better. February contained a holiday week, and holiday signups retain badly everywhere.

A randomised experiment removes the second and third explanations by construction, because the two groups are drawn from the same traffic at the same time. That is the whole reason to run one. If a randomised comparison is available, cohort analysis is not the answer to "did this work" — it is the answer to "for whom, and for how long".

When randomisation genuinely is not available — the change went to everyone, or the unit of exposure is a whole market — the honest tools are the quasi-experimental designs, not a cohort table read as if it were causal. The published guide to quasi-experimental design for product teams covers what those designs assume and how to test the assumptions.

Reading an experiment by cohort without inventing a result

Cutting a finished experiment by cohort is legitimate and useful. It is also the most common route to a false discovery on any experimentation team, because a table of twelve cohorts by four metrics is forty-eight chances for noise to look like a finding.

Three cuts worth making

New versus returning. The single most informative cut, because the two populations often experience a change in opposite directions. A navigation change that helps a first-time visitor find something can slow down a returning user who had already learned the old layout.

Time since exposure. Split the treated group by how long ago they first saw the variation. A lift that exists only in the first session is a novelty effect, and it will decay. A lift that grows with weeks since exposure is a learned behaviour, and it will not show up fully inside the test window.

Acquisition channel. Paid, organic and referral traffic differ enough that a single average can hide a variation that works for one and fails for another. This is the cut that most often changes a rollout decision rather than merely decorating it.

What to do when a cohort looks different

Treat it as a hypothesis, never as a result. A cohort difference found after the fact has no error control: it was chosen because it looked large, which is exactly the condition under which the estimate is inflated. The published note on the winner's curse and regression to the mean explains why the second measurement is almost always smaller than the first.

The disciplined response is to pre-register the cut in the next experiment and power it properly. A cohort that is a fifth of your traffic needs roughly five times the sample to detect the same effect, which is usually the real reason a subgroup finding cannot be confirmed. The sample size calculator will show the size before the test rather than after it. For the statistical machinery underneath subgroup readouts, see the guide to segmentation and heterogeneous treatment effects.

Building cohorts from Optimizely events

Optimizely stores the variation a user saw and the events they fired. That is enough to build cohorts, provided the join key is stable and the cohort attribute is captured at the right moment.

Two rules keep the data usable. Send the cohort attribute as an event property at the time the event happens, not as a user attribute you overwrite later. And key everything to the same identifier the experiment bucketed on, so a cohort cut cannot silently mix two identity spaces.

// Fire the conversion with the cohort attributes attached to the event itself,
// so a later profile update cannot rewrite the history of an old cohort.
window['optimizely'] = window['optimizely'] || []
window['optimizely'].push({
  type: 'event',
  eventName: 'subscription_started',
  tags: {
    revenue: 4900,
    signup_week: user.signupWeek,       // '2026-W31', assigned once at signup
    acquisition_channel: user.channel,  // 'paid' | 'organic' | 'referral'
    visitor_type: user.isReturning ? 'returning' : 'new',
  },
})

Exported to a warehouse, the cohort table is a group-by on the signup week and the number of weeks between the signup and the event.

SELECT
  signup_week,
  DATE_DIFF(event_week, signup_week, WEEK) AS weeks_since_signup,
  COUNT(DISTINCT visitor_id)              AS active_visitors
FROM experiment_events
WHERE variation = 'treatment'
GROUP BY signup_week, weeks_since_signup
ORDER BY signup_week, weeks_since_signup;

The DISTINCT matters. Counting events rather than users turns a table about retention into a table about how loudly a few users click.

A cohort table that survives review

The flow below is the order the work has to happen in. Reversing any two steps produces a table that looks the same and answers a different question.

flowchart TD
  A[Fix the cohort rule<br/>before looking at data] --> B[Assign each user<br/>once, permanently]
  B --> C[Align on relative time<br/>week 0, 1, 2 from own start]
  C --> D[Count distinct users,<br/>not events]
  D --> E{Cohort gap<br/>looks large?}
  E -->|Yes| F[Write it down<br/>as a hypothesis]
  E -->|No| G[Report the curve,<br/>make no claim]
  F --> H[Pre-register and power<br/>the cut in the next test]

Three habits make the difference between a table that informs a decision and one that decorates a deck. State the cohort rule in writing before the first query, so it cannot drift to fit the result. Show the cohort sizes next to the rates, because a 90% retention rate on eleven users is not a finding. And keep the last incomplete period visibly marked, since a cohort that has only had three days to convert will always look like a collapse.

Frequently asked questions

Is cohort analysis the same as segmentation?

No. Segmentation splits users by an attribute and compares them at a moment. Cohort analysis pins users to a start event and follows them over relative time. A cohort is a segment with a clock attached.

How many users does a cohort need?

Enough that the metric's confidence interval is narrower than the difference you would act on. There is no universal floor, but a cohort whose interval spans both "much better" and "much worse" cannot support any decision, and reporting its point estimate without the interval is how those cohorts end up in strategy documents.

Can cohort analysis prove causation?

Not on its own. Cohorts differ by arrival date and by everything correlated with it. Causal claims need randomisation, or a quasi-experimental design whose assumptions have been tested and stated.

What period length should a cohort use?

Match it to the product's natural cycle. Weekly cohorts suit products used several times a week; monthly cohorts suit subscription products with monthly billing. Whatever the choice, keep it fixed — switching from weekly to monthly halfway through makes older cohorts incomparable to newer ones.

Should the first period be counted as week 0 or week 1?

Week 0, and label it. Week 0 always shows near-100% activity, because membership in the cohort is defined by acting in that period. Presenting it as week 1 makes the first drop look like a catastrophe when it is a definition.