Observational vs Experimental Studies: What Randomisation Buys You

Almost every argument about whether a change worked is really an argument about which kind of study produced the number. In an experimental study the researcher decides who gets the treatment. In an observational study they do not — they watch what people did anyway and compare the groups that resulted. That single difference is why one design can support a causal claim and the other, on its own, cannot.

This page is about the distinction itself and why randomising is worth the trouble. It is not about what to do when randomising is off the table: quasi-experimental design for product teams answers that question — how to measure the impact of a change you could not randomise — and it is the page to read next once you accept the argument here. This one comes first.

What makes a study experimental

A study is experimental when the researcher controls assignment to the treatment. Not when it is quantitative, not when it is run in production, not when it has a control group in some loose sense. The test is narrow and mechanical: did somebody decide, by a rule independent of the participants, who would be exposed?

An A/B test qualifies. The bucketing hash does not know anything about the visitor's intent, their history or their likelihood to convert. It assigns them to a variant and then the outcome is measured. Everything else in the design — the metric, the duration, the guardrail metrics that catch collateral damage — is craft on top of that one structural property.

The property matters because it decides what else can differ between the groups. When assignment is random and the sample is big enough, the two groups are alike in every respect, including the ones nobody thought to record: motivation, prior tenure, device, mood, whether they arrived from a discount email. They are alike in unmeasured variables as well as measured ones, which is the part no amount of analysis can reproduce after the fact.

What an observational study is measuring instead

An observational study compares groups that formed themselves. Users who adopted the new feature against users who did not. Accounts that attended onboarding against accounts that skipped it. Visitors who opened the chat widget against visitors who ignored it.

These comparisons are not worthless — they are frequently the only data available, and they are where most hypotheses come from. But the groups differ in the treatment and in whatever made them choose it. The comparison estimates the sum of those two things, and nothing in the arithmetic separates them.

The usual shape of the error is flattering. People who opt into a feature are more engaged than people who do not; engaged people retain better regardless. So the feature inherits credit for the engagement that caused the adoption. The observational number is not noisy — with enough data it is beautifully precise. It is precisely measuring the wrong quantity.

Confounding, and a result that reverses

A confounder is a variable that influences both who gets the treatment and what the outcome is. Prior engagement is the classic one in product analytics, and it does more than inflate an effect. It can flip its sign.

Here is a worked case. A team ships a feature and looks at 90-day retention for users who adopted it against users who did not.

A 36-point gap on 2,000 users. It would clear any significance threshold you care to name, and it is the number that ends up on a slide.

Now split both groups by how active the user already was in the month before the feature shipped.

Within heavy users, adopters retain 12 points worse than non-adopters. Within light users, adopters retain 10 points worse. The feature is associated with lower retention in every segment, and with dramatically higher retention overall.

heavy users:  77.8%  vs  90.0%   ->  -12.2 points
light users:  20.0%  vs  30.0%   ->  -10.0 points
combined:     72.0%  vs  36.0%   ->  +36.0 points

Nothing is wrong with the arithmetic. The aggregate is dominated by the fact that 90% of adopters were heavy users and 90% of non-adopters were light ones, and heavy users retain far better either way. The aggregate comparison is mostly measuring who adopted, not what adoption did. This reversal has a name — Simpson's paradox — but the name is less useful than the mechanism: the groups were not comparable, and the imbalance was larger than the effect.

Randomise the same feature and the confounder disappears by construction. Force-enable it for a random half of eligible users, leave the other half alone, and after 90 days retention comes back at 41.5% in the treatment arm against 43.0% in the control — a small negative effect, consistent with what both segments were quietly saying. The observational study said the feature was the best thing the team had ever shipped. The experiment said it costs about a point and a half of retention.

Two footnotes on that experiment, because they are the ones people skip. Exposure has to be checked before the result is read: if the arms arrive at unequal sizes you have a sample ratio mismatch and the randomisation you are relying on did not happen. And a feature that is newly forced on people gets a reaction to its novelty as well as to its usefulness, which is why the read happens after the novelty and primacy effects have settled rather than on day two.

Why randomisation licenses the causal claim

Correlation is a statement about the data you have. Causation is a statement about a world that did not happen: what the same users would have done under the other condition. You never observe both, for anybody. Every causal estimate is a comparison against an unobserved counterfactual, and the whole design problem is finding a group that stands in for it credibly.

Randomisation solves that by making the control group a fair sample of the same population as the treatment group — fair in the measured covariates and, crucially, in the unmeasured ones. Then the difference in outcomes has only one remaining explanation besides chance, and the p-value is what quantifies the chance part.

This is also why randomisation is one of the three principles of design of experiments as a discipline rather than a nicety: randomisation removes systematic bias, replication gives you an estimate of the noise so you can tell a real effect from the noise, and blocking removes a known source of variation before it can obscure the effect. An online A/B test is a very simple instance of that framework — one factor, two levels — which is why the framework's vocabulary shows up the moment you run more than one factor at a time in a multivariate test.

Adjusting for what you measured is not the same thing

The obvious response to the reversal above is to control for prior engagement. Stratify, or put it in a regression, and the segment-level answer is recovered. That is correct and it is worth doing.

It is not equivalent to randomising, for one reason: it only works on confounders you thought of, recorded, and measured well. Randomisation balances the variables nobody has a column for. Adjustment balances the columns you have.

That gap is why observational estimates need an argument, not just a model. Which confounders could plausibly explain this? Are they all in the data? How badly measured are they? The honest observational analyst spends most of their time there, and a serious observational study states which unmeasured confounders would have to exist to overturn the result. An experiment does not need that paragraph, which is most of what you are buying.

Segment-level differences are a legitimate finding in an experiment too — treatment effects genuinely vary between people — but there the segments are compared within a randomised design. A/B test segmentation covers how to do that without turning a randomised experiment back into an observational one by slicing it after the fact, and false discovery rate control covers what happens to your error rate when you slice it many ways.

When observational is the right design

Randomising is not always available, and treating "we could not randomise" as a reason to learn nothing is its own failure. There are four common cases where the observational design is the correct one, not a compromise.

It would be unethical. You cannot randomly assign harm. Nobody runs a trial that assigns half the sample to smoke for twenty years, and the entire causal case against smoking was built from observational data plus a mechanism, a dose-response relationship and a great deal of argument about confounders. In a product setting the same logic applies to deliberately degrading a service, withholding a security fix, or exposing a protected group to a worse price.

It is impossible to assign. Some treatments cannot be handed out: a competitor's launch, a macroeconomic shock, a platform policy change, a customer's own industry. Nobody is randomising the weather.

The unit is too coarse or too few. Brand campaigns, pricing changes in a market, sales-territory changes: the natural unit of assignment is a country or a region, and there are twelve of them. That is not a sample you can randomise into balance, and it is where the quasi-experimental toolkit was invented.

It is the reconnaissance step. Observational data is where hypotheses come from and where you decide what is worth the cost of an experiment. A test needs traffic and time to reach a conclusion, and both are finite, so the observational pass that tells you where the effects might be is doing real work. It just should not be the last step.

What to do when you cannot randomise

The answer is not to fall back on the naive comparison and hope. It is to find something that stands in for the randomisation: a comparison group whose fate you have a reason to believe would have tracked the treated group, or a rule that assigned treatment for reasons unrelated to the outcome.

That is the subject of quasi-experimental design for product teams, which works through the designs that do this — difference-in-differences, synthetic control, regression discontinuity, interrupted time series — and, more importantly, the assumption each one needs in order to be believed. The assumptions are the substance. Each of those designs is trading the guarantee that randomisation gave you for an argument that a particular kind of comparison is good enough, and knowing which argument you are making is the difference between a defensible estimate and a naive comparison with more notation.

The other route, when the treatment is a spend rather than a product change, is to randomise something adjacent even if you cannot randomise the treatment. Holding out a random slice of geographies or of an audience from a campaign is a real experiment on a coarser unit, and incrementality testing covers how that is set up and read.

How to tell which one you are reading

When a number lands on your desk, the question is not whether the analysis was careful. It is who decided the assignment.

flowchart TD
  A[A result comparing two groups] --> B{Did a rule independent of the<br/>participants assign the treatment?}
  B -- yes --> C{Was that rule random?}
  B -- no --> D[Observational: the groups chose themselves]
  C -- yes --> E[Experimental: the comparison is licensed]
  C -- no --> F[Quasi-experimental: licensed by an assumption<br/>that has to be stated and defended]
  D --> G[Ask which confounders explain it,<br/>and whether they are in the data]
  F --> G

Three questions get you most of the way there:

1. Who assigned the treatment? If the answer is "the users did", it is observational, whatever the sample size.

2. What else differs between the groups? For an observational result, list the plausible confounders out loud before looking at the estimate. If prior engagement, tenure or intent is on that list — it usually is — the raw comparison is not the effect.

3. What was the comparison group's claim to be a counterfactual? Randomisation, an assumption, or nothing at all. All three are answers. Only the third is disqualifying.

The reason this matters commercially is that observational estimates are biased in a consistent direction: towards whatever the enthusiastic users were already going to do. A roadmap built on them systematically over-invests in features that attract engaged users and under-invests in the ones that create engagement. Getting the experiment design right on the questions that justify the cost, and being explicit about the assumptions on the questions that do not, is what keeps that from compounding.