Post Hoc Power Analysis: Why Recalculating Power After a Null Result Fails

An A/B test comes back inconclusive — no significant difference between variation and control — and someone on the team asks a reasonable-sounding question: "was the test even powered enough to detect a real effect?" The instinct is to run a power calculation using the effect size the test actually observed, plug it back into a power formula, and report the number. That calculation has a name, post hoc power analysis, and a well-documented problem: it is mathematically redundant with the p-value you already have, and treating it as new evidence is one of the most common ways a null result gets quietly reinterpreted as "probably a win we couldn't detect." This guide covers why the calculation doesn't do what it appears to do, what to check instead, and how to size a test correctly the first time so you never need the retrospective version.

What post hoc power analysis actually calculates

Prospective power — the kind you calculate before a test launches — asks: "if the true effect is X, and I collect N visitors, what is the probability I detect it as significant?" That calculation depends on three things you choose in advance: the minimum effect worth detecting, your sample size, and your significance threshold. It is a planning tool, and it is exactly what a sample size calculation is for.

Post hoc power analysis takes the same formula and swaps one input: instead of the minimum detectable effect you decided was worth chasing, it plugs in the effect size the experiment actually observed. Then it reports the probability that a test of this size would have detected an effect of this exact magnitude — the one already sitting in your results.

Why the observed-effect version is circular

The problem is not the arithmetic; it is what the output means. Prospective power tells you something you did not already know — whether your planned sample size is big enough for an effect you care about. Post hoc power computed from the observed effect tells you nothing you cannot already read off the p-value, because the two are mathematically tied together: for a fixed sample size, post hoc power is a strictly decreasing function of the p-value. A high p-value (a null result) will always produce a low post hoc power calculation, and a low p-value (a significant result) will always produce a high one. The calculation cannot come back and say "actually, this null result had high power" — that combination is not mathematically possible.

That means a stakeholder who asks for post hoc power on a null result and receives a low number has learned nothing beyond "the result was not significant," restated in a more technical-sounding unit. The number feels like independent confirmation that "we just needed more data," but it is an algebraic reflection of the p-value you already had, not a second measurement.

The retrospective-power fallacy in practice

The fallacy shows up in a specific, recognizable pattern: a test ends without significance, the team is reluctant to call the idea a loss, and a post hoc power calculation comes back low — say, 40%. That 40% gets reported as "the test was underpowered," which sounds like a design flaw rather than a null result, and the conclusion drawn is "the effect is probably real, we just need to run it longer or on more traffic."

That conclusion does not follow from the number. A low post hoc power figure is a restatement of "not significant," and it carries no information about whether the true effect exists at all. The actual data available for that judgment is the confidence interval around the observed effect, not a recomputed power figure — and the confidence interval is the tool that should be doing this work.

What to check instead of retrospective power

If a null result raises the question of whether the test just needed more data, the honest way to answer it is to look at the confidence interval around the effect estimate, not to recompute power.

A wide confidence interval that spans zero — say, a 95% CI of −1.5% to +4.0% on relative lift — genuinely is consistent with "we don't know yet, and a bigger sample might resolve it." That is real information: the interval is telling you the experiment did not have enough precision to distinguish a meaningful effect from no effect. This is functionally the same conclusion a well-designed prospective power calculation would have flagged before you ever launched, which is why sizing the test properly the first time — using an honestly chosen minimum detectable effect — avoids the whole situation.

A narrow confidence interval that spans zero — say, −0.3% to +0.4% — tells a different story: the test had plenty of precision, and it precisely located the effect at approximately zero. Rerunning that test with more traffic will not turn up a real effect, because the data already ruled out anything but a trivial one.

Wide CI spanning zero:    [-1.5%, +4.0%]  -> underpowered for this effect size; more data could resolve it
Narrow CI spanning zero:  [-0.3%, +0.4%]  -> well-powered; the effect is precisely estimated near zero

Both examples are "not significant." Only the confidence interval tells you which kind of "not significant" you are looking at, and that distinction is the actual answer to the question post hoc power analysis is being asked to provide.

How this connects to peeking and effect size

Post hoc power analysis is a close cousin of two other retrospective mistakes covered elsewhere on this site. Effect size reporting already explains why comparing the observed lift against your original MDE — rather than recalculating anything — is the correct way to judge whether a result was meaningfully sized. And the temptation to keep collecting data until the picture improves is exactly the optional-stopping problem: if you extend a test because a retrospective power number felt low, you are running an unplanned, uncontrolled extension of an experiment whose original stopping rule you have already broken.

The common thread across all three is the same: decisions made by looking backward at data you already have, dressed up as if they were new evidence, inflate your real error rate without showing up anywhere in the reported statistics.

When a retrospective power calculation is legitimate

There is one legitimate retrospective use of a power calculation, and it does not use the observed effect size at all. Before deciding whether to run a follow-up experiment, you can calculate what sample size would be needed to detect a specific, externally chosen effect size — for example, the smallest lift that would be worth the engineering cost of shipping the change permanently. That is a forward-looking calculation about a new, hypothetical test, using an effect size you chose for a reason unrelated to what the last test happened to observe. It looks similar to post hoc power analysis on the surface, but the input is a decision, not a readout, which is what keeps it from being circular.

Common mistakes

• Reporting post hoc power as if it were independent evidence. It is a deterministic function of the p-value you already had; it does not add information.

• Using the observed effect size as the input, then acting surprised the number is low on a null result. That outcome is guaranteed by the math, not a discovery about the experiment.

• Extending a test's runtime because a retrospective power figure "felt low." This breaks your stopping rule and reintroduces the peeking problem under a different name.

• Skipping a confidence-interval read entirely. The CI is doing the real diagnostic work a post hoc power calculation is being asked to do — check it first.

Key takeaways

• Post hoc power analysis recomputes power using the effect size a test actually observed, and that number is mathematically tied to the p-value you already have — it is not new evidence.

• A null result will always produce a low post hoc power figure; that is guaranteed by the algebra, not a finding about whether the effect is real.

• The confidence interval around the observed effect — not a recalculated power figure — is the correct tool for judging whether a null result reflects "no effect" or "not enough precision to tell yet."

• The fix that avoids the question entirely is sizing the test honestly before it launches, using a minimum detectable effect you actually intend to act on.

Frequently asked questions

Is post hoc power analysis ever useful?

Only when it uses an externally chosen effect size to plan a genuinely new test — for example, sizing a follow-up experiment around the smallest lift worth shipping. Using the just-observed effect size from the test you're analyzing is the circular version to avoid.

My test came back with low post hoc power. Does that mean I should run it longer?

Not on its own. Look at the confidence interval around the observed effect instead. A wide interval spanning zero suggests genuine imprecision that more data could resolve; a narrow interval spanning zero means the effect is precisely estimated at approximately zero, and more data will not change that conclusion.

How is this different from a normal, prospective power calculation?

A prospective power calculation, done before the test launches, uses an effect size you chose because it is the smallest one worth detecting — that is what a sample size calculator does. Post hoc power analysis instead plugs in the effect size the test happened to observe, which makes the output a restatement of the p-value rather than independent planning information.