Reading the Power Analysis Card

The power analysis card on an experiment’s Statistics tab answers the question a p-value cannot: is this experiment capable of detecting the difference it is looking for? It is the difference between "this change does nothing" and "we have not collected enough data to tell", which are two conclusions people routinely confuse.

Statistical power

Power is the chance this experiment would detect the difference it is currently measuring, assuming that difference is real. It is colour-coded, and the thresholds are worth internalising:

• 80% or above — sufficient. A flat result at this level is meaningful evidence that the change did not do much.
• 50% to 80% — borderline. Keep running.
• Below 50% — a null result tells you nothing at all. The experiment could easily miss a real effect of the size being measured.

This is the single most useful number for deciding whether to stop a test that has not reached significance. Below 80%, "not significant" means "not yet", not "no".

Resolvable effect

The resolvable effect is the smallest change in conversion rate this much traffic can reliably detect, expressed in percentage points.

It is a floor, and this is the part that surprises people: a difference smaller than this figure will not become significant by waiting at your current traffic level, however long you leave the test running. If your resolvable effect is 1.5 points and you are measuring 0.4, the honest answer is that this experiment cannot settle this question and no amount of patience changes that.

When that happens, your options are a bigger change, a higher-traffic page, or accepting that the difference is too small to matter commercially.

Time remaining and required sample size

The third figure projects when the experiment will have enough traffic to settle the difference it is currently measuring. On a finished experiment it becomes the sessions per variant that would have been needed, at 95% confidence and 80% power.

This projection moves as the measured difference moves, and it is not a bug. A smaller measured difference needs more traffic to confirm, so the estimate lengthens when the gap between variants narrows. Treat it as a live projection from the data so far, not a deadline.

Using it to decide

1. 1Power below 80% and no significance — keep running, if the projected time is acceptable.
2. 2Power above 80% and no significance — the change genuinely did not move the metric enough to matter. Stop and move on; that is a useful result.
3. 3Measured difference below the resolvable effect — stop. Waiting cannot resolve it.
4. 4Projected time is longer than you can wait — stop and test something bolder rather than running an underpowered test to no conclusion.

Notes

• Power is measured over the window the experiment was actually live, not up to today.
• The figures are sized against the difference currently being measured, not a fixed target set in advance.
• Read the traffic split check above the results first. If the split is wrong, none of these numbers describe your change.