Statistics you can defend

Minimum detectable effect

The smallest change this much traffic can reliably find. A difference below it will not become significant by waiting.

A floor, not a targetIn percentage pointsStated against the base

This is the failure mode that looks like "no result" and is really "not enough traffic to ever tell". An experiment measuring a difference smaller than the floor its sample can resolve will sit at not-significant indefinitely, and no amount of patience changes that at the current traffic level.

So the console reports the floor directly, in percentage points, alongside the control rate it is a fraction of. Half a percentage point means nothing without knowing whether the base rate is one per cent or forty.

It also states plainly whether the difference being measured is above or below that floor. Below it, the experiment cannot resolve what it is showing, and the useful responses are different from the ones significance suggests: test something bolder, move to a higher-traffic page, or accept the question is not answerable on this template.

The floor and the required sample size are defined as exact inverses of each other rather than by separate formulas. They used to be computed from different variance assumptions and so disagreed by a few per cent, which mattered because the same panel shows both and one of them decides whether a difference is resolvable.

Better experiments, better conversions

Test on all visitors with the world’s lightest script and make confident decisions powered by real-time reporting.