Power analysis

Power is the chance an experiment would detect a real difference of a given size. Below roughly eighty per cent, a null result means "not enough data yet" rather than "no effect", and treating those two as the same thing is how a good idea gets abandoned.

Pertento sizes the analysis against the difference the experiment is actually measuring, not against a fixed target lift chosen in advance. Sizing against a hardcoded five per cent answers "when could I detect five per cent", which is not the question anybody asks of a running experiment. The question is when the effect I am seeing becomes conclusive.

There is a deliberate guard in that calculation. The measured difference is never fed back through the detectable-effect floor, which is itself derived from the current sample. Doing so makes the answer self-fulfilling and reports sufficient data for an experiment that is badly underpowered.

When the measured difference is too small to size against at all, a near-zero effect needs an unbounded sample, the analysis falls back to a target lift and says which of the two it used. A precise-looking number with no indication of what it assumed is worse than an honest one.