Sequential testing

A fixed-horizon test is valid if you look once, at the end. Look every day and stop at the first crossing, and the false-positive rate climbs far above the five per cent you think you are running at. This is not a subtlety; it is the single most common way experimentation programmes fool themselves.

Sequential testing is the test designed for that. Its error rates are controlled under continuous monitoring, so a look on day nine is as valid as a look on day thirty, and a clear result can be acted on when it arrives rather than when the calendar says so.

It also calls the other outcome, which fixed-horizon testing cannot. "No effect of the size this test was built to find" is a real conclusion, and acting on it frees the traffic for the next idea instead of leaving a flat test running out of hope.

It is deliberately conservative about automation. An automated stop requires the sequential test and the corrected p-value to agree, on the same variant, in the same direction. They will not always agree, and the known case is a real effect smaller than the one the sequential test is sized for: it reads no effect while the p-value eventually reads significant, nothing fires, and the experiment runs to its scheduled end. That is the intended direction to fail in.