Statistics you can defend

Sample-ratio-mismatch alarms

The check that runs before every other number means anything: did traffic actually split the way you configured it.

Chi-squareStrict thresholdBlocks automation

A sample ratio mismatch means the split you configured is not the split that happened. Usually the cause is a variant failing to apply, a redirect losing visitors on the way, or bot filtering hitting one arm harder than the other.

It is a validity gate rather than a metric, and that is why it is checked first. If assignment is broken the two groups are not comparable, and every p-value, interval and recommendation on the page is measuring that fault rather than measuring your change.

The test is a chi-square goodness-of-fit against the configured weights, at a threshold far stricter than the usual one. With real traffic the split is never exactly nominal, and at 0.05 a perfectly healthy experiment would trip the alarm one time in twenty, which teaches everybody to ignore it. It also declines to run at all below a minimum sample or with fewer than two arms taking traffic, because at that point an absence of evidence is not evidence of a problem.

Exactly one implementation of this exists. The alarm that emails you, the screen you land on and the automation that refuses to act all reach the same verdict from the same code, because an alert that contradicts the page it links to is worse than no alert.

Better experiments, better conversions

Test on all visitors with the world’s lightest script and make confident decisions powered by real-time reporting.