Correcting for Multiple Variants

Every comparison you make is another chance for a variant to look like a winner by luck. Test one variant at the usual 0.05 threshold and there is roughly a 5% chance of a false positive. Test five and that chance rises to about 23%.

The correction method control on the Statistics page compensates by tightening the bar. Pertento offers four options.

The methods

• Holm-Bonferroni — the strictest. With five comparisons the hardest step is 0.01, relaxing towards 0.05 for the weaker results, and it stops at the first comparison that fails. Use it when a false winner would be expensive.
• Benjamini-Hochberg — controls the *share* of false positives among the variants you declare winners, rather than the chance of any false positive at all. Less strict than Holm, and usually the better trade-off when you are screening several ideas and can tolerate the occasional dud.
• Šidák — applies one slightly relaxed threshold to every comparison. Marginally less strict than Bonferroni, and assumes the comparisons are independent.
• No correction — every comparison judged at 0.05 with no adjustment. Honest only when you have a single variant.

What a correction does and does not change

This is the most common misunderstanding, so it is worth stating plainly. A correction only changes the significant / not significant verdict. It never changes the p-value, the confidence interval or the z-score, all of which are properties of the data.

So if you switch methods and the numbers do not move, nothing is broken. The threshold they are judged against moved instead.

With one variant, corrections do nothing

When there is only one variant being compared, all four methods reduce to the same 0.05 threshold. Switching between them cannot change the result, and Pertento says so on the page rather than leaving you to test each option.

Most experiments are in this position, which is why the control appears inert most of the time. It becomes meaningful the moment you add a second variant.

Practical advice

• Fewer variants is better. Each one costs traffic and raises the bar. Three well-chosen variants beat six speculative ones.
• Decide the correction before you look at the results. Choosing it afterwards, on the basis of which method makes your favourite variant significant, is exactly the behaviour corrections exist to prevent.
• Corrections do not rescue an underpowered test. If the power analysis says the experiment cannot resolve the difference, no threshold makes the answer trustworthy.

Notes

• The correction applies across the variants in one experiment, not across separate experiments.
• A corrected result that is no longer significant has not been disproved. It means the evidence does not clear the higher bar that testing several variants requires.
• Corrections apply to frequentist evaluation. Bayesian reporting handles multiple variants through the probability each is best.