Statistics you can defend

Expected loss

What you would give up by shipping this variant if it turns out not to be the best. Often the number that actually settles the decision.

Cost of being wrongPer variantBayesian

Significance answers whether you can tell two things apart. Expected loss answers something more useful when you are about to act: if you pick this one and you are wrong, how much worse off are you.

It is computed from the same posterior sampling as the probability of being best. For each draw it measures how far behind the best arm this variant fell, and averages that over all of them. An arm that is usually best and only slightly behind when it is not has a small expected loss.

This is what makes close calls decidable. Two variants that are hard to separate statistically are often trivial to choose between commercially, because being wrong costs almost nothing. Waiting weeks for significance on a decision that cheap is a bad trade of traffic for certainty.

It cuts the other way too. A large expected loss on a variant that is nominally ahead is a warning: the upside is real but the downside is not small, and this is an experiment worth running longer.

Better experiments, better conversions

Test on all visitors with the world’s lightest script and make confident decisions powered by real-time reporting.