Frequentist and Bayesian Evaluation

Pertento can report an experiment two ways, chosen with the evaluation method control on the Statistics page. Both read the same data. They differ in the question they answer, and one of them is usually a better fit for the decision in front of you.

Frequentist

Frequentist evaluation asks: if this variant changed nothing, how often would a difference this large appear by chance? That is the p-value, and below 0.05 is the conventional bar.

You get:

• p-value — how easily the observed difference could be explained by chance.
• z-score — how many standard errors separate the variant from the control. Its sign gives the direction.
• Confidence intervals — a range for the variant’s own rate, and a range for its difference against the control. If the whole difference interval sits above zero, the variant is genuinely ahead; if it straddles zero, "no difference" remains consistent with the data.
• A significant / not significant verdict, after any multiple-comparison correction.

The awkwardness of the frequentist approach is that a p-value is not the probability the variant is better, though it is almost universally read as if it were.

Bayesian

Bayesian evaluation asks the question people actually have: which variant is best, and what does choosing wrongly cost?

You get:

• Probability to be best — the chance this variant is the best of those tested. Read it directly as a probability, because that is what it is.
• Expected loss — what you would expect to give up by shipping this variant if it turns out not to be the best. This is frequently the number that settles an argument: a variant with a 70% chance of being best and a negligible expected loss is a cheap decision even if it is wrong.
• Credible interval — a range for the variant’s true rate, taken from the posterior distribution.

Which to use

• Use frequentist when you need a conventional, defensible verdict, or when your organisation already reports on p-values and confidence.
• Use Bayesian when you have to make a decision with imperfect evidence and want to weigh the cost of being wrong. Expected loss makes "good enough to ship" a quantifiable statement.

They can legitimately disagree, and that disagreement is information: it usually means the result is genuinely marginal, and the right response is to consider the size of the effect and the cost of the change rather than to hunt for the method that returns the answer you wanted.

Sequential testing

Alongside both, Pertento reports a sequential test that can call a clear winner or a clear non-effect before a fixed sample size is reached. It answers a different question from the p-value and the two can also disagree without either being wrong.

Notes

• Switching method changes how the same data is reported. It never changes the data.
• Choosing the method that gives the friendlier answer after seeing both is a good way to ship changes that do nothing. Decide which you report on before you look.
• Both methods assume the traffic split is sound. Check that first.