Understanding Significance in the Statistics Page

The Statistics page provides a detailed view of how your experiment is performing and whether the observed differences between variants are statistically meaningful. Each metric on this page helps you determine if you should continue the experiment, stop it or roll out a change.

Results overview

The Results section summarizes the raw performance of each variant. It includes:

• Sessions: the number of visitors who saw the variant
• Conversions: the number of completed conversion events
• Average: the conversion rate for the variant
• Difference: the absolute improvement compared to the original
• Improvement: the relative percentage lift over the original

These values show the current direction of the experiment but do not confirm whether the result is statistically reliable.

Significance section

The Significance section evaluates whether the observed differences are likely due to the variant itself rather than randomness.

Pertento calculates significance in real time using multiple statistical checks. The page displays:

• zScore A standardized measure of how far the variant’s performance deviates from the original.Higher absolute values indicate stronger evidence.
• pValue Indicates the probability that the observed difference happened by chance.Lower values mean stronger confidence in the result.A high p-value typically means the experiment needs more data.
• SPRT Result Shows whether the Sequential Probability Ratio Test recommends continuing the experiment, stopping early with a winner or stopping because the variant is unlikely to outperform the original.This helps prevent wasted traffic and accelerates decision making.
• Significant A combined assessment of all significance models.It tells you whether the result is considered statistically significant based on Pertento’s full evaluation.If marked No Significance, the variant has not yet demonstrated reliable evidence of improvement.

Significance per currency and per metric

Pertento calculates significance independently for every available currency and every active metric in your experiment. This ensures that performance conclusions are not generalized across incompatible datasets. For example, if your site supports multiple currencies or collects several event types, each one is evaluated separately with its own zScore, pValue, SPRT assessment and overall significance status.

This separation allows you to review the stability and reliability of each metric in isolation. A variant may reach significance for one currency or event type while still requiring more data for others. Pertento highlights these differences so you can make decisions based on precise, metric-specific evidence rather than aggregated results.

How to interpret the page

• If the variant shows positive improvement but No Significance, the test is trending in a good direction but still requires more data.
• If SPRT says Continue experiment, the evidence is not yet strong enough to make a decision.
• If SPRT indicates a winner or loser, and the significance indicator confirms this, you can confidently stop the test.
• If the variant’s difference fluctuates heavily over time, it is normal; significance focuses on stability, not short-term spikes.

When to take action

• Roll out the variant when significance is achieved and SPRT confirms a strong winner.
• Stop the experiment early if SPRT determines the variant is highly unlikely to outperform the original.
• Keep running the experiment if metrics show movement but significance has not been reached.