When Traffic Does Not Split as Configured

Above the results on every experiment’s Statistics tab, Pertento reports whether traffic actually split the way you configured it. When it did, this is a single quiet line. When it did not, it becomes a warning, and it is placed above the results deliberately: it decides whether anything below it is worth reading.

This check is for what the industry calls a sample ratio mismatch, or SRM.

Why it invalidates results

An A/B test works because the groups are equivalent apart from the change. If assignment is faulty, the groups differ in some other way too, and every figure below the check is measuring that fault rather than your change.

A mismatch is not a small data-quality nuisance. It means the experiment is not an experiment, and a confident-looking p-value on mismatched traffic is worse than no result at all, because someone will act on it.

How the check works

Pertento runs a chi-square goodness-of-fit test comparing the sessions each variant received against the split you configured. The alarm threshold is deliberately strict — far stricter than the usual 0.05 — because with real traffic volumes small harmless fluctuations would otherwise trigger it constantly.

The check does not run when it cannot say anything useful, and tells you which case applies:

• Fewer than two variants receiving traffic.
• Fewer than 100 sessions in total, where the split cannot mean anything yet.
• No traffic weights configured.

Deployed variants are excluded, because a rollout is not a randomised arm and its share of traffic is not something the configured split predicts.

Common causes

Work through these in order:

1. 1A variant failing to apply. If a variant’s changes error on some pages, those visitors may never be registered. This is the most frequent cause.
2. 2A redirect losing visitors. In a URL Redirect experiment, visitors who abandon during the redirect are lost from that arm only. The slower the redirect, the worse the imbalance.
3. 3Filtering that hits one arm harder. Bot filtering or consent gating that correlates with the variant removes traffic unevenly.
4. 4Configuration changed mid-run. A variant added, paused, or reweighted while the experiment ran. The activity log will show it, and the split for the run as a whole will not match any single configuration.
5. 5Weights that do not match reality. On a Server Side experiment, this usually means the configured weights do not describe what your own allocator does. Fix the configuration, not the allocator.

What to do about it

Treat every result on the experiment as suspect until you have an explanation. Find the cause, fix it, and start a fresh experiment — data collected during a mismatch cannot be repaired after the fact, and salvaging it by excluding a period reintroduces exactly the bias the randomisation was there to remove.

Pertento also raises a mismatch as a notification and lists the experiment as Broken on the Health page, so it does not depend on somebody opening the Statistics tab to be noticed.

Notes

• A healthy check is shown as one line, with the per-variant detail on hover. It stays small on purpose.
• The traffic weights may drift slightly from what you typed, because Pertento corrects for assignment drift automatically. The check accounts for the configured values.
• A mismatch is about *who was assigned where*, not about conversion rates. It is a validity problem, not a result.