The Variant Combination Report

When two experiments run on the same pages at the same time, some visitors see both changes. Usually that is harmless and each test measures its own effect correctly. Occasionally the two changes interact — they reinforce each other, or they interfere — and each experiment’s individual result becomes misleading.

The Combination Report is where you check. Select two overlapping experiments and it reports every combination of their variants that visitors actually saw.

Reading the interaction

The report’s central figure is an interaction odds ratio, with a confidence interval, and it is described in words as well as numbers:

• Above 1 and significant — the two changes reinforce each other. Together they do better than their individual effects predict.
• Below 1 and significant — the two changes interfere. Together they do worse than their individual effects predict.
• Interval includes 1 — no interaction detected. As far as this data can tell, the two changes behave independently, and each experiment’s own result stands.

The third case is the common one, and it is the reassuring one: it means you can read each experiment normally.

Why there is no conversion rate

The report shows conversions and average order value per combination, but deliberately no conversion rate, and it says so on the page. This is a limitation worth understanding rather than a missing feature.

A rate needs exposure — how many visitors saw each combination — and exposure per combination is not reliably measurable. Impression data is retained for around two weeks while conversions are kept for years; the runtime records one impression per experiment per page view rather than per visitor; and a visitor’s set of enrolments fills up during their visit, so early page views carry incomplete combinations.

What *is* measurable from conversions alone is the interaction odds ratio, because allocation cancels out of it, and average order value, because revenue and conversions come from the same events. So the report gives you those and does not rank combinations by a rate it cannot compute honestly.

Using it

1. 1Run the report when two experiments overlap on the same pages, particularly if both touch the same funnel step.
2. 2If no interaction is detected, read each experiment as normal.
3. 3If an interaction is detected, do not roll out both winners on the strength of their individual results. The combined effect is not the sum of the parts.
4. 4Where an interaction is strong, separate the tests — by URL targeting or by running them in sequence — and retest.

Notes

• The report needs both experiments to have run concurrently with enough shared traffic to compare.
• Overlap is normal and mostly harmless. The report exists so you can confirm that rather than assume it.
• Average order value is comparable across combinations; conversion counts alone are not, because allocation differs.