Statistics you can defend

Combination reporting

When two experiments run at once, do their changes interact? A narrow question, answered properly.

Two or more testsOdds ratioOne website

Running experiments concurrently costs nothing in sample size, because each randomises independently. What it can cost is additivity: two changes that each work alone may not combine the way you would assume. Combination reporting is the check for that.

Pick two or more experiments on the same website and the report measures whether their effects interact, and whether order values differ between the combinations visitors landed in. It is computed from conversion counts alone using an odds ratio, and the algebra works out because the allocation terms cancel exactly under independent randomisation.

It is precise about what it does not answer. Without an exposure denominator it cannot tell you which combination converts best; it can tell you whether the two changes interact, which is the question that decides whether each result will replicate on its own.

This is the narrow, statistical question. The broader operational one, whether two experiments are corrupting each other rather than merely combining, is what collision and interaction detection cover, and those raise themselves without being asked.

Better experiments, better conversions

Test on all visitors with the world’s lightest script and make confident decisions powered by real-time reporting.