Problem tests left out

Pooling only helps if what goes into the pool can be read. A test whose traffic split was broken, or which recorded no data, is left out of every pattern, because its numbers do not mean what they appear to. Both are measured, so nobody’s word changes them.

The debrief adds what the data cannot see. A test whose debrief reports a tracking issue or a site release during the run is also left out. Any other outside factor the team reports, such as a campaign or a season, keeps the test in but makes it count for less, once, however many factors are ticked, because people tend to tick everything that might apply.

Smaller problems are shown rather than acted on: too few visitors, a run shorter than a week, or an overlap with another test appear as notes on the learning card. The Evidence view lists what is not counted yet, so a test never quietly disappears.

When a reported problem turns out to be fixed, somebody who can create experiments opens the test’s debrief page and chooses Tracking is fixed, include it. That sets aside a factor the team reported. It cannot override a broken split or missing data, which were measured.