Adjusted for multiple comparisons

Learnings checks dozens of patterns at once: every page and change type, every mechanism, theme and element. The more ways a website’s tests are sliced, the more likely it is that one slice clears zero by chance alone. A page that drew every pattern’s range at the usual 95% would present a few lucky streaks as findings.

So every range on the page is drawn at one adjusted width, from the Benjamini-Hochberg procedure with false coverage rate intervals at 5%. A pattern selected as conclusive has a range that excludes zero, and one that is not selected has a range that crosses it, so the evidence level and the range can never disagree.

The checks are counted as the distinct sets of tests behind the patterns, not as the patterns themselves. A test that changed four things sits in four groups, but it is one piece of evidence seen four times, and counting it four times would over-correct and hide real findings.

It is the same discipline the statistics apply to a single test with several variants, carried up to a whole programme. That is why Learnings can look at many angles at once without turning a large programme into a source of false confidence.