How Learnings Weighs the Evidence

Learnings follows one rule throughout: code computes every number. Lifts, ranges and evidence levels come from Pertento's statistics, and code decides which patterns, surprises and opportunities exist. The AI labels the tests, runs its analysis on those figures and words the result. It never produces a number of its own: a sentence containing one is thrown away.

From tests to patterns

Every finished test gets a learning that labels what each variant changed: its change types, page type, element and theme. Learnings groups those labels four ways (page and change type, mechanism, theme and element) and pools each group into one estimate with a range.

• Single results are pulled towards zero on purpose. Results are combined by how precise each one is, starting from a sceptical assumption of no effect, so one lucky test cannot read as a rule.
• Two variants of one test count once. They share a control, so they enter as one result.
• Inconclusive tests count. An inconclusive result still moves the range.
• Only the statistical result counts. Deploying a variant does not make it a win here, so the ideas people liked do not look like the ideas that worked.

A test that changed several things appears in several groups, so the groups do not add up to the number of tests.

Ranges are adjusted for how many ways the tests are sliced

Learnings checks many patterns at once, and the more ways your tests are sliced, the more likely it is that one slice looks strong by luck. Every range on the page is widened for that, using the Benjamini-Hochberg procedure with a false coverage rate of 5%. The adjustment counts distinct sets of tests rather than patterns: a test that sits in four groups is one piece of evidence, not four.

When two patterns rest on exactly the same tests, the page says so, for example "the same 2 tests also make up Layout on checkout", and features only one of them.

The five evidence levels

Each pattern gets one level, read from its adjusted range. The first that fits is the one shown.

• Mixed: tests of the same thing disagree, with one variant confidently up and another confidently down. Mixed results are never averaged into "no effect".
• Consistent: the range excludes zero, from 3 or more tests.
• Early signal: the range excludes zero, from 1 or 2 tests.
• Small effect: the range includes zero and lies within 2% either side. Whatever this kind of change does, it does not do much.
• Not enough evidence: anything else.

The wording follows the range as well as the level, so an early signal below zero reads as a change that hurts.

Surprises

Learnings flags a result as a surprise when:

• a test rated in the top quarter of the website's PXL scores lost with confidence;
• a test won with confidence but had no expected mechanism, or was rated in the bottom quarter;
• the team's debrief expected the opposite of what happened.

The PXL comparisons need at least four scored tests with a spread of scores. A surprise raises an Explain a surprise opportunity. See Prioritising Experiments with PXL Scores.

What leaves a test out

• A broken traffic split, or no data. Both are measured, and nothing overrides them. See When Traffic Does Not Split as Configured.
• A debrief that reported a tracking issue or a site release. The test stays out until someone who can create experiments chooses Tracking is fixed, include it on its debrief page. That sets aside what the debrief reported. It cannot override a broken split or missing data.

Any other outside factor reported in the debrief, such as a campaign or a season, keeps the test in but makes it count for less. It counts for less once, however many factors are ticked.

Too few visitors, a run under a week, and an overlap with another test are shown as notes on the learning. They do not leave the test out.

What the debrief adds

The debrief is optional, and everything works without it. When there is one:

• The team's lesson and explanation are given to the AI as context and shown beside its learning as "The team's read". They never change a number, and neither overwrites the other.
• Outside factors become notes on the learning, and can leave the test out or make it count for less, as above.
• The team's expectation is never shown to the AI and never weighed. It is only used to flag a surprise.
• The next step is shown, and not given to the AI.