Patterns across tests

One test is an anecdote. Patterns are what a programme of tests adds up to. Pertento groups every learning card on a website four ways: by page and change type, by the mechanism the test expected to work through, by theme, and by the element that changed. Each group is a question your tests have been answering, whether anyone set out to ask it or not.

Each group gets one pooled estimate of the lift, with a range. Results are combined by how precise each one is, against a sceptical starting point centred on zero, so one lucky test is pulled towards no effect instead of being read as a rule. Two variants of the same test share a control, so they count as one entry rather than two.

Only the statistical outcome counts. Deploying a variant is a person’s decision, and counting it as a win would make the ideas people liked look like the ideas that worked. An inconclusive test still counts: it moves the range, which is exactly what it should do.

Code computes the patterns afresh from the cards and the statistics. The AI words the featured ones and offers a possible reason, marked as such, and every figure in those sentences is filled in by code when the page is read. Learnings is strictly per website: nothing is pooled across websites or across customers.