Experiment Debriefs

When a test ends, the numbers tell you what happened. They do not tell you why, what you expected, or what you will do next. A debrief is the short note that captures those things, saved with the test, so you and your team can read it months later.

It takes about a minute. Every question is optional, and ending a test never waits for it.

Where you find it

• The Debrief tab. Every test that has ended, or that you have deployed a winner from, has a Debrief tab next to Settings, Brief and Statistics. Tests that are still running do not have one.
• The prompt when you end a test. If you end a test yourself from its page, Pertento offers to walk you through the debrief straight away. You can skip it and come back any time.

Tests that end on their own (a scheduled stop, or a stop in a pipeline) do not show the prompt, because nobody is on the page. Open the Debrief tab on those whenever you like.

The prompt, step by step

The prompt has three screens. Your answers are saved as you move on, so leaving partway keeps the earlier screens.

1. The result. You see how each variant performed (conversion rate, how far ahead or behind the original, and whether the difference is significant), the hypothesis you wrote in the Brief tab, and one question: Did that hold? Choose As expected, Partly, The opposite, or No clear result.

2. Pertento's read of the test. Pertento drafts what it can see in the data and asks you to confirm or correct each line:

• Outside the test. A change in your website traffic during the run, if there was one. Pertento compares each full week with the first week, for the devices the test targeted, and flags a shift of 15% or more. This needs a test that ran for at least 14 days.
• Tracking. Days when no visitors were recorded, or a visitor split that drifted from the weights you set.

Press Right if a line is correct, or Not quite if it is not. Your answer decides which outside factors end up on the debrief. You can also tick anything Pertento cannot see, such as a promotion, a season, or a change to your site.

Then choose what you are doing next (Ship the winner, Iterate, Retest, Drop the idea, or Not decided yet), and write what explains the result and what someone else should know.

3. Review. Everything on one page, next to the finished debrief, so you can check it before you save. Press Save debrief and you land on the Debrief tab.

Skipping

Press Skip for now, close the window, or press Escape at any point. Pertento asks you to confirm first, and reminds you that you can fill the debrief in later from the Debrief tab. Answers from screens you already moved past are saved. What is on the current screen is not.

The Debrief tab

The tab shows the finished debrief as one card.

On the left, the facts from your data. A headline written from the test's statistics, then the uplift, confidence, number of visitors, and how many days the test ran, and the hypothesis from your Brief. You never type the headline, so it always matches the numbers.

On the right, what your team said. The lesson comes first, then why you think it happened, the caveats (outside factors), and the next step. A small label at the top of the card, such as held up or inconclusive, is your own answer to "Did that hold?".

Your answers. Underneath the card is a bar that says how many of the five answers are filled in. Press Edit answers to open them. The card above changes as you type. Save debrief keeps your changes, and Cancel throws them away.

Anyone who can open the test can fill in or change its debrief. The card shows who wrote it last and when.

When the headline says "not by enough to call it"

Some tests move, but not by enough to be sure the movement is real. You will see a headline such as Conversions rose 7.6% on mobile with V-1, but not by enough to call it. That is Pertento telling you what it saw, and that the data does not prove it yet. It does not call the result a win, and it does not claim that nothing happened.

If several variants were compared with the original, you may also see "once 2 variants are compared". The more variants you compare, the easier it is to get a lucky result by chance, so Pertento asks more of each one before it calls a win.

"Did not move" appears only when the change was under 1% in either direction.

Why the card and the Statistics tab can show different percentages

The Confidence figure on the card measures how unlikely the difference is to be down to chance. The recommendation on the Statistics tab uses a different measure: the probability that a variant is the best one. For the same test they can read, for example, 93% and 97%. Neither is wrong. If you want more certainty, the usual answer is the one the Statistics tab gives: let the test run longer.

Questions

Do I have to fill it in? No. Every answer is optional, and a test ends the same way whether you do or not.

Can I do it later? Yes. Open the Debrief tab on the test at any time. If you skipped everything, the answers open ready to fill in.

Can I change it afterwards? Yes. Press Edit answers, change what you like and save.

Why does it say "No statistics are stored for this test yet"? The test has no results recorded, for example because it never received traffic. You can still write the debrief. The card simply has no numbers to show.

Will Pertento spot a change to my website during the test? Not yet. It cannot see changes made to your site, so tick "A site release" yourself if one affected the result.

How long can the answers be? Up to 2,000 characters for why and for the lesson, and 1,000 for the note under "Something else". A counter under each box shows what is left.

Who can see it? Everyone with access to the test, in the Debrief tab. It is saved with the test. When someone asks a question in Ask about this experiment, the debrief is sent to our AI provider with the rest of the experiment so the answer can take it into account. See Asking AI about an experiment.

Can I ask AI about the debrief? Yes. Ask about this experiment reads the debrief as your team's own reading of the result, not as a measured result, and says so if the numbers disagree.