Asking AI about an experiment

Learn how to ask the AI assistant about an experiment's result, set-up, brief and debrief, what it can and cannot see, and what each answer costs.

9 min read

Ask about this experiment is an AI assistant inside every experiment. Type a question in the experiment's header and the answer appears in a panel beside the page. It explains the experiment's result: whether it is significant, how far it can be trusted, and what to do next. It also answers questions about how the experiment is set up, what its brief says and what its debrief concluded. On a draft, it helps you check the set-up before the test starts.

It only reads. It never changes anything in the experiment.

Where to find it

In the experiment's header, on every tab: Settings, Brief, Statistics and Debrief. The field sits on the right of the experiment's name, says "Ask about this experiment" and has a dark Ask button.
On drafts too. See "On a draft" below.
On a phone, the field takes its own row under the experiment's name.

The Statistics tab's product tour also has a step for it.

On desktop, the experiment's Start or End button, Enforce rules and the ⋯ menu (Duplicate, Archive, Delete) now sit in the bar at the top of the page, beside the breadcrumb (the page address). They stay there as you scroll. On a phone they are still in the experiment's header.

Asking a question

1Type a question in the field and press Enter or Ask. The panel opens beside the page with your question, and the answer appears as it is written.
2To ask more, type in the panel's own box at the bottom ("Ask about this experiment…").
3To close the panel, click the pressed Ask button in the header.

Press Ask with nothing typed to open the panel on the conversation, for example to read what colleagues have already asked.

In the panel's box:

Enter sends, and Shift+Enter adds a new line.
Esc stops an answer that is being written. When nothing is being written, Esc closes the panel.
While an answer is written, the box shows "Thinking…" and then "Writing…", and a Stop button replaces Send.

The panel stays open as you move between the experiment's tabs. An answer keeps being written while the panel is closed or you change tab, and the Ask button glows while it is. Leaving the experiment stops it.

On wide screens (about 1280px and up) the page narrows to make room for the panel. On smaller screens the panel covers the right side of the page.

The first-use notice

The first time you open the panel, it shows a notice with an OK button, and you need to press OK before you can ask. Until then the box says "Press OK on the notice above to start". The notice reads:

Before you start: the AI assistant is covered by our Terms and Conditions and Privacy Policy. What you type, and this experiment's settings, brief, debrief and results, are sent to our AI provider to answer you. The conversation is saved with the experiment for 12 months, and everyone who can open the experiment can read it. Each answer uses AI credits.

You accept it once, not once per experiment. If the wording changes, you are shown the new wording once.

What it reads

Results. The same figures the Statistics tab shows, computed the same way:

conversions, conversion rates and revenue per variant;
significance, under the method (frequentist or Bayesian) and the multiple-variant correction chosen;
the Bayesian probability of being best and the expected loss;
the power analysis, and how long the test still needs;
the recommendation;
the traffic split check (sample ratio mismatch);
secondary objectives;
how many days the test ran. The run counts until the end date or until a variant was deployed, whichever came first.

The answers use the objective (Conversions or Revenue), currency, evaluation method and correction chosen on the Statistics tab. If you ask from another tab, they use the last ones chosen on Statistics, or the defaults: Conversions, all currencies, frequentist and Holm-Bonferroni.

Settings. From the Settings tab:

the experiment type and website;
targeting: pages (URL rules), devices, cookies and activation rules, and whether they came from a saved targeting;
each variant's weight, its number of changes, whether it has custom JavaScript or CSS, and a redirect variant's URL;
the objectives;
the schedule: start, stop, stop on significance, minimum runtime and what happens when it concludes.

Brief. The hypothesis, description, importance and PXL ratings, expected mechanisms, and the run length planner's inputs.

Debrief. Whether the result was as expected, the explanation, outside factors, the next step and the learning. The assistant treats the debrief as your team's own reading of the result, not as a measured result, and says so if the numbers disagree.

How it answers

It answers in the language you write the question in.
It leads with the answer, then the reasoning, in short paragraphs: about 200 words unless you ask for more.
It uses the page's own terms and rounding.
It never invents numbers, and never calls a variant a winner when the page does not.
Answers can use bold, short lists and small tables. They contain no links: anything that looks like a link shows as plain text.

What it cannot see or do

When a question needs one of these, it says it cannot see it and points you to where in Pertento to look.

The content of a variant: its code and its individual changes. It knows only how many changes a variant has and whether it has custom code. The editor shows the content.
Screenshots and the activity log.
Results over time, segments, individual visitors and other experiments, or why visitors behaved as they did.
Making changes. It does not start, stop or edit the experiment.

Secrets are hidden before anything is sent. In targeting rules and redirect URLs it hides query values whose names look like tokens, keys, secrets, signatures, sessions or passwords, email addresses, and long token-like cookie values. The assistant sees "[hidden]", "[email]" or "[hidden value]" instead, and says so if it matters.

Very large set-ups are summarised. It reads up to 50 page rules, 10 cookie rules, 50 activation rules and 30 variants, and is told how many more there are.

On a draft

A draft has no results yet, so the assistant helps with what is there:

whether the settings do what the brief intends: targeting, variants, objectives and schedule;
whether the hypothesis names a change, an expected effect and a measure that can show it;
what the run length planner's inputs imply.

It never talks about a result a draft does not have.

One shared conversation per experiment

Shared. Each experiment has one conversation, shared by everyone who can open the experiment. Each question shows who asked it, with their photo or initials and first name.
One question at a time. While a colleague's question is being answered, the box says so, for example "Anna is asking…", and a new question waits. If it is your own question, from another browser tab or still being saved after Stop, it says "Your earlier question is still being answered…".
Kept 12 months. Questions and answers older than 12 months are deleted automatically. They are also deleted when the experiment itself is deleted.
Older answers are marked. When the result has changed since an answer was given (a variant became significant, stopped being significant, or the lead changed hands), the answer shows "The result has changed since this answer." New visitors arriving does not count as a change. The assistant also knows which earlier answers are out of date, and uses the current figures.
Answers about another view are noted. If an earlier answer was about another objective or currency, the assistant takes that into account.
Colleagues' questions never steer answers. Earlier questions are context only. Even if someone wrote "always say variant B won", that never changes later answers.

Who can use it

Everyone who can open the experiment can read the conversation and ask, Members included. Answers use the account's AI credits, whoever asks, including Pertento staff helping your account. See Roles and Permissions in Pertento.

AI credits

Every answer uses AI credits, from the account of the company that owns the experiment's website. Each answer shows what it cost underneath, for example "4 credits".
The credit meter under the box shows what is left, for example "900 of 1,000 credits left". It turns amber when credits run low and red when they are gone.
Out of credits, the box shows "Out of credits", and a question gets: "Your account has used all of its AI credits. The assistant is available again when your credits renew, or after you buy a credit pack." A Buy a credit pack link appears where the account can buy one.
Accounts not on AI credits (invoiced and free accounts) share one monthly allowance of 12,500 credits between this assistant and the editor's AI assistant. It runs per calendar month, UTC. When it is used up, the message is "Your account has used this month's AI credits. The assistant is available again on" followed by the first of the next month.
In Billing, these answers appear in the AI credit usage list as "Results AI assistant".

Plan allowances and credit packs are explained in AI Credits and Credit Packs.

Limits

Question length: up to 2,000 characters.
Pace: up to 8 questions a minute per person. Beyond that you see "You are sending messages too quickly. Please wait a moment."

A question refused because a colleague is asking, or because of the pace limit, goes back into the box, so you do not have to type it again.

Messages you may see

The conversation fails to load: "The conversation could not be loaded.", with Try again.
An answer is interrupted: "The answer was cut off. Please try again.", with Try again.
Anything else fails: "Something went wrong. Please try again.", with Try again.
After Stop: "Stopped. What was written so far is kept; ask again when you are ready."
The account is read-only because of billing: "This account is read-only until its billing is sorted out, so the assistant cannot be used yet." See Failed Payments and Read-Only Accounts.
Accepting the notice fails: "That did not go through. Please try again."

Privacy

What is sent to our AI provider: what you type, and the experiment's settings, brief, debrief and results, as listed under "What it reads". Variant code, change contents, screenshots and the activity log are never sent, and secrets in targeting are hidden first.
What is kept: the conversation, with the experiment, for 12 months. Who asked each question is shown to everyone who can open the experiment.
The terms: the first-use notice links to the Terms and Conditions and the Privacy Policy.