Feature Flag Guardrails

A guardrail answers one question for each step of a rollout plan: is On, on the flag's primary goal, worse than Off by more than the margin? It checks every five minutes that On does no worse than Off on the primary goal, and the plan only moves on once it is satisfied.

A guardrail step stops a large drop. A small one can still pass. That is the honest summary, and the rest of this guide explains why.

Setting it up

Three things decide what the guardrail does.

• Primary goal, on the Setup tab: "The event the guardrail judges On and Off on." Choose it from the website's events. It is fixed once the flag is live, so set it on the draft. See Setting Experiment Goals from Analytics Events for how events reach Pertento.
• Guardrail margin %, on the Rollout plan card: how much worse On may convert before the step is a breach. The default is 2%, which means On must convert at least 98% as well as Off. It can be anything from 0.01% to 50%. A smaller margin is stricter.
• Moves on after on each step: the guardrail runs only on steps set to Guardrail or Time and guardrail. A step set to Time moves on without a check. See Rolling Out a Feature Flag.

The guardrail needs the website's visitor ID. Without it the guardrail cannot count visitors and stays at Collecting, and a plan with a guardrail step is refused: "The guardrail needs the website visitor ID. Turn it on in the website settings to use it." See Anonymous Visitor ID.

A guardrail step at 0%, or at 100% with no holdback, is refused when the plan is saved. Nobody would be left on one side to compare.

Who is compared

The guardrail compares only the visitors who were new to the flag during the step, On against Off. Visitors who joined at an earlier step are not counted again, so each step is judged on its own evidence. Holdback visitors count as Off. Ring members are not part of the comparison.

The verdicts

The Guardrail card on the Rollout tab shows a chip and a line of explanation for the current step:

• Collecting: "Not enough data yet. It needs 200 visitors a side and 1,000 Off conversions." The step stays where it is until there is enough to judge.
• Clear: "No harm detected, so the step will move on." The guardrail has not proven On safe, but it has seen no sign of harm, and the plan continues.
• Pass: "On is proven no worse than Off, within the margin."
• Breach: "On converts worse than Off, beyond the margin." The plan pauses at its current share.

The card also shows the step's On and Off visitors and conversions, which is the only place a flag shows counts. A flag has no statistics page and no winner.

When the guardrail pauses a plan

A breach pauses the plan where it is. The rollout is not lowered: visitors who have On keep it until you decide. The flag's status card explains it once, as Paused by the guardrail: "On did worse than Off on the primary goal. Check the Guardrail card on the Rollout tab, then resume the plan or turn off." In the Library, the Next step column reads "Guardrail breach". A notification goes out if "Flag rollout paused by its guardrail" is switched on in your notification settings. See Notifications in Pertento.

Then it is your call:

• Turn off sends everyone to Off at once. It is the safe choice when you believe the drop is real.
• Resume continues the plan from the step it paused on, if you have looked at the numbers and decided the drop is not a problem.
• Fix the change and keep going. For an editor flag, change On in the visual editor. For a code flag, ship a fix, then resume.

What it catches, and what it does not

The guardrail is built to stop a large drop early, not to prove a small one. Measured on a 5% conversion rate with the default settings, a 20% drop is caught before the step moves on about 94% of the time, a 10% drop about 21% of the time, and a 5% drop about 3% of the time. A change that makes no real difference moves on.

For a stricter check, give the step more data: a long Stay at least with Time and guardrail keeps the step open while the guardrail keeps counting. A smaller margin makes a breach more likely to be called, but it does not add evidence on its own.

The statistics are sequential-safe. Checking every five minutes for weeks adds no false alarms, so a long step is not a riskier step.

Notifications

Only what the scheduler or the guardrail did is notified. Changes you or a colleague made by hand are on the Activity tab only. The Feature flags group in Settings, Notifications, covers a scheduled publish, a plan moving to its next step, a plan reaching its last step, a scheduled turn off, a guardrail breach and the 30-day reminder on an editor flag. Each comes as email and in-app, and each can be switched off.