Pipelines: Running Experiments in Stages

A pipeline is an ordered run of stages, and a stage is a set of experiments that run at the same time. It is a sequence of parallel groups rather than a general graph, because order and combination is what a testing programme actually needs to express.

What a stage is

Each stage holds one or more experiments that run together. Every stage runs at full traffic in turn. Traffic is not divided between the stages, so a three stage pipeline is not a third of your audience per stage, it is your whole audience three times over.

That is the main thing building a pipeline buys you. Two experiments that would collide go into different stages instead of being made mutually exclusive at runtime, which would split the audience between them and cost both of them sample.

How a stage advances

Each stage advances on a rule you pick when you build it:

• All concluded: the stage ends when every experiment in it has concluded. This is the default, and the only rule that cannot cut an experiment short.
• First significant result: the stage ends as soon as one experiment reaches a conclusive result. The rest of the stage stops with it.
• Manual: the stage waits until a person advances it.

The two automation switches

Two separate switches govern how much the engine may do on its own. They are separate because auto stopping the wrong experiment and auto deploying a losing variant are not comparable mistakes, and you may well want one without the other.

Both default to off. With automation off the engine still evaluates the board on every tick and logs what it would have done. That dry run is the same function with the same inputs as the live path rather than a second code path, which is what makes it worth reading before you turn anything on.

When to build one

Concurrency costs nothing in sample size, so most experiments do not need a pipeline and should simply run alongside each other. Reach for one when the order matters: when two experiments touch the same element or the same audience and collision detection has flagged them, or when the second test only makes sense once the first has an answer.

Notes

• Stages are the exclusion mechanism: if two experiments must not overlap, put them in different stages rather than trying to keep them apart with targeting.
• The dry run is honest: it runs the real evaluation and records the decision instead of acting on it.
• Automation still respects the trust gates: a stage cannot advance on a result the platform does not trust.