Scheduling Experiment Starts and Stops

A schedule starts or stops an experiment without a person present. You set it when you build the test, which is the moment you actually know how long it should run, rather than at the end when the answer is whatever feels right.

The engine ticks every minute.

The schedule states

• Armed: waiting for its start time.
• Held: the start time arrived but starting now would collide with something already running, so the start is held rather than forced. It is re evaluated every tick and fires as soon as the conflict clears.
• Active: the experiment is running and the schedule is waiting for a stop condition.
• Complete: the stop condition was met and the schedule is done.
• Blocked: something unmodelled went wrong. A blocked schedule does not clear itself.
• Cancelled: somebody cancelled it.

Held is not blocked

These are separate states because they need different responses. A held schedule clears itself once whatever it is waiting on goes away. A blocked one never does, and needs a person to look at it. Blocking emits one alert rather than one a minute, because every action has already had its preconditions checked, and repeating a failure every sixty seconds turns one problem into a thousand.

Stopping on time

A time based stop is not gated on anything. Ending an experiment is always safe, and leaving a broken one running past its planned end compounds one problem with another. Whatever else is happening, a scheduled end time ends the experiment.

Stopping on significance

Stopping the moment a test crosses p below 0.05 is textbook optional stopping, and an automated check that peeks every minute is the worst case of it. So the rule is deliberately conservative: an automated stop requires the sequential test and the corrected p value to agree, on the same variant, in the same direction.

A scheduled end date is mandatory alongside this setting. The sequential test is sized for a fixed target lift, so an experiment with a real but smaller effect can leave the two tests permanently disagreeing, in which case nothing fires and the experiment needs its scheduled end to finish at all. Significance may only ever end an experiment early. It is never the only thing that can end one.

The minimum runtime

A minimum runtime is enforced before an automated stop on significance is allowed. The default is fourteen days, which is two full business cycles, and it cannot be configured below seven days at all. Traffic and intent are not the same on a Tuesday as on a Sunday, and a test stopped after four days has measured whichever days those were.

The guard rail gates the significance path only. A scheduled end still ends the experiment whenever it arrives, and a person can always stop one by hand.

The three trust gates

Three conditions mean the reading cannot be believed, and each one blocks an automated stop on significance:

• A sample ratio mismatch: traffic did not split as configured, so the groups are not comparable.
• No traffic for a full day: nothing is being measured.
• Interference: another running experiment is corrupting this reading.

When a gate blocks, the reason is stated in plain words rather than as a status code.

Notes

• Trust gates never block a time stop: they guard the significance path only.
• A collision holds a start, it does not cancel it: the schedule fires when the conflict clears.
• Set the schedule while you are building the test: that is when you know what the answer should be.