Experimentation

Anti-flicker rendering

Nobody watches the original page turn into the test. And no failure is allowed to leave the page hidden.

No flash of originalFour fail-safesErrors reported, not thrown

When a variant changes something above the fold, the visitor must not see the original first. The runtime hides the page while changes are applied and reveals it immediately afterwards, so the first thing rendered is the version being tested.

The interesting engineering is in the second half. Hiding a customer page is the most dangerous thing this software does, so the reveal is called from four independent code paths: after changes are applied, from the inner error handler, from the outer error handler, and in the runtime served to sites with no experiments running at all.

That redundancy is deliberate and is treated as an invariant rather than as duplication to be tidied into one place. If one path fails, another still runs. The failure being defended against is a customer site staying blank for their own visitors.

Errors in experiment code are captured and reported afterwards rather than thrown, for the same reason: a broken variant should cost you the test, never the page.

Better experiments, better conversions

Test on all visitors with the world’s lightest script and make confident decisions powered by real-time reporting.