/Docs

Experiments

Experiments are at the core of A vs B. Each experiment compares two or more versions of a page or user flow against a measurable goal. This section covers everything from creating your first experiment to managing experiments that are already live.

How experiments work

An experiment divides your visitors into groups. Each group sees a different version of your page (a "variation"). One group always sees the original, unchanged page — this is the Control. The other groups see modified versions — these are the Variants. A vs B tracks a metric for each group and uses statistical analysis to determine which version performs better.

The five-step builder

Creating an experiment in A vs B uses a guided five-step builder:

  1. Targeting — Choose which pages the experiment runs on and which audience segments are eligible.
  2. Variations — Define the control and variant(s), write their CSS and JS, and set traffic splits.
  3. Metrics — Select the goals you are measuring.
  4. Analysis — Choose the stats engine, confidence level, variance-reduction mode, and (optionally) seal an analysis plan that pre-registers the primary metric and guardrails.
  5. Review — Pre-flight checks and the publish button.

Pages in this section

Creating an ExperimentStart a new experiment and understand the four-step builder flow.
TargetingURL rules and audience selection — control exactly which visitors see your experiment.
VariationsSet up control and variants, write CSS/JS, and configure traffic splits.
MetricsSelect the metrics that determine which variation wins.
Review & PublishPre-flight checks and publishing your experiment to production.
Managing Running ExperimentsPause, stop, edit, and manage experiments that are already live.
Experiment StatusesDraft, Running, Paused, Completed, and Archived — what each status means.
Frequentist EngineHow p-values, confidence intervals, and significance work in A vs B — and when to pick Frequentist.
Sequential EngineAlways-valid inference — peek as often as you like, stop the moment the evidence is in. Slightly wider intervals as the trade.
Choosing a Stats EngineA plain-English comparison of Bayesian, Frequentist, and Sequential, and when each is the right call.
Comparing enginesRe-render any experiment's results under a different stats engine without changing the official engine of record.
Analysis plans & pre-registrationLock the primary metric, guardrails, engine, and significance threshold before launch. Every later change goes through a documented amendment.
Early stopping & peek protectionWhy stopping a Frequentist experiment early inflates the false-positive rate, and how A vs B protects you with a banner, a modal, and an audit stamp.