Creating an Experiment

Creating an experiment in A vs B takes you through a five-step guided builder. Each step focuses on one aspect of the experiment — where it runs, what each group sees, how success is measured, how the data will be analyzed, and a final review before going live.

Before you start

Before creating an experiment, make sure:

The A vs B snippet is installed and verified on your website. Without the snippet, experiments cannot run.
You have at least one metric created for this project. You will need to select a metric in Step 3. You can create one during the experiment flow, but it is easier to set them up in advance from the Metrics section.
You have a clear hypothesis: what are you changing, why do you think it will improve things, and how will you measure success?

Starting the builder

Go to the Experiments page

In the left sidebar, click Experiments. This opens the experiments list for your current project.

Click 'New Experiment'

Click the New Experiment button in the top right corner of the experiments list. A modal will appear asking for an experiment name.

Enter an experiment name

Give your experiment a descriptive name that explains what you are testing and where. Good names make it easy to identify experiments later. Examples:

“Homepage hero headline — formal vs casual”
“Pricing page CTA button color — grey vs green”
“Checkout flow — single-page vs multi-step”

Click Create. You are taken to the experiment builder.

The five-step builder

The builder has five steps. You must complete them in order — you cannot jump ahead to Step 3 without completing Step 2. You can always go back to a previous step to make changes.

Picking an experiment type

The first thing the New Experiment drawer asks is what kind of experiment you want to run. There are three options:

Visual — the most common choice. Make changes directly on your page using the point-and-click visual editor. No code required.
Custom Code — write JavaScript and CSS for each variation by hand. Use this when you need fine-grained control over behaviour that the visual editor cannot express.
Split URL — redirect a portion of qualifying traffic from one page to a separate destination URL. Use this when the change is too large for in-place editing, or when you want to compare two fully independent pages. See Split URL Experiments for the full guide.

All three types are chosen in a single step. The choice is locked once the experiment is created — pick the right one up front, or delete and recreate if you change your mind.

Step 1 — Targeting

Define which pages this experiment runs on and which visitors are eligible. You set URL targeting rules and optionally select audience segments. See Targeting for the full guide to this step.

Step 2 — Variations

Set up your control and variant(s). Write the CSS and JavaScript that define what each variant looks like and does. Configure the traffic split between variations. See Variations for the full guide.

Step 3 — Metrics

Choose which metrics determine the winner. The first metric you select becomes the primary metric — the one that drives statistical significance. You can add more metrics to monitor as secondary goals. See Metrics for the full guide.

Step 4 — Analysis

Choose the stats engine (Bayesian, Frequentist, or Sequential), the confidence level, the variance-reduction mode, and any per-experiment analysis overrides. If pre-registration is enabled for the project, seal an analysis plan here before launch. See Choosing a stats engine and Analysis plans for the full background on these decisions.

Step 5 — Review & Publish

A vs B runs pre-flight checks to verify the experiment is configured correctly. Once everything passes, you can publish the experiment and it goes live immediately. See Review & Publish for the full guide.

You can save a draft at any point and come back to finish it later. Experiments stay in Draft status until you explicitly publish them. See Experiment Statuses for details on the experiment lifecycle.

After creating

Once your experiment is published, it begins running immediately. You can monitor its progress on the experiment's results page. Results start accumulating as visitors load your website and are bucketed into variations.