/Docs

Your First Experiment

This is a complete walkthrough of creating your first A/B experiment. By the end you will have a live experiment running on your website, splitting your visitors between two versions and tracking which one performs better.

What we are building

For this walkthrough, imagine you want to test whether changing the color of your homepage call-to-action button from grey to green increases clicks. The original (grey button) is called the Control. The new version (green button) is the Variant. Half your visitors will see the control, half will see the variant, and A vs B will track how many people click the button in each group.

Creating the experiment

1

Create the experiment

From your project dashboard, click Experiments in the left sidebar, then click the New Experiment button in the top right corner. A drawer slides in from the right side.

Enter a Name(for example, “Homepage Hero Banner Test”) and an optional Description explaining what you are testing. Click Create Experiment. The experiment is saved as a draft and you land on the experiment builder.
2

Step 1 – Targeting: Choose where the experiment runs

The first step is targeting. This is where you tell A vs B which pages of your website should trigger the experiment.

Click Add rule. A new URL rule appears. Set the match type to Simple match and enter / (a single forward slash). This means the experiment will run on your homepage.

If you wanted the experiment to run on every page, you could use the Substring match type with /. If you wanted only a specific page like /pricing, you would use Exact match.

For now, leave the audience picker on its default Everyone tag — we want all visitors to be eligible. Click Next to continue.
Tip
Simple match checks that the visitor's URL path starts with what you entered. So / matches the homepage, /blog matches any URL under your blog section, and so on.
3

Step 2 – Variations: Define what each group sees

Step 2 is where you set up what each group of visitors will experience.

You will see two variations by default: Control and Variant 1. The control is always your original, unchanged experience. The variant is what you want to test. You can add up to four variations total (the control plus three variants) if you want to test multiple ideas at once.

Click on Variant 1 to open its code editor. In the CSS tab, paste the following to change your button color:
css
1/* Change the CTA button to green */
2.cta-button {
3  background-color: #22c55e;
4  border-color: #16a34a;
5}
The traffic split defaults to 50% control / 50% variant, which is what we want. Click Next.
Info
You have two ways to write variation code:
  • Platform editor— write CSS and JavaScript directly in A vs B's built-in Monaco code editor. Great for quick changes.
  • CLI + Chrome extension — clone the experiment locally with avsb clone, write TypeScript and SCSS in your own IDE with hot-reload support, then push the compiled code back with avsb push. See the Local Development docs for details.
4

Step 3 – Metrics: Choose what you are measuring

Step 3 is where you select the metric that determines which variation wins.

If you have already created a click metric for your CTA button, select it from the list. If you have not created any metrics yet, click Create new metric. Choose Click as the metric type, give it a name like “CTA Button Click”, and enter the CSS selector for your button (for example, .cta-button).

The first metric you select becomes the primary metric — the one that determines statistical significance for your experiment. Click Next.
5

Step 4 – Review: Check everything and publish

The final step shows you a pre-flight checklist. A vs B runs a series of checks to make sure the experiment is ready to go live:
  • Snippet is installed and detected
  • At least one URL targeting rule is set
  • At least two variations exist with valid traffic splits
  • At least one metric is selected
All four checks should show green checkmarks. If any are red, click the Edit button next to that step to go back and fix the issue.

When everything is green, click Publish experiment. A confirmation modal will appear — click Publish to confirm. Your experiment is now live.
6

Watch results come in

After publishing, you will be redirected to the experiment's results page. At first it will show no data — that is normal. Results start appearing as visitors load your website and are bucketed into variations.

You will see the number of visitors in each variation, the conversion rate for each variation, and — once enough data has been collected — a statistical significance indicator showing which variation is winning.
Tip
A/B tests take time. A useful result usually requires a few hundred conversions per variation at minimum. Resist the urge to stop the experiment after only a day or two — let it run until you have enough data to make a confident decision.

What happens next

You have now set up your account, installed the snippet, and launched your first experiment. Here are some good next steps: