Variance Reduction (CUPED)

Variance reduction is a pre-processing step that shrinks the noise in your experiment results before A vs B runs the statistical engine. On revenue, pageview, and engagement metrics it typically cuts the traffic needed to reach significance by 30–50% — faster tests, tighter intervals, the same answer.

This applies to both routes you can run an A/B test in A vs B: experiments built in the Web Experimentation builder, and feature-flag rules of type A/B Test driven by the avsb-js / avsb-node SDKs. The same Auto/Off controls, the same CUPED math, and the same adjustment label appear in both the experiment Results page and the Flag Rule Results modal.

What CUPED actually does

Most of the noise in an experiment comes from users just being different from each other — some buy a lot, some never buy, some browse for an hour, some leave in 30 seconds. CUPED uses each visitor's pre-experiment behaviour as a baseline and subtracts that baseline from the value you observe during the experiment. The remaining signal is what your variation actually changed, with most of the user-to-user noise removed.

The math is a single linear adjustment per visitor:

adjusted = during − θ × (pre − mean_pre)

Here θ (theta) is the optimal weight for the pre-period value, derived from the covariance between pre and during values across all your visitors. A vs B computes θ automatically from the data when the metric is eligible.

Auto vs Off

Variance reduction is controlled by a single setting with two options. New projects default to Auto.

Auto (recommended)

A vs B applies CUPED only when it will actually help. For each metric on the experiment, A vs B checks the pre-experiment data and decides whether the adjustment is worth running. When the data isn't there, A vs B silently skips the adjustment and reports the raw numbers — nothing is distorted.

Auto looks at four things before applying CUPED:

• Metric type. Continuous and count metrics (pageviews, custom numeric events, revenue) are eligible. Binary conversion-rate metrics (clicks) are out of scope for V1.
• Pre-experiment history. The project needs at least 14 days of tracking data before the experiment started.
• Sample size. At least 500 visitors with a pre-experiment value for the metric.
• Correlation. The pre-period values must be at least weakly correlated with the during-period values for the adjustment to do anything useful.

Off

A vs B always reports the raw observed numbers with no adjustment. Use this when you want a baseline you can reproduce by hand — for auditing, debugging, or in regulated workflows where every step of the calculation must be visible.

What you'll see on the Results page

Every Results page shows a small label near the top summarising the analysis configuration. The label combines the stats engine and the variance-reduction decision into a single line, formatted as <Engine> · <variance-reduction state>:

• The engine is Bayesian, Frequentist, or Sequential depending on what the experiment was launched with.
• The variance-reduction state is one of:
  • Off— variance reduction is turned off; numbers are raw.
  • Auto (CUPED applied)— CUPED ran on every metric; numbers are variance-reduced.
  • Auto (no adjustment — reason)— CUPED was eligible but the data did not support it. The reason is shown in plain English.
  • Auto (CUPED applied to some metrics)— mixed: CUPED ran on some metrics and was skipped on others. Per-metric tooltips explain each case.

Examples: Bayesian · Off, Frequentist · Auto (CUPED applied), Sequential · Auto (no adjustment — pre-period correlation too low).

Where to configure variance reduction

Variance reduction can be configured in three places:

• Project level— set the project-wide default in Project Settings → Analysis. Every new experiment and every new feature-flag A/B test rule in the project picks up that default.
• Experiment level— override the project default for a single experiment on the Analysis step before launching. Once the experiment is running, the choice is locked — mixing analyses mid-flight invalidates the result.
• Flag rule level— override the project default on a single feature-flag A/B test rule from the rule's Analysis section. Once the rule has ever been enabled, the choice is locked for the same reason.

What's not in V1

• Binary conversion-rate metrics. CUPED is applied to continuous and count metrics in V1. Click-conversion metrics decline with reason this metric type is not yet supported.
• Multi-covariate adjustment (ANCOVA). V1 uses a single covariate (the pre-experiment value of the same metric).
• Custom pre-period windows. V1 uses a fixed 30-day pre-experiment window.
• Per-metric variance-reduction toggles. The setting applies experiment-wide; Auto then makes a per-metric decision automatically.