Analysis Defaults

The Analysis tab in Project Settings controls how A vs B analyses every A/B test in this project — whether that test runs as a Web Experimentation experiment or as a feature-flag rule of type A/B Test. There are two independent settings — Stats Engine and Variance Reduction. Either can be overridden on an individual experiment or flag rule before it goes live.

Accessing the Analysis tab

Open your project, click the gear icon in the left sidebar to open Project Settings, then click the Analysis tab. The tab is available in both Web Experimentation and Feature Flag projects.

Stats Engine

The Stats Engine is the statistical method A vs B uses to decide whether a result is real. Three engines are available:

• Bayesian(default) — reports the probability each variation beats Control, along with credible intervals. Easy to read and forgiving of mid-experiment peeks.
• Frequentist— classical p-values and confidence intervals. Declares a winner only when the significance level you set (α) is met. Requires sticking to your pre-declared sample size.
• Sequential— always-valid inference. Peek as often as you like, stop as early as the evidence is in. Slightly wider intervals in exchange for the flexibility.

See Choosing a Stats Engine for a deeper comparison and guidance on when to pick each.

Planning sample size

Use the Sample-Size Calculator (linked from the bottom of the Analysis tab) to plan your experiment before launch. It has three modes:

• Fixed-horizon — required sample size per variation and estimated duration.
• Power calculator — achieved statistical power at a given sample size.
• Duration estimator — how long until the experiment reaches its target.

Each mode is engine-aware: picking Bayesian, Frequentist, or Sequential changes both the inputs and the meaning of the output. See the calculator guide for details on each mode.

Variance Reduction

Variance reduction is a pre-processing step that runs before the engine. It reduces the noise in your results by subtracting each visitor's pre-experiment behaviour from their during-experiment value. There are two options:

• Auto (recommended)— A vs B applies CUPED variance reduction whenever the metric has a meaningful pre-experiment signal. When the data isn't there, A vs B silently skips the adjustment so the numbers match the raw values exactly.
• Off— A vs B always reports the raw observed numbers. Useful for auditing, debugging, or regulated workflows where every step of the calculation must be reproducible by hand.

See Variance Reduction (CUPED)for a deeper explanation of what the adjustment does, when Auto applies it, and what you'll see on the Results page.

Who can change defaults

Only team members with the Edit Project Settings permission can change the analysis defaults. By default this includes the Owner, Admin, and Developer roles.