Analysis plans & pre-registration

A pre-registered analysis plan is a short declaration written before an experiment starts: which metric is the primary, which metrics are guardrails, which engine and significance threshold the analysis will use, and how big the experiment is meant to get. A vs B locks the plan at launch and records every change made afterwards, so the difference between what was pre-declared and what was decided later is always visible on the results page and in the audit log.

Why it matters

The danger most teams underestimate in experimentation is researcher degrees of freedom: the small post-hoc choices that quietly inflate false-positive rates. Adding a metric after the fact because it's the only one moving. Picking a different segment because it tells a better story. Switching engines because the official one doesn't reach significance. Each of these is reasonable in isolation; together they erode the credibility of every result.

Pre-registration is the cheapest, most effective safeguard against this. Ron Kohavi's Trustworthy Online Controlled Experiments (the standard reference for the field) argues that locking down the analysis plan before data starts arriving is the single most impactful thing a team can do to keep results trustworthy. A vs B makes pre-registration a first-class feature of the experiment lifecycle rather than a side-channel convention.

Declaring a plan

The experiment builder has a dedicated Analysis step (between Metrics and Review). That step contains an Analysis Plan card capturing eight fields — most of them inherited from project defaults so you only have to think about what differs from the norm.

• Primary metric. The single metric the experiment is meant to move. Required.
• Guardrail metrics. Metrics that must not get worse — typically conversion rate, revenue, error rate, or load time. Optional but strongly recommended.
• Secondary metrics.Anything else you want to track but don't want to gate the decision on.
• Stats engine.Bayesian, Frequentist, or Sequential. Defaults to the project's configured engine.
• Significance threshold. Reads as α for Frequentist, the confidence level for Sequential, or the chance-to-beat-control threshold for Bayesian.
• Multiple-comparison correction. Bonferroni, Holm-Bonferroni, Benjamini-Hochberg, or none. Visible only when the engine is Frequentist and there are more than two variations.
• Target sample size. Pre-computed by the calculator from baseline rate, MDE, and power. Editable.
• Target duration. The expected runtime in days. Used for the day-counter in the results header.

See Choosing a stats engine for the engine decision and Early stopping for what the target sample size means in practice.

Sealing

The plan is sealed the moment the experiment launches. From that point on:

• The plan's fields become read-only in the builder.
• The results page surfaces the sealed plan in a dedicated card.
• Each metric on the results page is tagged Pre-registered or Exploratory, depending on whether it was on the plan when sealing happened.
• The official stats engine and the alpha/threshold are locked. The Explore-under and Compare engines views still let you re-render under a different engine, but those views are clearly labelled exploratory and never overwrite the sealed result.

Amendments

Sometimes a sealed plan has to change. A guardrail you forgot to declare turns out to matter; the target sample size needs to grow because traffic is lower than estimated; a new variation gets added. A vs B handles this through an explicit amendment workflow instead of letting fields silently change.

To amend a sealed plan:

1. Open the experiment's Analysis Plan card on the results page.
2. Click Propose amendment. A modal opens with the current value of every field; you change only what you mean to change.
3. Add a short reason — one line is fine. The audit log captures who, when, what changed, and why.
4. Submit. The change is applied immediately, and the amendment becomes visible in the plan card and in the experiment's audit history.

Every amendment is preserved indefinitely. The full chain — original sealed plan plus every amendment with timestamp, actor, before/after values, and reason — is exported as part of the experiment's record when you download it from the audit log or the per-experiment export.

Pre-registered vs exploratory

On the results page, every metric is tagged with one of two pills:

• Pre-registered — the metric was on the sealed plan at launch (or was added through a documented amendment).
• Exploratory— the metric was added after sealing without going through the amendment workflow. Useful for poking around your data, but not appropriate for drawing conclusions you'd defend in a stakeholder review.

The visual distinction is deliberately subtle in the UI — pills, not warnings — but the information is always there for anyone reviewing the result. When in doubt, pre-register.

The project-level toggle

Open Project Settings → Analysis defaults. The Require pre-registration toggle, when on, forces every experiment in the project to complete the analysis-plan card before it can launch. With it off, declaring a plan is opt-in per experiment.

Most teams start with the toggle off, treat pre-registration as a habit on important experiments, and turn it on once the team is fluent with the workflow. Regulated environments — pharma, finance, compliance-sensitive product areas — typically turn it on from day one.

When you'd want this

• Multi-stakeholder reviews.A pre-registered plan eliminates the “but didn't we say we cared about X?” debate after results land.
• Regulated environments. An auditable record of what was decided in advance is non-negotiable for some compliance regimes. The amendment log makes this straightforward.
• Experiments where the team has a strong prior. The temptation to rationalise a non-result is strongest when you really wanted the variation to win. Pre- registering keeps you honest with yourself.
• Reusable templates. Once a team agrees on its standard analysis plan for a given surface (checkout, signup, onboarding), pre-registration becomes a checklist rather than a debate.