/Docs

Holdouts

A holdout reserves a fixed percentage of your users as a global control group — held back from every experiment in the holdout — so you can measure the cumulative impact of your entire experimentation programme rather than individual experiments in isolation.

What it is

A holdout is a named configuration object that references one or more flags. When a user is bucketed into the holdout (using the same deterministic MurmurHash3 algorithm used by regular rules), every flag they evaluate that participates in the holdout returns the configured default variation — typically the control — regardless of which rules would otherwise match. The SDK returns source: 'holdout' so you can see exactly how many decisions were held out.

Exposure events fire with ruleType: 'holdout' so your analytics pipeline can exclude held-out cohorts from per-experiment analysis while keeping them in holdout-level reports.

When to use it

Holdouts are most valuable in high-velocity product teams where many experiments run concurrently. Typical use cases:

  • Measuring programme ROI: compare the held-out group (no experiments) to everyone else to see the net effect of your experimentation pipeline on a primary business metric like retention or revenue.
  • Long-running baseline:keep a 2–5% holdout for months at a time so you always have a clean comparison point when leadership asks “what would our metrics look like with no product changes?”
  • Interaction detection: if two experiments ship simultaneously and you suspect interaction effects, the holdout group — exposed to neither — acts as a neutral baseline.
Warning
Holdout groups reduce the traffic available for experiments. A 5% holdout means every experiment effectively operates on 95% of your users. Plan holdout size accordingly, especially for experiments that need statistical power.

How it works

The datafile carries a top-level holdouts array. Each entry lists the participating flag IDs, a traffic allocation (0–1), a hash attribute ('user.key' by default), and a per-flag map of the default variation to serve held-out users:

ts
1// Excerpt from a v2 datafile — illustrative
2{
3  holdouts: [
4    {
5      id: "hld_abc",
6      key: "global-holdout-2024",
7      trafficAllocation: 0.05,       // 5% of users
8      hashAttribute: "user.key",
9      flagIds: ["checkout-v2", "pricing-experiment", "nav-redesign"],
10      defaultVariationByFlag: {
11        "checkout-v2": "control",
12        "pricing-experiment": "control",
13        "nav-redesign": "control",
14      },
15    },
16  ],
17}

Before evaluating any flag's rules, the SDK checks every holdout the flag participates in. If the context hashes into the holdout bucket, the flag returns the holdout's default variation and evaluation short-circuits. The hash uses the same consistent bucketing as regular rules, so users never change holdout membership unless the trafficAllocation shrinks.

Because all flags in a holdout share the same hashAttribute and allocation, the same 5% of users are held out across all participating flags. This is the mutual exclusion guarantee: held-out users never enter any experiment in the set.

Per-SDK usage

Holdouts are evaluated transparently — no SDK call changes. The SDK reads the holdout configuration from the datafile and applies it during every getFlag call. You can inspect whether a decision was held out by checking flag.source:

@avsbhq/browser
ts
1import { AvsbClient } from '@avsbhq/browser'
2

3const client = new AvsbClient({ sdkKey: 'sdk_production_abc123' })
4await client.onReady()
5

6const flag = client.getFlag('checkout-v2', false, { includeReasons: true })
7

8if (flag.source === 'holdout') {
9  // This user is in the holdout — they see the control variation
10  analytics.track('holdout_exposure', { flagKey: 'checkout-v2' })
11}
@avsbhq/node
ts
1import { AvsbServer } from '@avsbhq/node'
2import { DecideOption } from '@avsbhq/core'
3

4const server = new AvsbServer({ sdkKey: process.env.AVSB_SDK_KEY })
5await server.onReady()
6

7const flag = server.getFlag('checkout-v2', false, context, {
8  decideOptions: [DecideOption.INCLUDE_REASONS],
9})
10

11// flag.source === 'holdout' | 'rule' | 'default' | ...
12// flag.reasons contains the holdout key when source is 'holdout'
avsb-python
python
1from avsb import AvsbServer
2from avsb.core.types import DecideOption
3

4server = AvsbServer(sdk_key=os.environ["AVSB_SDK_KEY"])
5server.wait_for_ready()
6

7flag = server.get_flag(
8    "checkout-v2",
9    False,
10    context,
11    decide_options=[DecideOption.INCLUDE_REASONS],
12)
13

14if flag.source == "holdout":
15    logger.info("User is in holdout", extra={"reasons": flag.reasons})
avsb-go
go
1client, _ := avsb.NewServer(avsb.Options{SDKKey: os.Getenv("AVSB_SDK_KEY")})
2client.WaitForReady(context.Background())
3

4flag := client.GetFlagWithOptions("checkout-v2", false, ctx, avsb.DecideOptions{
5    IncludeReasons: true,
6})
7

8if flag.Source == avsb.SourceHoldout {
9    log.Printf("holdout decision: reasons=%v", flag.Reasons)
10}