Notification Keywords

Notification keywords are A vs B's convention-based gate for experiment-level notifications. Instead of asking every author to configure routing for every experiment, the gate uses a small list of organization-wide keywords: name an experiment so it begins with one of those keywords and it fires notifications; name it anything else and it stays silent.

This keeps QA experiments, internal A/A tests, and exploratory work out of production channels — while the experiments that matter (launches, optimization rollouts) flow through to Slack, Teams, Jira, and webhooks automatically.

Built-in keywords: LIVE and PROD

LIVE and PROD are recognized in every organization automatically — no setup required:

• LIVE — for production-impacting experiments your team wants to broadcast.
• PROD — alternate naming convention for teams that prefer it.

They are always active and can't be turned off, so a production-named experiment can never be accidentally silenced. You can add your own keywords on top (see below); the built-ins keep working alongside them.

How matching works

Matching is a case-insensitive prefix check with a separator. An experiment name matches a keyword when:

1. The name starts with the keyword (any case), and
2. The character immediately after the keyword is a separator — a space, dash (-), underscore (_), or colon (:), and
3. There is at least one more character after the separator.

Examples with the default keywords LIVE and PROD:

• LIVE Checkout redesign — matches (space after keyword).
• live-hero-cta — matches (case-insensitive, dash separator).
• PROD_q3_pricing — matches.
• LIVE:add-to-cart — matches (colon separator).
• Homepage LIVE test — does not match. The keyword has to be at the start of the name.
• LIVE on its own — does not match. The convention requires something after the separator.
• Internal QA copy — does not match. No keyword prefix.

Configuring keywords

Open Organisation Settings → Integrations and the Notification keywords panel sits at the top. Keywords are organisation-wide — they apply to every project under the organisation.

Adding a keyword

Click Add keyword, type the keyword (letters and numbers only, up to 32 characters — no spaces or punctuation), and save. The new keyword takes effect immediately for every event fired afterward.

Editing a keyword

Hover the keyword row and click Edit. The same validation rules apply (letters and numbers only, 32 character maximum). Editing changes which experiment names match — running experiments whose names no longer match will silence on the next event; ones whose names now match will start firing.

Removing a keyword (with impact preview)

Click the trash icon on a keyword row. A confirmation modal shows the impact preview: every running experiment whose notifications will be silenced if you confirm. Review the list, then confirm the removal.

Safety events bypass the gate

Two events ignore the keyword convention and always fire, regardless of the experiment name:

• SRM failed(sample ratio mismatch) — fires when an experiment's traffic split diverges from the configured weights to a statistically significant degree. SRM signals data-quality problems that need immediate attention even on QA experiments.
• Guardrail breached — fires when a traffic-health guardrail trips. Catching this early on any experiment, gated or not, is the safer default.

Both events are flagged as bypassedGate=true in the delivery record so you can filter them in the delivery log.

The Notifications badge

Every experiment list row and detail page header displays a small Notifications badge:

• Notifications: On — the experiment name matches a keyword. Hovering shows which keyword it matched.
• Notifications: Off — the experiment name does not match any keyword. Hovering shows the current keyword list and suggests renaming to enable.

The badge updates in real time as you rename an experiment — no page refresh required.

Troubleshooting

"I renamed the experiment but notifications still aren't firing"

Most common causes:

• Name doesn't start with the keyword. The keyword has to be at the very beginning. Homepage LIVE test does not match — LIVE Homepage test does.
• Missing separator. LIVECheckoutcopydoes not match — there's no space, dash, underscore, or colon after the keyword.
• Nothing after the separator. LIVE - alone does not match.
• Destination disabled or has no route for the event. Check the destination card in the Integrations tab. The badge can read "On" while a specific destination is paused.
• Keyword was just added. Events fired before the keyword existed are not retroactively delivered.

"I'm seeing notifications for a QA experiment — why?"

Two possibilities:

• The name accidentally starts with a keyword. Something like Live-test-thing matches the built-in LIVE. Rename the experiment — LIVE / PROD are built in and can't be removed. For a custom keyword, you can instead remove it from the org config if no one is using it intentionally.
• It was a safety event. SRM and guardrail events always fire. The delivery row will show bypassedGate=true— that's by design.

"We haven't added any custom keywords"

You don't need to. LIVE and PROD are built in, so experiments named like LIVE - Checkout notify out of the box. With no custom keywords added, every other experiment stays silent — except safety events, which always fire. Add custom keywords in the Keywords panel to extend the convention.

Related