Workflow A/B Testing

An A/B test is a weighted split in the workflow graph. Each subscriber run is assigned to one route, and the workflow dashboard compares conversion rates from your global success goals.

What A/B testing does

• Assigns each run to exactly one weighted branch
• Keeps the assignment stable if that run retries the split step
• Lets each branch contain different emails, waits, actions, or subgraphs
• Attributes success when the run records a configured success goal

It does not create email variants automatically, calculate statistical significance, select a winner, or move future traffic to the leading branch.

Add an A/B test

1. Click the plus control where the experiment should begin.
2. Choose A/B test.
3. Select the split node.
4. Name each variant by what actually differs.
5. Set percentages that total exactly 100%.
6. Replace each branch placeholder with its first real step.

For two variants, the slider keeps both percentages complementary. Multi-variant tests expose a percentage for each route; the total indicator must reach 100% before publishing.

Build useful variants

Change one meaningful variable at a time when you want an interpretable result:

Each path needs its own email step when email content differs. Do not point two branches at one email and expect separate content.

Define success before publishing

A/B results use top-level Success goals, not merely reaching the end of a branch. Open Settings > Goals and define the same conversion signal for every variant.

Good success signals include:

• Payment for a specific product
• Customer tag added
• Booking field populated
• Minimum lifetime revenue reached
• A downstream workflow goal reached

Use Complete workflow only when conversion should stop all remaining nurture steps. Otherwise Lumail records the conversion and lets the run continue.

Read Goals and Success Goals for event and filter semantics.

How assignment works

Lumail derives a stable percentage from the run ID and split-step ID. A retry of the same run remains in the same variant. A later repeat enrollment creates a new run and may receive a different variant.

Changing weights affects runs that reach the newly published split version. Existing runs remain pinned to the version and definition with which they enrolled.

Read the results

The workflow dashboard shows assigned runs, successful runs, and conversion rate for every branch.

Conversion rate is:

successful runs / assigned runs × 100

Only non-dry runs count. A run is successful when it has a recorded success goal. If several success goals could match, the first matching goal in workflow order is recorded.

Interpret results carefully

• Wait until each variant has enough assignments for the difference to be meaningful.
• Compare the same time window and conversion definition.
• Account for delayed conversions before declaring a winner.
• Do not compare a newly edited variant against historical traffic from an older version without noting the version change.
• Check deliverability and click behavior when a conversion difference may be caused by email delivery rather than copy.

Lumail displays observed rates; it does not currently claim statistical significance.

Common mistakes

Related documentation

• Branching
• Goals and Success Goals
• Publishing and Versions
• Test Runs and Results