---
title: Tell whether an A/B variant is ahead by more than chance
description: One function that reads a campaign's step stats and says, for each step with two or more variants, whether the leader's gap is larger than chance would give, plus which step the replies come from.
sidebar:
  label: A/B test verdict
related:
  - /docs/insights/copy
  - /guides/pre-launch-audit
  - /guides/diagnose-campaign
---

Variant A has a 6% reply rate and variant B has 3%. Whether that is a result or noise depends on how many leads each one reached. `variantTest` answers that for every step of a campaign that has two or more variants: it takes the variant with the highest rate, compares it with each of the others by a two-proportion z-test, and returns one of three verdicts: `winner`, `no_difference_yet` or `too_little_data`. `stepFunnel` adds which step the replies come from, so you know how much a winner is worth.

It does not tell you when a test is finished. It has no stopping rule, it makes no correction when a step has three or more variants, and reading it every day until it says `winner` will produce winners that are not real. [Limits](#limits) says what to do about each. It calls no AI model and changes nothing in your vendor tool: turning a variant off is up to you.

> **Preview: not published yet**
>
> `outbound-sdk` is not on npm, and its GitHub repository is private, so there
> is no install command that works today. Instantly and HeyReach are `beta`:
> their reads were run, read-only, against one live workspace each. Smartlead is
> `unverified`: written from its public documentation and tested offline only.
> See [what is verified](/docs/status).

## Before you start

- Node 22 or later.
- The code below runs on the in-memory adapter, so it needs no vendor key. Each sample on this page is a file in the repository that the test suite runs; the output shown is what it printed.
- For real data: an Instantly or Smartlead API key. A read-only key is enough. HeyReach cannot serve this guide, see [what each vendor can serve](#what-each-vendor-can-serve).

## What you get

```text title="Output"
A/B tests of cmp_founders, judged on replied at alpha 0.05

Step 0: winner
  Variant A (0.06) beats every other variant by more than chance at alpha 0.05.
  A: 6.0% (48 of 800 contacted)
  B: 3.0% (24 of 800 contacted)
  p = 0.0038

Step 1: no_difference_yet
  Variant B (0.04) leads variant A (0.0286), but a gap this size could be chance.
  A: 2.9% (20 of 700 contacted)
  B: 4.0% (28 of 700 contacted)
  p = 0.24
  at these rates, about 2498 more sends before the gap would count

Step 2: too_little_data
  Fewer than two variants have a base of 100, so no comparison is made.
  A: 5.0% (3 of 60 contacted), too few to compare
  B: 1.3% (5 of 400 contacted)
  40 more sends before the variants are compared

Step 0 (counted from 0) brings the most replies: 72 of 128. Every step up to step 2 still earns its place.
  step 0: 56.3% of all replies
  step 1: 37.5% of all replies
  step 2: 6.3% of all replies
```

## Write the verdict

`outbound.insights.variantTest` takes a campaign and returns one entry per tested step. `outbound.insights.stepFunnel` takes the same campaign and returns one row per step.

```ts title="verdict.ts"
import type { Outbound } from "outbound-sdk";
import type { InsightsExtension } from "outbound-sdk/insights";

const percent = (rate: number | null): string =>
  rate === null ? "not reported" : `${(rate * 100).toFixed(1)}%`;

/** Where one campaign's A/B tests stand today, as plain text. */
export const abVerdict = async (
  outbound: Outbound & InsightsExtension,
  campaign: string
): Promise<string> => {
  const test = await outbound.insights.variantTest({ campaign });
  const funnel = await outbound.insights.stepFunnel({ campaign });

  const lines = [
    `A/B tests of ${campaign}, judged on ${test.data.metric} at alpha ${test.data.alpha}`,
  ];

  for (const step of test.data.steps) {
    lines.push("", `Step ${step.step}: ${step.verdict}`, `  ${step.reason}`);

    for (const variant of step.variants) {
      lines.push(
        `  ${variant.variantKey}: ${percent(variant.rate)} (${variant.count} of ${variant.trials} ${variant.basis})${variant.enoughData ? "" : ", too few to compare"}`
      );
    }

    if (step.pValue !== null) {
      lines.push(`  p = ${step.pValue}`);
    }

    // A rough figure, not a date the test ends on. Under the minimum base it
    // is the sends still missing; otherwise it assumes both rates stay put.
    if (step.moreSendsNeeded !== null) {
      lines.push(
        step.verdict === "too_little_data"
          ? `  ${step.moreSendsNeeded} more sends before the variants are compared`
          : `  at these rates, about ${step.moreSendsNeeded} more sends before the gap would count`
      );
    }
  }

  // Which step the replies come from says how much a winner is worth.
  lines.push("", funnel.data.finding);

  for (const step of funnel.data.steps) {
    lines.push(
      `  step ${step.step}: ${percent(step.shareOfReplies)} of all replies`
    );
  }

  const notes = [...test.meta.notes, ...funnel.meta.notes];

  if (notes.length > 0) {
    lines.push(
      "",
      "Read this before relying on the numbers:",
      ...notes.map((note) => `  - ${note}`)
    );
  }

  return lines.join("\n");
};
```

How a step gets its verdict:

- **The rate.** Replies and positive replies are divided by leads contacted, or by sends when the vendor reports no contacted count. Opens and clicks are divided by messages delivered, or by sends. `basis` on each variant says which. Choose the metric with `metric` (default `replied`).
- **Enough data.** A variant whose base is under `min_sent` (default 100) is not compared, even when its raw rate is the highest. With fewer than two variants over that bar the verdict is `too_little_data`.
- **The test.** The leader is the variant with the highest rate among those with enough data. It is compared with every other one by a two-sided two-proportion z-test. It is a `winner` only when every comparison has a p-value under `alpha` (default 0.05, at most 0.5). Otherwise the verdict is `no_difference_yet`, and `pValue` is the leader against its closest rival.
- **The interval.** Each variant also carries the 95% Wilson interval of its rate in `interval`.
- **`moreSendsNeeded` is rough.** For `no_difference_yet` it is how much larger the sample would have to be if both rates stayed exactly where they are, which they will not. For `too_little_data` it is the sends still missing to bring every variant to `min_sent`. It is `null` for a winner.

A count the vendor does not report is `null`, never `0`, and a rate built on it is `null` too. `from` and `to` (`YYYY-MM-DD`) limit the days counted.

## Try it without a key

The in-memory adapter takes plain data. A stats row in its seed can name a step and a variant (`stepIndex` and `variantKey`), so this seed holds one row per variant, which is the shape the Instantly and Smartlead adapters produce for step stats.

```ts title="seed.ts"
// One made-up email campaign, "SaaS founders", that tests two versions of
// each of its three steps.
import type { MemoryData, MemoryStats } from "outbound-sdk/memory";

export const CAMPAIGN = "cmp_founders";

interface VariantCounts {
  readonly step: number;
  readonly variantKey: string;
  readonly contacted: number;
  readonly sent: number;
  readonly replied: number;
}

// What a vendor that splits step stats by variant would report.
const VARIANTS: readonly VariantCounts[] = [
  // Step 0: A is well ahead.
  { contacted: 800, replied: 48, sent: 800, step: 0, variantKey: "A" },
  { contacted: 800, replied: 24, sent: 800, step: 0, variantKey: "B" },
  // Step 1: B is ahead, by a gap that chance alone could produce.
  { contacted: 700, replied: 20, sent: 700, step: 1, variantKey: "A" },
  { contacted: 700, replied: 28, sent: 700, step: 1, variantKey: "B" },
  // Step 2: A has only reached 60 leads.
  { contacted: 60, replied: 3, sent: 60, step: 2, variantKey: "A" },
  { contacted: 400, replied: 5, sent: 400, step: 2, variantKey: "B" },
];

/** One stats row per variant, as the in-memory adapter's seed takes it. */
const toStats = (variant: VariantCounts): MemoryStats => ({
  campaignId: CAMPAIGN,
  counts: {
    contacted: variant.contacted,
    replied: variant.replied,
    sent: variant.sent,
  },
  date: "2026-11-13",
  stepIndex: variant.step,
  variantKey: variant.variantKey,
});

export const seed: MemoryData = {
  campaigns: [
    {
      id: CAMPAIGN,
      name: "SaaS founders",
      steps: [
        {
          variants: [
            {
              body: "Hi {{firstName}}, a question.",
              subject: "Quick question",
            },
            { body: "Hi {{firstName}}, an idea.", subject: "An idea for you" },
          ],
        },
        {
          delayDays: 3,
          variants: [
            { body: "Did you see my note?", subject: "Quick question" },
            { body: "One more thought.", subject: "Quick question" },
          ],
        },
        {
          delayDays: 5,
          variants: [
            { body: "Should I close the file?", subject: "Quick question" },
            { body: "Last note from me.", subject: "Quick question" },
          ],
        },
      ],
    },
  ],
  stats: VARIANTS.map(toStats),
};

/**
 * The same campaign as a vendor that does not split step stats by variant
 * reports it: no row names a variant, so each step is one total.
 */
export const unsplitSeed: MemoryData = {
  ...seed,
  stats: VARIANTS.map((variant) => {
    const { variantKey: _variantKey, ...row } = toStats(variant);

    return row;
  }),
};
```

```ts title="run.ts"
import { createOutbound } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { memory } from "outbound-sdk/memory";

import { CAMPAIGN, seed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

const outbound = createOutbound({
  adapter: memory({ data: seed, workspaceId: "acme" }),
  plugins: [insights()],
});

console.log(await abVerdict(outbound, CAMPAIGN));
```

That is the script that printed the verdict above. Step 0 is 48 of 800 against 24 of 800, a gap chance would produce about 4 times in 1,000. Step 1 is 28 of 700 against 20 of 700, a gap chance would produce about one time in four.

## Point it at your vendor

Swap the adapter. Nothing else changes.

```ts title="outbound.ts"
import { createOutbound } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { instantly } from "outbound-sdk/instantly";

// Reads INSTANTLY_API_KEY. Swap the import for smartlead(). HeyReach cannot
// serve this guide: it has no stats per step.
export const outbound = createOutbound({
  adapter: instantly({ workspaceId: "acme" }),
  // The two insights spend 3 reads between them.
  maxRequests: 10,
  plugins: [insights()],
});
```

Pass the campaign's vendor id or its `ref` to the function.

## What each vendor can serve

The same function, run on the in-memory adapter limited to what each vendor's adapter declares. This shows which calls a vendor's adapter would refuse. It does not show what that vendor's API returns.

```ts title="vendors.ts"
import { createOutbound, OutboundError } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";

import { asVendor, VENDORS } from "../_shared/as-vendor.ts";
import type { Vendor } from "../_shared/as-vendor.ts";
import { CAMPAIGN, seed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

const tryAs = async (vendor: Vendor): Promise<string> => {
  const outbound = createOutbound({
    adapter: asVendor(vendor, seed, "acme"),
    plugins: [insights()],
  });

  try {
    const verdict = await abVerdict(outbound, CAMPAIGN);

    return `${vendor}: ok, ${verdict.split("\n").length} lines`;
  } catch (error) {
    if (!OutboundError.is(error)) {
      throw error;
    }

    return `${vendor}: ${error.code}: ${error.message}`;
  }
};

const lines = await Promise.all(VENDORS.map(tryAs));

console.log(lines.join("\n"));
```

```text title="Output"
instantly: ok, 25 lines
smartlead: ok, 25 lines
heyreach: Unsupported: variant_test is not available: the heyreach adapter does not support stats.byStep
```

Both insights need `stats.byStep`. The HeyReach adapter declares it `false` because HeyReach's API has no statistics per step, so the call is refused before any request. There is no fallback: without counts per variant there is nothing to test.

## What it costs in vendor requests

`variantTest` is one stats read. `stepFunnel` is two: the sequence and the same stats read. Neither grows with the size of the campaign.

## When it cannot answer

```ts title="failures.ts"
import { createOutbound, OutboundError } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { memory } from "outbound-sdk/memory";

import { CAMPAIGN, seed, unsplitSeed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

// 1. A vendor whose step stats are not split by variant: the answer is an
//    empty list and a note, not a verdict.
const unsplit = createOutbound({
  adapter: memory({ data: unsplitSeed, workspaceId: "acme" }),
  plugins: [insights()],
});

const none = await unsplit.insights.variantTest({ campaign: CAMPAIGN });

console.log(
  "no variant rows:",
  none.data.variantStatsReported,
  none.data.steps.length,
  none.meta.notes
);

const outbound = createOutbound({
  adapter: memory({ data: seed, workspaceId: "acme" }),
  plugins: [insights()],
});

// 2. A metric the vendor does not report for the variants. Opens are not in
//    this data, so no step can be judged on them.
const opens = await outbound.insights.variantTest({
  campaign: CAMPAIGN,
  metric: "opened",
});

console.log("unreported metric:", opens.data.steps[0]?.reason);

// 3. An input the insight does not take is refused, never dropped.
try {
  // @ts-expect-error -- `step` is not an input of variantTest
  await outbound.insights.variantTest({ campaign: CAMPAIGN, step: 0 });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("unknown input:", error.code, "-", error.message);
  }
}

// 4. An alpha outside its range is refused before any request.
try {
  await outbound.insights.variantTest({ alpha: 0.9, campaign: CAMPAIGN });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("bad input:", error.code, "-", error.message);
  }
}

// 5. A campaign that does not exist. The first read of it says so.
try {
  await abVerdict(outbound, "cmp_missing");
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("unknown campaign:", error.code, "-", error.message);
  }
}
```

```text title="Output"
no variant rows: false 0 [
  'The step stats carry no variant rows, so there is nothing to compare.'
]
unreported metric: The vendor does not report opened for every variant of this step, so they cannot be compared.
unknown input: Invalid - variant_test: "step" is not an input. It takes: alpha, campaign, from, metric, min_sent, to
bad input: Invalid - variant_test: alpha Too big: expected number to be <=0.5
unknown campaign: NotFound - Campaign cmp_missing was not found
```

| What happened | What you get | What to do |
| --- | --- | --- |
| The step stats have no row per variant | `variantStatsReported` is `false`, `steps` is empty and a note says there is nothing to compare | Nothing can be tested from this data. Compare the variants in the vendor's own interface. |
| The vendor does not report the metric for every variant | `too_little_data`, with the reason in `reason` and `moreSendsNeeded` `null` | Pick a metric the vendor reports. More sends will not help. |
| An input the insight does not take | `Invalid`, naming the key and listing the inputs | Fix the key. There is no filter by step: the answer covers every tested step. |
| Input out of range | `Invalid`, before any request | Fix the value. |
| A campaign that does not exist | `NotFound`, from the first read of it | Check the id. |
| The adapter has no stats per step | `Unsupported`, before any request | Use an adapter that declares `stats.byStep`. |

## Limits

- **No stopping rule.** The test assumes you look once, at a sample size you chose in advance. Decide before launch how many leads each variant should reach, then read the verdict at that point. `moreSendsNeeded` is a rough guide to that size, not a signal to stop.
- **Checking daily is peeking.** Each look at a fixed `alpha` is another chance for noise to cross the line. If you run this on a schedule and act on the first `winner`, you will call more false winners than `alpha` says. Act on a verdict that holds at the planned sample size, or pass a smaller `alpha`.
- **No correction for three or more variants.** The leader is picked because it has the highest rate and is then tested against each other variant at the same `alpha`. With several variants the chance that the best-looking one clears every test by luck is higher than with two. Use a smaller `alpha` for a step with more than two variants.
- It tests what the vendor counts. If the vendor attributes a reply to the wrong step or variant, so does the verdict.
- A `winner` says the gap is unlikely to be chance. It does not say the gap is large enough to matter: read the two intervals for that.