Tell whether an A/B variant is ahead by more than chance

One function that reads a campaign's step stats and says, for each step with two or more variants, whether the leader's gap is larger than chance would give, plus which step the replies come from.

Variant A has a 6% reply rate and variant B has 3%. Whether that is a result or noise depends on how many leads each one reached. variantTest answers that for every step of a campaign that has two or more variants: it takes the variant with the highest rate, compares it with each of the others by a two-proportion z-test, and returns one of three verdicts: winner, no_difference_yet or too_little_data. stepFunnel adds which step the replies come from, so you know how much a winner is worth.

It does not tell you when a test is finished. It has no stopping rule, it makes no correction when a step has three or more variants, and reading it every day until it says winner will produce winners that are not real. Limits says what to do about each. It calls no AI model and changes nothing in your vendor tool: turning a variant off is up to you.

Before you start

• Node 22 or later.
• The code below runs on the in-memory adapter, so it needs no vendor key. Each sample on this page is a file in the repository that the test suite runs; the output shown is what it printed.
• For real data: an Instantly or Smartlead API key. A read-only key is enough. HeyReach cannot serve this guide, see what each vendor can serve.

What you get

A/B tests of cmp_founders, judged on replied at alpha 0.05

Step 0: winner
  Variant A (0.06) beats every other variant by more than chance at alpha 0.05.
  A: 6.0% (48 of 800 contacted)
  B: 3.0% (24 of 800 contacted)
  p = 0.0038

Step 1: no_difference_yet
  Variant B (0.04) leads variant A (0.0286), but a gap this size could be chance.
  A: 2.9% (20 of 700 contacted)
  B: 4.0% (28 of 700 contacted)
  p = 0.24
  at these rates, about 2498 more sends before the gap would count

Step 2: too_little_data
  Fewer than two variants have a base of 100, so no comparison is made.
  A: 5.0% (3 of 60 contacted), too few to compare
  B: 1.3% (5 of 400 contacted)
  40 more sends before the variants are compared

Step 0 (counted from 0) brings the most replies: 72 of 128. Every step up to step 2 still earns its place.
  step 0: 56.3% of all replies
  step 1: 37.5% of all replies
  step 2: 6.3% of all replies

Write the verdict

outbound.insights.variantTest takes a campaign and returns one entry per tested step. outbound.insights.stepFunnel takes the same campaign and returns one row per step.

import type { Outbound } from "outbound-sdk";
import type { InsightsExtension } from "outbound-sdk/insights";

const percent = (rate: number | null): string =>
  rate === null ? "not reported" : `${(rate * 100).toFixed(1)}%`;

/** Where one campaign's A/B tests stand today, as plain text. */
export const abVerdict = async (
  outbound: Outbound & InsightsExtension,
  campaign: string
): Promise<string> => {
  const test = await outbound.insights.variantTest({ campaign });
  const funnel = await outbound.insights.stepFunnel({ campaign });

  const lines = [
    `A/B tests of ${campaign}, judged on ${test.data.metric} at alpha ${test.data.alpha}`,
  ];

  for (const step of test.data.steps) {
    lines.push("", `Step ${step.step}: ${step.verdict}`, `  ${step.reason}`);

    for (const variant of step.variants) {
      lines.push(
        `  ${variant.variantKey}: ${percent(variant.rate)} (${variant.count} of ${variant.trials} ${variant.basis})${variant.enoughData ? "" : ", too few to compare"}`
      );
    }

    if (step.pValue !== null) {
      lines.push(`  p = ${step.pValue}`);
    }

    // A rough figure, not a date the test ends on. Under the minimum base it
    // is the sends still missing; otherwise it assumes both rates stay put.
    if (step.moreSendsNeeded !== null) {
      lines.push(
        step.verdict === "too_little_data"
          ? `  ${step.moreSendsNeeded} more sends before the variants are compared`
          : `  at these rates, about ${step.moreSendsNeeded} more sends before the gap would count`
      );
    }
  }

  // Which step the replies come from says how much a winner is worth.
  lines.push("", funnel.data.finding);

  for (const step of funnel.data.steps) {
    lines.push(
      `  step ${step.step}: ${percent(step.shareOfReplies)} of all replies`
    );
  }

  const notes = [...test.meta.notes, ...funnel.meta.notes];

  if (notes.length > 0) {
    lines.push(
      "",
      "Read this before relying on the numbers:",
      ...notes.map((note) => `  - ${note}`)
    );
  }

  return lines.join("\n");
};

How a step gets its verdict:

• The rate. Replies and positive replies are divided by leads contacted, or by sends when the vendor reports no contacted count. Opens and clicks are divided by messages delivered, or by sends. basis on each variant says which. Choose the metric with metric (default replied).
• Enough data. A variant whose base is under min_sent (default 100) is not compared, even when its raw rate is the highest. With fewer than two variants over that bar the verdict is too_little_data.
• The test. The leader is the variant with the highest rate among those with enough data. It is compared with every other one by a two-sided two-proportion z-test. It is a winner only when every comparison has a p-value under alpha (default 0.05, at most 0.5). Otherwise the verdict is no_difference_yet, and pValue is the leader against its closest rival.
• The interval. Each variant also carries the 95% Wilson interval of its rate in interval.
• moreSendsNeeded is rough. For no_difference_yet it is how much larger the sample would have to be if both rates stayed exactly where they are, which they will not. For too_little_data it is the sends still missing to bring every variant to min_sent. It is null for a winner.

A count the vendor does not report is null, never 0, and a rate built on it is null too. from and to (YYYY-MM-DD) limit the days counted.

Try it without a key

The in-memory adapter takes plain data. A stats row in its seed can name a step and a variant (stepIndex and variantKey), so this seed holds one row per variant, which is the shape the Instantly and Smartlead adapters produce for step stats.

// One made-up email campaign, "SaaS founders", that tests two versions of
// each of its three steps.
import type { MemoryData, MemoryStats } from "outbound-sdk/memory";

export const CAMPAIGN = "cmp_founders";

interface VariantCounts {
  readonly step: number;
  readonly variantKey: string;
  readonly contacted: number;
  readonly sent: number;
  readonly replied: number;
}

// What a vendor that splits step stats by variant would report.
const VARIANTS: readonly VariantCounts[] = [
  // Step 0: A is well ahead.
  { contacted: 800, replied: 48, sent: 800, step: 0, variantKey: "A" },
  { contacted: 800, replied: 24, sent: 800, step: 0, variantKey: "B" },
  // Step 1: B is ahead, by a gap that chance alone could produce.
  { contacted: 700, replied: 20, sent: 700, step: 1, variantKey: "A" },
  { contacted: 700, replied: 28, sent: 700, step: 1, variantKey: "B" },
  // Step 2: A has only reached 60 leads.
  { contacted: 60, replied: 3, sent: 60, step: 2, variantKey: "A" },
  { contacted: 400, replied: 5, sent: 400, step: 2, variantKey: "B" },
];

/** One stats row per variant, as the in-memory adapter's seed takes it. */
const toStats = (variant: VariantCounts): MemoryStats => ({
  campaignId: CAMPAIGN,
  counts: {
    contacted: variant.contacted,
    replied: variant.replied,
    sent: variant.sent,
  },
  date: "2026-11-13",
  stepIndex: variant.step,
  variantKey: variant.variantKey,
});

export const seed: MemoryData = {
  campaigns: [
    {
      id: CAMPAIGN,
      name: "SaaS founders",
      steps: [
        {
          variants: [
            {
              body: "Hi {{firstName}}, a question.",
              subject: "Quick question",
            },
            { body: "Hi {{firstName}}, an idea.", subject: "An idea for you" },
          ],
        },
        {
          delayDays: 3,
          variants: [
            { body: "Did you see my note?", subject: "Quick question" },
            { body: "One more thought.", subject: "Quick question" },
          ],
        },
        {
          delayDays: 5,
          variants: [
            { body: "Should I close the file?", subject: "Quick question" },
            { body: "Last note from me.", subject: "Quick question" },
          ],
        },
      ],
    },
  ],
  stats: VARIANTS.map(toStats),
};

/**
 * The same campaign as a vendor that does not split step stats by variant
 * reports it: no row names a variant, so each step is one total.
 */
export const unsplitSeed: MemoryData = {
  ...seed,
  stats: VARIANTS.map((variant) => {
    const { variantKey: _variantKey, ...row } = toStats(variant);

    return row;
  }),
};

import { createOutbound } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { memory } from "outbound-sdk/memory";

import { CAMPAIGN, seed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

const outbound = createOutbound({
  adapter: memory({ data: seed, workspaceId: "acme" }),
  plugins: [insights()],
});

console.log(await abVerdict(outbound, CAMPAIGN));

That is the script that printed the verdict above. Step 0 is 48 of 800 against 24 of 800, a gap chance would produce about 4 times in 1,000. Step 1 is 28 of 700 against 20 of 700, a gap chance would produce about one time in four.

Point it at your vendor

Swap the adapter. Nothing else changes.

import { createOutbound } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { instantly } from "outbound-sdk/instantly";

// Reads INSTANTLY_API_KEY. Swap the import for smartlead(). HeyReach cannot
// serve this guide: it has no stats per step.
export const outbound = createOutbound({
  adapter: instantly({ workspaceId: "acme" }),
  // The two insights spend 3 reads between them.
  maxRequests: 10,
  plugins: [insights()],
});

Pass the campaign’s vendor id or its ref to the function.

What each vendor can serve

The same function, run on the in-memory adapter limited to what each vendor’s adapter declares. This shows which calls a vendor’s adapter would refuse. It does not show what that vendor’s API returns.

import { createOutbound, OutboundError } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";

import { asVendor, VENDORS } from "../_shared/as-vendor.ts";
import type { Vendor } from "../_shared/as-vendor.ts";
import { CAMPAIGN, seed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

const tryAs = async (vendor: Vendor): Promise<string> => {
  const outbound = createOutbound({
    adapter: asVendor(vendor, seed, "acme"),
    plugins: [insights()],
  });

  try {
    const verdict = await abVerdict(outbound, CAMPAIGN);

    return `${vendor}: ok, ${verdict.split("\n").length} lines`;
  } catch (error) {
    if (!OutboundError.is(error)) {
      throw error;
    }

    return `${vendor}: ${error.code}: ${error.message}`;
  }
};

const lines = await Promise.all(VENDORS.map(tryAs));

console.log(lines.join("\n"));

instantly: ok, 25 lines
smartlead: ok, 25 lines
heyreach: Unsupported: variant_test is not available: the heyreach adapter does not support stats.byStep

Both insights need stats.byStep. The HeyReach adapter declares it false because HeyReach’s API has no statistics per step, so the call is refused before any request. There is no fallback: without counts per variant there is nothing to test.

What it costs in vendor requests

variantTest is one stats read. stepFunnel is two: the sequence and the same stats read. Neither grows with the size of the campaign.

When it cannot answer

import { createOutbound, OutboundError } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { memory } from "outbound-sdk/memory";

import { CAMPAIGN, seed, unsplitSeed } from "./seed.ts";
import { abVerdict } from "./verdict.ts";

// 1. A vendor whose step stats are not split by variant: the answer is an
//    empty list and a note, not a verdict.
const unsplit = createOutbound({
  adapter: memory({ data: unsplitSeed, workspaceId: "acme" }),
  plugins: [insights()],
});

const none = await unsplit.insights.variantTest({ campaign: CAMPAIGN });

console.log(
  "no variant rows:",
  none.data.variantStatsReported,
  none.data.steps.length,
  none.meta.notes
);

const outbound = createOutbound({
  adapter: memory({ data: seed, workspaceId: "acme" }),
  plugins: [insights()],
});

// 2. A metric the vendor does not report for the variants. Opens are not in
//    this data, so no step can be judged on them.
const opens = await outbound.insights.variantTest({
  campaign: CAMPAIGN,
  metric: "opened",
});

console.log("unreported metric:", opens.data.steps[0]?.reason);

// 3. An input the insight does not take is refused, never dropped.
try {
  // @ts-expect-error -- `step` is not an input of variantTest
  await outbound.insights.variantTest({ campaign: CAMPAIGN, step: 0 });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("unknown input:", error.code, "-", error.message);
  }
}

// 4. An alpha outside its range is refused before any request.
try {
  await outbound.insights.variantTest({ alpha: 0.9, campaign: CAMPAIGN });
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("bad input:", error.code, "-", error.message);
  }
}

// 5. A campaign that does not exist. The first read of it says so.
try {
  await abVerdict(outbound, "cmp_missing");
} catch (error) {
  if (OutboundError.is(error)) {
    console.log("unknown campaign:", error.code, "-", error.message);
  }
}

no variant rows: false 0 [
  'The step stats carry no variant rows, so there is nothing to compare.'
]
unreported metric: The vendor does not report opened for every variant of this step, so they cannot be compared.
unknown input: Invalid - variant_test: "step" is not an input. It takes: alpha, campaign, from, metric, min_sent, to
bad input: Invalid - variant_test: alpha Too big: expected number to be <=0.5
unknown campaign: NotFound - Campaign cmp_missing was not found

Limits

• No stopping rule. The test assumes you look once, at a sample size you chose in advance. Decide before launch how many leads each variant should reach, then read the verdict at that point. moreSendsNeeded is a rough guide to that size, not a signal to stop.
• Checking daily is peeking. Each look at a fixed alpha is another chance for noise to cross the line. If you run this on a schedule and act on the first winner, you will call more false winners than alpha says. Act on a verdict that holds at the planned sample size, or pass a smaller alpha.
• No correction for three or more variants. The leader is picked because it has the highest rate and is then tested against each other variant at the same alpha. With several variants the chance that the best-looking one clears every test by luck is higher than with two. Use a smaller alpha for a step with more than two variants.
• It tests what the vendor counts. If the vendor attributes a reply to the wrong step or variant, so does the verdict.
• A winner says the gap is unlikely to be chance. It does not say the gap is large enough to matter: read the two intervals for that.