Copy and sequence insights

Reference for the five insights about copy (stepFunnel, variantTest, copyAudit, copyReuse, replyTiming), with every input, what the answer holds and a sample that runs.

Five insights about what a campaign says and when. Each is a method on outbound.insights once the insights plugin is added. What every answer carries and how inputs are checked is on the overview.

Three of them need per-step stats (stats.byStep). An adapter that does not declare it refuses them as Unsupported: see which adapter serves which.

A reply rate here is replies divided by leads contacted, or by messages sent when the vendor does not count leads. Each answer says which in basis or replyRateBasis.

The data the samples run on

Two email campaigns. “SaaS founders” tests two variants on its first two steps and has a careless third step. “Agency owners” reuses two pieces of its copy.

// Made-up data for one email tool: a campaign with two A/B tested steps and a
// careless third step, and a second campaign that reuses two pieces of copy.
import type {
  MemoryData,
  MemoryMessage,
  MemoryStats,
} from "outbound-sdk/memory";

const OPENER = {
  body: "Hi {{firstName}}, I saw {{company}} is hiring two account executives. Worth a short call next week?",
  subject: "Quick question about {{company}}",
};

const FOLLOW_UP = { body: "Hi {{firstName}}, did my note reach you?" };

const pair = (
  lead: string,
  sentAt: string,
  ...repliedAt: string[]
): MemoryMessage[] => [
  {
    campaignId: "founders",
    direction: "out",
    id: `${lead}_out`,
    leadId: lead,
    sentAt,
    threadId: `thread_${lead}`,
  },
  ...repliedAt.map((at, index) => ({
    campaignId: "founders",
    direction: "in" as const,
    id: `${lead}_in${index}`,
    leadId: lead,
    sentAt: at,
    threadId: `thread_${lead}`,
  })),
];

/** One step's counts. A row with a `variantKey` is one variant of the step. */
const step = (
  campaignId: string,
  stepIndex: number,
  counts: { replied: number; sent: number }
): MemoryStats => ({ campaignId, counts, date: "2026-11-10", stepIndex });

export const seed: MemoryData = {
  campaigns: [
    {
      id: "founders",
      name: "SaaS founders",
      steps: [
        {
          variants: [
            OPENER,
            {
              body: "Hi {{firstName}}, most teams your size lose a day a week to manual follow-up. Want to see how {{company}} could get it back?",
              subject: "Idea for {{company}}",
            },
          ],
        },
        {
          delayDays: 3,
          variants: [
            FOLLOW_UP,
            { body: "Hi {{firstName}}, is this on your list this quarter?" },
          ],
        },
        {
          delayDays: 5,
          variants: [
            {
              body: "Hi {{firstName}, see https://example.com/pricing and https://example.com/demo before FRIDAY. This offer ENDS soon!!",
              subject:
                "Last chance: a free trial for {{company}}, with no obligation at all",
            },
          ],
        },
      ],
    },
    {
      id: "agencies",
      name: "Agency owners",
      steps: [{ variants: [OPENER] }, { delayDays: 3, variants: [FOLLOW_UP] }],
    },
  ],
  messages: [
    // 2026-11-09 is a Monday. Replies after 0.5, 6, 24, 48 and 96 hours.
    ...pair("lead1", "2026-11-09T09:00:00.000Z", "2026-11-09T09:30:00.000Z"),
    ...pair("lead2", "2026-11-09T09:00:00.000Z", "2026-11-09T15:00:00.000Z"),
    // The second message from lead3 is a follow-on, not a second reply time.
    ...pair(
      "lead3",
      "2026-11-10T09:00:00.000Z",
      "2026-11-11T09:00:00.000Z",
      "2026-11-11T10:00:00.000Z"
    ),
    ...pair("lead4", "2026-11-10T10:00:00.000Z", "2026-11-12T10:00:00.000Z"),
    ...pair("lead5", "2026-11-09T09:00:00.000Z", "2026-11-13T09:00:00.000Z"),
    // A reply with no send before it in what was read.
    {
      campaignId: "founders",
      direction: "in",
      id: "lead6_in0",
      leadId: "lead6",
      sentAt: "2026-11-12T15:00:00.000Z",
      threadId: "thread_lead6",
    },
  ],
  stats: [
    step("founders", 0, { replied: 60, sent: 1000 }),
    // How the vendor splits the first two steps of "SaaS founders" between their variants.
    { ...step("founders", 0, { replied: 40, sent: 500 }), variantKey: "A" },
    { ...step("founders", 0, { replied: 20, sent: 500 }), variantKey: "B" },
    step("founders", 1, { replied: 16, sent: 800 }),
    { ...step("founders", 1, { replied: 10, sent: 400 }), variantKey: "A" },
    { ...step("founders", 1, { replied: 6, sent: 400 }), variantKey: "B" },
    step("founders", 2, { replied: 1, sent: 600 }),
    step("agencies", 0, { replied: 12, sent: 600 }),
    step("agencies", 1, { replied: 9, sent: 450 }),
  ],
};

A stats row in the in-memory adapter’s seed can name a variant as well as a step. “SaaS founders” has a row per step and, for its first two steps, a row per variant, which is what a vendor that splits a step by variant returns. The instance is the in-memory adapter with the insights plugin:

import { createOutbound } from "outbound-sdk";
import { insights } from "outbound-sdk/insights";
import { memory } from "outbound-sdk/memory";

import { seed } from "./seed.ts";

export const outbound = createOutbound({
  adapter: memory({ data: seed, workspaceId: "acme" }),
  plugins: [insights()],
});

stepFunnel

Where a campaign’s replies come from, one row per sequence step.

data (StepFunnel):

• finding: what the numbers say, in one or two sentences.

• topStep: the step with the most replies, or null.

• lastStepEarningPlace: the last step whose reply rate is reliably above min_reply_rate, or null.

• steps: one entry per step, counted from 0, with action, delayDays, inSequence, sent, replies, replyRate, replyRateInterval, replyRateBasis, shareOfReplies, sentDrop and sentDropRate (how far sends fell from the step before), earnsPlace, and variants (empty when the vendor does not split step stats by variant).

• totals: sent and replies.

• unattributed: counts the vendor tied to no step (rows, sent, replies). They are in no step’s row.

• Needs: sequence, stats.byStep.

• Costs: 2 requests.

• Notes it can add: some counts are tied to no step by the vendor and are reported as unattributed.

• Does not: tell you why a step is weak. A step whose stats name it but which the sequence no longer has is listed with inSequence: false. A null count means the vendor does not report it.

import { outbound } from "./outbound.ts";

const { data } = await outbound.insights.stepFunnel({
  campaign: "memory:acme:founders",
  min_reply_rate: 0.01,
});

console.log(data.finding);

for (const step of data.steps) {
  console.log(
    `step ${step.step}: sent ${step.sent}, replies ${step.replies}, rate ${step.replyRate} (${step.replyRateInterval?.low} to ${step.replyRateInterval?.high}), share ${step.shareOfReplies}, drop ${step.sentDropRate}, earns place ${step.earnsPlace}`
  );
}

console.log(
  "step 0 variants:",
  data.steps[0]?.variants.map((each) => `${each.variantKey} ${each.replyRate}`)
);

Step 0 (counted from 0) brings the most replies: 60 of 77. Step 1 is the last whose reply rate is reliably above the bar; the 1 later step(s) sent 600 messages for 1 replies.
step 0: sent 1000, replies 60, rate 0.06 (0.0469 to 0.0765), share 0.7792, drop null, earns place true
step 1: sent 800, replies 16, rate 0.02 (0.0123 to 0.0322), share 0.2078, drop 0.2, earns place true
step 2: sent 600, replies 1, rate 0.0017 (0.0003 to 0.0094), share 0.013, drop 0.25, earns place false
step 0 variants: [ 'A 0.08', 'B 0.04' ]

Step 2 has a reply, but one reply from 600 sends gives an interval of 0.03% to 0.94%, which does not clear the 1% bar set here.

variantTest

Whether a step’s A/B leader is ahead by more than chance.

data (VariantTest):

• campaign, metric, alpha: what was asked.

• variantStatsReported: false when the vendor’s step stats carry no variant rows at all.

• steps: one entry per step with two or more variants, with verdict (winner, no_difference_yet or too_little_data), leader, pValue (the leader against its closest rival), moreSendsNeeded, a reason sentence, and variants (each with sent, count, trials, basis, rate, interval, enoughData and pValueVsLeader). Winners first.

• totals: stepsTested, winners, noDifferenceYet and tooLittleData.

• Needs: stats.byStep.

• Costs: 1 request.

• Notes it can add: the step stats carry no variant rows, so there is nothing to compare.

• Does not: correct for comparing three or more variants. A leader must beat every other variant with enough data to be called a winner. moreSendsNeeded is rough: it assumes both rates stay as they are. It pauses nothing.

import { outbound } from "./outbound.ts";

const { data } = await outbound.insights.variantTest({
  campaign: "memory:acme:founders",
});

for (const step of data.steps) {
  console.log(`step ${step.step}: ${step.verdict}, p-value ${step.pValue}`);
  console.log(`  ${step.reason}`);
  console.log("  more sends needed:", step.moreSendsNeeded);
}

console.log("totals:", data.totals);

step 0: winner, p-value 0.0077
  Variant A (0.08) beats every other variant by more than chance at alpha 0.05.
  more sends needed: null
step 1: no_difference_yet, p-value 0.3124
  Variant A (0.025) leads variant B (0.015), but a gap this size could be chance.
  more sends needed: 2213
totals: { noDifferenceYet: 1, stepsTested: 2, tooLittleData: 0, winners: 1 }

On step 1, variant A leads 2.5% to 1.5%, but with 400 sends each that gap has a p-value of 0.31. Pausing B now would be a guess.

copyAudit

Problems in a campaign’s copy, found from fixed rules before it sends.

The default spam_words are: 100% free, act now, buy now, cash bonus, click here, double your, earn money, free trial, guarantee, limited time, no obligation, risk-free, urgent, winner.

data (CopyAudit):

• campaign, channel.
• clean: true when no check found anything in the copy that could be read.
• findings: one entry per problem, with step, variantKey, action, check, severity, a finding sentence and the text that triggered it. High severity first.
• byCheck: every check that was run, with how often it fired.
• bySeverity: high, medium, low.
• stepsChecked, variantsChecked, variantsWithoutBody.

The checks:

• Needs: sequence.
• Costs: 2 requests.
• Notes it can add: some variants have no body in the vendor’s answer, so their body checks were skipped.
• Does not: read the copy with a model, judge tone or check facts. It reads template text, never a rendered message. A variant whose body the vendor does not return is counted in variantsWithoutBody, not judged.

import { outbound } from "./outbound.ts";

const { data } = await outbound.insights.copyAudit({
  campaign: "memory:acme:founders",
});

for (const finding of data.findings) {
  console.log(
    `step ${finding.step} ${finding.severity} ${finding.check}: ${finding.finding}`
  );
}

console.log("by severity:", data.bySeverity);

console.log(
  `checked ${data.variantsChecked} variants in ${data.stepsChecked} steps, clean: ${data.clean}`
);

step 2 high malformed_variable: A variable's braces are not balanced, so it sends as raw text.
step 2 medium spam_words: The copy uses 2 word(s) or phrase(s) that spam filters weigh.
step 2 medium subject_breaks_thread: This follow-up has a subject that differs from the first email's, so it starts a new thread instead of continuing the first.
step 2 medium too_many_links: The body has 2 links; the limit is 1.
step 2 low all_caps: 2 word(s) of four or more letters are in capitals; the limit is 1.
step 2 low exclamation_marks: The copy has 2 exclamation marks; the limit is 1.
step 2 low subject_too_long: The subject is 68 characters; the limit is 60.
by severity: { high: 1, low: 3, medium: 3 }
checked 5 variants in 3 steps, clean: false

copyReuse

How the same piece of copy performs in different campaigns. Variants are matched by the hash of their subject and body.

data (CopyReuse):

• reused: every piece of copy used in two or more campaigns, with contentHash, subject, bodyPreview (the first 120 characters), campaigns, pooled sent, replies, trials, replyRate and replyRateInterval, uses (per campaign: places as step:variant, counts and rate), spread (best minus worst reply rate), best, worst, spreadPValue, spreadIsReal and a finding sentence. Widest spread first.

• bestCopy: the copy with the highest pooled reply rate among those with a base of min_sent, or null.

• campaignsRead, piecesOfCopy, variantsWithoutStats.

• Needs: sequence, stats.byStep.

• Costs: 2 requests per campaign, plus the campaign listing.

• Notes it can add: some variants have no counts of their own, because the vendor reports their step as a whole or not at all, so their copy is listed without numbers.

• Does not: match copy that differs by one character: the hash is of the exact subject and body. The spread test is always at the 0.05 bar. When a step has several variants and the vendor does not split them, those variants have no numbers.

import { outbound } from "./outbound.ts";

const { data } = await outbound.insights.copyReuse({});

for (const copy of data.reused) {
  console.log(`"${copy.bodyPreview?.slice(0, 40)}..."`);
  console.log(`  ${copy.finding}`);
  console.log(
    `  spread ${copy.spread}, p-value ${copy.spreadPValue}, real: ${copy.spreadIsReal}`
  );
}

console.log(
  `best copy: "${data.bestCopy?.subject}" at ${data.bestCopy?.replyRate} over ${data.bestCopy?.trials} sends`
);

console.log(
  `${data.piecesOfCopy} pieces of copy in ${data.campaignsRead} campaigns`
);

"Hi {{firstName}}, I saw {{company}} is h..."
  The same copy gets 0.08 in SaaS founders and 0.02 in Agency owners; the gap is more than chance, so the audience or the sender differs, not the copy.
  spread 0.06, p-value 0, real: true
"Hi {{firstName}}, did my note reach you?..."
  The same copy gets 0.025 in SaaS founders and 0.02 in Agency owners; a gap this size could be chance.
  spread 0.005, p-value 0.6226, real: false
best copy: "Quick question about {{company}}" at 0.0473 over 1100 sends
5 pieces of copy in 2 campaigns

The opener gets 8% in one campaign and 2% in the other. Same words, so the audience or the sender differs. The best copy’s 4.73% is pooled: 52 replies over 1,100 sends in both campaigns.

replyTiming

How fast people reply, and on which days and hours.

data (ReplyTiming):

• campaign, since, timezoneOffsetMinutes: what was used.

• timeToReply: sample (replies paired with the message they answer), medianHours, p75Hours, p90Hours, and the share within1Hour, within24Hours and within72Hours.

• replies: when replies arrive, as byWeekday (Monday first), byHour (0 to 23), topWeekday and topHour.

• sendsThatDrewReplies: the same tables for the sends that drew a reply.

• counts: messagesRead, sent, received, paired, followOnReplies and unpaired.

• Needs: messages.list.

• Costs: 1 request per page of messages, up to max_messages.

• Notes it can add: the adapter cannot filter messages by date, so the cap applies to all messages and older ones were dropped after reading; a cap was reached.

• Does not: separate automatic replies, because messages carry no automatic flag. A reply is paired with the last message sent before it in the same thread, or to the same lead. A further message from someone who already replied is counted in followOnReplies and not timed. A reply with no earlier send in what was read is unpaired. The offset is fixed, so it does not follow daylight saving.

import { outbound } from "./outbound.ts";

const { data } = await outbound.insights.replyTiming({});

console.log("time to reply:", data.timeToReply);

console.log(
  `replies arrive most on ${data.replies.topWeekday} and at hour ${data.replies.topHour}`
);

console.log(
  `sends that drew a reply went out most on ${data.sendsThatDrewReplies.topWeekday} at hour ${data.sendsThatDrewReplies.topHour}`
);

console.log("counts:", data.counts);

time to reply: {
  medianHours: 24,
  p75Hours: 48,
  p90Hours: 76.8,
  sample: 5,
  within1Hour: 0.2,
  within24Hours: 0.6,
  within72Hours: 0.8
}
replies arrive most on Monday and at hour 9
sends that drew a reply went out most on Monday at hour 9
counts: {
  followOnReplies: 1,
  messagesRead: 12,
  paired: 5,
  received: 7,
  sent: 5,
  unpaired: 1
}

Five replies could be paired, after 0.5, 6, 24, 48 and 96 hours. Every figure in timeToReply rests on those five: read sample before trusting a percentile.