Skip to main content
You think the shorter opener will book more. Find out. This walk-through takes a live campaign, adds a second variant, changes one setting, and reads the results side by side. Concepts are in Agent split tests.

​
Prerequisites

  • A live campaign with steady lead flow. A split test on ten leads a week tells you nothing for months.
  • If the campaign is a master agent, work from the parent. The split reaches every linked project, and each applies the weights to its own leads.
  • An API key with agents:write, and dashboard:read to read results.
  • One hypothesis. Change one thing per test, or you will not know which change did it.

​
Step 1: Convert the campaign

Call Convert campaign to split test for the agent. The response is the split: your existing agent as the primary variant, a second variant cloned from it, equal weight on each, status: active, and a revision. Keep the revision; every write wants it back. Nothing has changed for leads yet. Both variants are identical.

​
Step 2: Change one thing on the new variant

The second variant is a real agent with its own id. Call Update agent on that id and change the one setting you are testing. For an opener test that is config.llm.beginMessage. For a voice test it is config.voice.voiceId. Only the llm, voice, interaction, language and callScreening blocks differ per variant. Anything else you change fans out to every variant, which is the point: skills and outcomes stay identical so the comparison is fair.

​
Step 3: Set the weights

Skip this for an even test. Otherwise call Update split weights with every variant listed exactly once, whole numbers totalling 100, and the current revision. A 409 means someone else saved first: read the split again and resend. Weights only steer new leads. A lead who has already met a variant stays with it for every follow-up.

​
Step 4: Let it run

Leave it alone. Resist the urge to peek at day two and switch. Every variant’s name appears as agent on each call record, so you can listen to a few calls from each while you wait.

​
Step 5: Read the results

Each variant has an agentVariantId. Call Get analytics summary once per variant with the same window:
Compare bookingRate, conversationRate and pickupRate. Pickup should be near identical, since both variants dial from the same numbers at the same times. If it is not, something other than your change is different. To hear why one wins, filter call records with the same campaignId and agentVariantId and read transcripts from each.
Two variants at a 5% booking rate need hundreds of leads each before a gap of a point or two means anything. There is no formula in the product for this. Treat small differences on small samples as noise.

​
Step 6: Keep the winner

If the winner is a challenger, carry its winning setting onto the primary with Update agent. The primary is the campaign’s own agent and always takes traffic, so that is what makes the campaign run the version that won. Then decide what to do with the losers:
  • Exclude them. Send their weight as 0 in Update split weights. They stop taking calls, and stay in the split so their numbers stay comparable and you can put them back later.
  • Remove them. Call Remove split variant with the revision. Their weight goes to the live survivors. When one member is left, status reads ended and the campaign is an ordinary campaign again.
Exclude while you are still learning. Remove when you are certain, because it is the one that cannot be undone.

​
What happens to an excluded variant’s leads

Those leads still have calls owed to them, and they still get made. A live variant covers each one, and it is the same live variant every time, so the lead is not introduced to a new agent on every call. The cover is temporary and nothing about the lead is rewritten. Send the excluded variant a weight of 1 or more and it has its leads back on their very next call. That is the difference between excluding and removing: a removed variant’s leads are re-drawn once and belong to their new variant for good. Calls made while covering do not count towards the covering variant’s own numbers. The lead was never drawn into that variant’s share, so counting them would flatter whichever variant happens to absorb the cover and make the comparison you are running unreadable. They stay in the campaign’s totals, because the calls did happen.

​
Step 7: Line up the next challenger

Add a variant and exclude it straight away, in that order: Add split variant, then Update split weights with 0 for the new one. A new variant joins the cycle live, so until you exclude it, it is taking a share of your leads. Now build it in the open. Edit the prompt, place test calls against it, and compare them against the calls the live arms are producing. It costs you nothing in live traffic until you send it a weight of 1 or more, and shared settings keep reaching it while you work, so there is no catching up to do on the day you include it. To test something else from a clean baseline, end the split and convert again. The new split assigns leads afresh, so compare results from the day you converted.

​
Next steps