Campaign engine
- TypeScript
- LLMs
- Instantly
- HubSpot
- DuckDB
The problem
An enriched lead isn't yet something you can send. It still needs a cold sequence written to its pain: 5–7 emails in one thread plus one SMS. That copy has to get past spam filters, and the row has to load into the sequencer with the 13 contact fields the sequencer-to-HubSpot sync reads.
The predecessor repo had four campaign types and four duplicated runners. A fix to scoring, QC or export in one runner didn't reach the other three.
Constraints
- Deliverability. Cold email gets flagged for spam words, for links in early emails, and for sequences that all share one skeleton.
- LLMs are bad at rules. Models invent merge tags and bracket placeholders, add extra question marks and em dashes, and drift on word count.
- Spend. The larger model costs about 20× the small one per token, and each rewrite resends the whole prompt.
- Bad rows break the CRM. A row missing its email, company name or place ID fails the HubSpot sync.
How it works
Each campaign type is a plugin with four parts: a scorer, a segmenter, a narrator and a deployer. One runner drives all four types. The rule is LLMs write and judge; code decides. Segmentation, scoring, delays, subjects, signatures, tracking links and the hard gate are TypeScript. The model only writes prose and gives a subjective QC score.
- Load. The runner reads enriched leads and runs 40 at a time.
- Score the pain. Each pain signal has a confidence, a weight and a category weight, which combine into a 0–100 pain score.
- Segment by rules. The first matching rule wins, across 18 segments in four types (for example "phone-only" or "after-hours leak"). A lead that fits nothing is disqualified before any LLM call.
- Check it's sendable. Required fields and email-validation status are checked before generation, so no LLM spend goes to a lead that can't ship.
- Write, then clean and check. The LLM writes the sequence. Code normalizes the copy: dashes become commas, stray merge tags and brackets are stripped, and only one question mark survives. A regex scan looks for spam words, with known-safe phrases masked out first. Structural checks cover word count, question marks, required links and where the brand may appear.
- Fix only what failed. Each failing email is rewritten on its own, in parallel, for up to 3 rounds.
- Judge. A separate LLM call scores tone, personalization, segment bleed and repetition. If the judge's output won't parse, that counts as a fail, not a pass.
- Hard gate. In three of the four types, a lead whose copy still has spam words after retries is dropped and logged. Softer issues ship with the QC column set to false so a human can filter them out.
- Assemble in code. Only email 1 gets a subject, and the rest stay in the same thread. A fixed delay schedule sends on days 0, 2, 3, 5, 6, 8 and 10. Signatures rotate per step, and tracking links are added. All of this happens after QC, so the model can't break it.
- Export. A CSV for the sequencer, plus run and email history in DuckDB.
An enriched lead is scored and segmented by rules, checked for sendability, then the LLM writes a sequence that code cleans, checks, rewrites email by email and has judged; leads with persistent spam words are dropped, and the rest get subjects, delays and signatures from code before export.
Receipts
- Before
- 4 campaign types, 4 duplicated runners
- Action
- One plugin runner; the LLM writes prose, code owns structure
- After
- New type = one folder + one register call
- Verified by
- Registry test pins exactly four types
- 3 rounds at most of per-email rewrites before the judge.
- 3 merge tags allowed in any copy. Code strips the rest.
- Pass thresholds on the judge's 0–100 spam score: below 30 for the strictest type and below 60 for the two longest.
For context, the outbound program as a whole reached a 17% reply rate (adjusted) and 200+ meetings booked in its best month. In one month, $104K+ of $250K new ARR came from campaign-sourced deals closed by AEs. These numbers are self-reported and belong to the whole funnel (enrichment, this engine, the sequencer, lead scoring and the sales team), not to this engine alone.
What broke, and what I changed
- Four runners became one. The old repo had a runner per campaign type. I wrote one runner plus a plugin contract (scorer, segmenter, narrator, deployer). Adding a fifth type means one new folder, one register call and its tests. The CLI and the local UI pick it up automatically.
- The spam scan flagged safe phrases. The scan matched fragments such as "top" inside "top 5" and "rate" inside "google rating". I added a safe-phrase pass that masks those phrases, longest first, before the keyword scan runs.
- The gate log lost lines. All campaign types wrote hard-gate entries to one shared file, and lines went missing when many leads wrote at once. Each type now has its own log file, appends are queued one at a time, and a test checks that concurrent writes lose no lines.
What I'd do next
- Close the loop. The campaign history table already has sent, opened, replied and bounced columns waiting to be filled from outbound events. Filling them would show reply rate by segment and by campaign type.
- Build an offline eval set for copy. Quality is checked per lead, at generation time, by the regex layers and the judge. A fixed set of leads with graded sequences would let me compare prompt and model changes before they ship.