Call intelligence with a blind judge
- Gong
- Claude
- Gemini
- Langfuse
- Postgres
- Runner-up
- 20 made-up values in 1,349 · 50.8% precision
- Method
- Blind A/B LLM judge, 30 calls × 2 orders
- Chosen
- 0 made-up values in 1,898 · 85.0% precision
- Judge check
- 84.4% agreement on order swap
The problem
Client calls hold the facts sales and customer success need: pain points, budget, competitors, provider volume, open issues. At a healthcare credentialing company selling to health plans, those facts lived only in call recordings. Nobody had time to write them down.
The goal: after every client call, fill in a fixed form of 75 entities in four blocks (deal, account qualification, customer delivery, product and meeting). 13 of them use controlled pick lists. A teammate wrote the entity catalogue. I built the pipeline, the grounding check, the blind judge and its scoring.
The hard part was not getting a model to fill the form. It was proving the form was true before anything reached a rep.
Constraints
- A wrong fact costs more than a missing one. A made-up competitor written to the CRM misleads a rep for months. So my scoring docks a made-up value 1.5 points and gives a correct one 1.0.
- Most calls aren't client calls. Of 977 synced calls, 676 were client-facing. Internal stand-ups, vendor calls and podcasts must never reach the model.
- No human gold labels. The plan called for a hand-labelled set; 0 of 200 values were ever labelled. Every quality number below is an LLM-judge result, not a human-verified one.
- Cost and time per call. The model I chose costs about $0.77 and 4.5 minutes per call, against $0.04 and 71 seconds for the cheaper candidate. The gap had to be worth it.
How it works
A synced call is checked by one SQL rule for whether it is client-facing. Client calls go to the extraction model, every value's quote is checked against the transcript in code, and only grounded values are stored. CRM write-back is the next phase.
- One rule decides eligibility. A single SQL classification marks a call client-facing. The same rule feeds the Gong drift digest, so the two can't disagree.
- One model call per call. The model reads the transcript, speakers and CRM context. It returns every value with a verbatim quote, who said it, and a confidence.
- Code checks every quote. Every quote must appear in the transcript after whitespace and case are normalised. Values that fail are dropped and counted. This is the one hallucination check that can't itself hallucinate.
- Each result is stored once. Each call's result is one row in Postgres, tagged with the prompt version and model.
Picking the model with a blind judge
A first pass on the same 30 calls (1,111 minutes of audio) showed Opus 5.5 finding 67 values per call to Gemini 3.8 Flash's 36. The grounding check dropped 1 of 2,012 Opus values and 30 of 1,081 Gemini values. Volume isn't quality, though, so I built a judge.
Both models' answers are blinded as A and B, judged twice with the order swapped, given one fixed verdict per entity, then scored and un-blinded by a script.
- Blind. The judge sees "A" and "B", never model names. This matters because the judge is Opus 5.5 itself.
- Swapped. Every call is judged twice with A and B flipped, so 60 runs in all. A judge that always prefers the first answer gets caught.
- One verdict per entity. The judge never says "A is better." It gives each entity one fixed verdict (correct, partly correct, speculative, made up, and so on) with the transcript line it checked. A script adds up the verdicts afterwards.
Receipts
- Gemini 3.8 Flash
- 1.5% made up · 6.4% speculative · ~70% recall
- Judge trust
- 84.4% same verdict on swap · 19.0% flip · slot A 51.9%
- Opus 5.5
- 0.0% made up · 1.7% speculative · ~96% recall
- Verified by
- LLM judge only; 0 of 200 human labels
- Is the judge trustworthy? My bar was over 80% verdict agreement across orders and under 25% contested flips. It came in at 84.4% and 19.0%. Slot A won 51.9%, close to the 50% a fair judge should show.
- Net quality: 0.86 vs 0.49 per entity, on my weighted score.
- Automatic analysis went live for new calls only from 6 October 2026. At 15 to 20 client calls a week, it costs about $12 to $15 a week.
- A later challenger round (14 October) ran two newer models through the same judge. The most precise one reached 87.8% grounded precision but only 78.0% recall. Opus stayed.
What broke, and what I changed
- The first model returned nothing. Sonnet 5 thinks by default and bills thinking against the output limit. It spent the whole 16k budget thinking and returned no text. I added a setting that maps reasoning effort onto the provider's thinking controls.
- The judge's first smoke run marked every pick-list entity as a rule break. I had fed it a spreadsheet export that dropped the normalised codes. I switched the input to the raw traced model answers, which also meant no re-extraction spend.
- Laptops analysed production's calls. On 16 October, two local workers ran the nightly catch-up against their own databases. They sent 22 calls to the model for $9.54. Automatic analysis now never runs locally or in CI.
What I'd do next
- Build the golden set: 5 to 10 transcripts hand-labelled across call types, so the numbers stop depending on an LLM judge.
- Add a human check of contested verdicts before the results go into a decision document.
- Phase two: a versioned per-account summary rebuilt after each call, then copying its high- and medium-confidence values to read-only Salesforce fields, with HubSpot syncing a subset.