work / measured-ai

Pillar 03 · Measured AI

AI that shows its work

Every AI feature here was measured before it was trusted: how often it's wrong, what it costs, and what happens when it doesn't know.

  • Claude
  • Gemini
  • Langfuse
  • LangGraph
  • Next.js
  • n8n

Why this had to be built

LLM output was about to flow into the CRM, and nobody could say how often it would be wrong or what it would cost.

Production

Call intelligence with a blind judge

  • Gong
  • Claude
  • Gemini
  • Langfuse
  • Postgres
the deep dive →

What it is

Every client call is turned into structured intelligence: 75 entities, each backed by a verbatim quote that code checks against the transcript.

What it did

  • Blind A/B judge on 30 calls, both answer orders (60 runs).
  • Opus 5.5 vs Gemini 3.8 Flash: 0 made-up values in 1,898 entities vs 20 in 1,349; grounded precision 85.0% vs 50.8%; recall ~96% vs ~70%.
  • The judge was checked too: 84.4% verdict agreement when answer order was swapped.
  • An honest negative: a vendor's confidence score barely predicted correctness (AUC 0.58–0.65), so I stopped relying on it.
  • About $0.77 per call.
Runner-up
20 made-up values in 1,349 · 50.8% precision
Method
Blind A/B LLM judge, 30 calls × 2 orders
Chosen
0 made-up values in 1,898 · 85.0% precision
Judge check
84.4% agreement on order swap
Production

Knowledge layer with eval gates

  • Python
  • Postgres
  • LangGraph
  • Confluence
the deep dive →

What it is

A retrieval agent over internal docs that says “I don't know” instead of guessing.

What it did

  • 114 docs, 1,772 chunks.
  • On the eval set: recall@5 100%, rank-1 93%, MRR 0.967, 0% false-confident.
  • Eval tiers gate every merge: critical 100%, standard 98%.
  • Questions with no answer in the docs return a miss: 5 of 5 on the live corpus.
Production

Marketing ledger

  • Next.js
  • Vercel
  • n8n
  • Google Sheets
  • Salesforce
the deep dive →

What it is

The dashboard marketing uses to see where pipeline came from. I took over a colleague's first version, audited it, and rewrote nearly all of it.

What it did

  • A 5-route linker ties every opportunity to a source person. Unlinked opps went from 49 of 105 to 1 of 104.
  • An attribution tab splits each opp's value evenly across its people's lead sources; it matched the analysis sheet on 1,847 rows across 154 opps with 0 differences.
  • Inbound SLA tracking and monthly pipeline cohorts against the 6-month average and plan.
  • Daily Salesforce sync through n8n that checks counts before writing; every merge to main passes self-checks before it deploys.
Next pillar →Clean Data

Want systems like these at your org?

Message me on LinkedIn