kritwish.
  • Work
  • Systems
  • Playbook
  • Career

kritwish mondal, gtm engineer · diagrams sanitized: no customer data, no secrets

← Measured AI

Production

Knowledge layer with eval gates

  • Python
  • Postgres
  • LangGraph
  • Confluence

The problem

Product knowledge lived in 114 Confluence documents written for engineers. Reps and other agents needed answers from it. The dangerous failure for a retrieval system is not a crash. It is a confident paragraph built from the wrong page, or an answer to a question the docs never cover.

The obvious fix, a similarity cutoff, does not work here. I measured it on the live corpus: the weakest question that did have an answer scored 0.691 cosine similarity, and the strongest question with no answer scored 0.682. The ranges overlap, so any cutoff would either refuse real answers or invent fake ones.

Constraints

  • Managed Postgres. The BM25 search extension isn't available, and the vector index (HNSW) only goes up to 2,000 dimensions, so embeddings run at 1,536.
  • CI has no model key. In CI the vector search runs on an offline word-hashing stand-in. So CI tests the plumbing, not whether the search understands meaning.
  • Who may see what. I split it into three surfaces: outbound (customer-facing), internal and research. A caller that doesn't say which one it is gets the strictest.
  • The corpus is engineering prose. endpoint appears in 114 of 114 documents.

How it works

  1. Sync. Only Confluence pages with a platform label get indexed. Change checks skip unchanged pages, so a second full sync of an unchanged space fetched 0 page bodies.
  2. Chunk. Pages split on headings, tiny sections merge into the next one, and tables repeat their header row on every piece. 1,772 chunks in all.
  3. Search three ways. A stemmed keyword search, an exact keyword search (so acronyms stay exact) and a vector search (gemini-embedding-2). The results are fused by rank (reciprocal rank fusion), not by blending scores.
  4. Return a verdict. hit, weak, miss or unavailable, based on grounding (how many of the question's content words show up in the top passages) and corroboration (whether two independent signals agree).
  5. The retry rule lives in code. miss goes straight to the answer step, every time. weak gets one reworded search. unavailable says the search failed, never "not in the docs".
  6. Answer. On a miss the model is never called. The agent says "I don't have that" and shows what is indexed, built from data. On a hit, a pinned model (gpt-5.6-luna) answers with numbered citations.

A question goes to three search legs whose results are fused and given a verdict; a hit is answered with citations, a weak verdict gets one reworded search, and a miss returns 'I don't know' with coverage, without calling the model.

A miss ends the loop instead of starting a guess

Receipts

Before
No similarity cutoff separates answerable from absent questions
Action
Verdict from grounding and agreement; a miss always ends the loop
After
Live corpus: 7 of 8 answerable hit, 5 of 5 absent miss
Verified by
Live run over 114 documents

The headline numbers come from a fixture eval set, not production traffic: 15 fictional documents and 24 retrieval cases (12 literal, 2 paraphrased, 3 with no answer, 7 that must never come back on a given surface).

  • On the blocking tiers: recall@5 100%, rank-1 93%, MRR 0.967, false-confident 0%, leaks 0.
  • Including the two exploratory paraphrase cases: rank-1 82%, false-miss 12%. The offline embedder matches words, not meaning, so those two can't pass in CI by design.
  • The merge gate: critical cases must pass 100%, standard 98%, and exploratory cases are tracked but never block. On top of that, floors: recall@5 ≥ 95%, rank-1 ≥ 85%, MRR ≥ 0.90, and false-confident = 0%, which is not negotiable. It runs on every push in about a second.
  • Eight agent scenarios use a scripted model to check behaviour. A miss, for example, may use at most 2 tool calls.
  • Eight sabotage tests prove each assertion type actually fails when it is broken. If any case returns unavailable, the whole run is thrown out.

What 100% does not mean

Ranking 15 documents is easier than ranking thousands, and I wrote the golden set myself. The 100% shows the code paths work. It doesn't predict accuracy in production.

What broke, and what I changed

  • Paging that never ended. Confluence's search ignores the offset parameter, so the first sync got the same 100 results 57 times before Confluence refused. I switched to its cursor and added a hard stop that fails loudly.
  • weak meant nothing. The first rule counted any word overlap as a signal, so every question with no answer came back weak and triggered a pointless second search. weak now requires grounding, with one exception for close paraphrases.
  • Neither bug showed up in tests. The full test suite was green for both. Running against real data found them.
  • RFP wording. Four of twelve RFP-style questions missed even though the right page ranked first. Words like "describe" and "explain" diluted the grounding score. Stripping those words took the set from 3 of 12 answerable to 5 of 12, with false-confident still at 0%.
  • Random local rankings. The old dev stub returned zero vectors, and cosine similarity on a zero vector is undefined, so every local search came out in random order. I replaced it with the word-hashing stand-in.

Honest negative

The grounding threshold is too strict for this corpus. A sweep showed it refuses 3 of 10 answerable questions. Lowering it far enough to fix that let 1 of 5 absent questions through, so I kept it strict for now.

What I'd do next

  • Recalibrate the grounding threshold and the similarity thresholds against the real corpus. The similarity thresholds were inherited from a different embedding model.
  • Weight grounding by question length, because a three-word question can only score 0, 0.33, 0.67 or 1.
  • Build the staleness sweep and the weekly change digest. Both are specified but not built.
  • Run a live agent with a judge model. The scripted runner can test wiring but not judgement.
  • Build curation, the step that promotes synced pages into human-reviewed articles. Until then the outbound surface returns nothing on purpose.
  • Add a reranker only if rank-1 falls clearly behind recall@5 and starts causing false misses. Today it isn't.

Want systems like these at your org?

Message me on LinkedIn