Stop shipping retrieval changes on vibes

You've spent five posts making the system better. The honest question: better than what, by how much, on which queries? Without a number, every change is a vibe. With one, you can ship ef_search = 80 instead of 40 because you measured a six-point recall gain at a two-millisecond cost, and you can refuse to ship the reranker because it didn't move the needle on your queries even though it was supposed to.

This is the post that makes the rest of the series defensible.

Two recalls, not one

The single most common eval mistake is conflating two different things called "recall."

ANN recall measures the index, not the system. Given a query and a fixed exact answer (the true top-k from a brute-force scan), how many of those true top-k did the index return? This is what ef_search tunes. It needs no human labels. The ground truth is exact search.

Retrieval recall measures the system, not the index. Given a query and a labeled set of "relevant" chunks (humans say these answer the question), how many of the relevant chunks made it into the top-k? This needs labels.

You want both. They tell you different things. ANN recall at 99% on a chunker that misses the answer is a useless system. Retrieval recall at 90% on an index running at 50% ANN recall is a fragile system that's one ingestion away from regressing for reasons you can't trace.

ANN recall, with no human labels

This is the cheap, honest one. Bear with the SQL; it's worth knowing.

-- Disable the ANN index for this query, get the exact top-k.
SET LOCAL enable_indexscan = off;
SET LOCAL enable_bitmapscan = off;
 
SELECT id
FROM chunks
ORDER BY embedding <=> '[...]'
LIMIT 10;
 
RESET enable_indexscan;
RESET enable_bitmapscan;
 
-- Now the ANN top-k, with the index.
SELECT id
FROM chunks
ORDER BY embedding <=> '[...]'
LIMIT 10;

recall@k = |ann_topk ∩ exact_topk| / k.

Wrap it in a script and run it over a sample of queries. Real queries from logs are best; synthetically sampled vectors from the table are fine if you don't have query logs yet.

// scripts/ann-recall.ts
import { db } from "../src/db";
import { embed } from "../src/embed";
 
const queries = [
  "how do I recover from a failed concurrent index build",
  "what does ts_rank actually compute",
  // ... 30-50 of these is plenty
];
 
async function recallAt10(queryVector: number[]) {
  await db.execute(sql`SET LOCAL enable_indexscan = off`);
  await db.execute(sql`SET LOCAL enable_bitmapscan = off`);
  const exact = await db.execute<{ id: number }>(sql`
    SELECT id FROM chunks
    ORDER BY embedding <=> ${JSON.stringify(queryVector)}::vector
    LIMIT 10
  `);
  await db.execute(sql`RESET enable_indexscan`);
  await db.execute(sql`RESET enable_bitmapscan`);
 
  const ann = await db.execute<{ id: number }>(sql`
    SELECT id FROM chunks
    ORDER BY embedding <=> ${JSON.stringify(queryVector)}::vector
    LIMIT 10
  `);
 
  const exactIds = new Set(exact.map((r) => r.id));
  const hits = ann.filter((r) => exactIds.has(r.id)).length;
  return hits / 10;
}
 
const results: number[] = [];
for (const q of queries) {
  const [v] = await embed([[{ type: "text", text: q }]]);
  results.push(await recallAt10(v));
}
 
const mean = results.reduce((a, b) => a + b, 0) / results.length;
console.log(`ANN recall@10: ${(mean * 100).toFixed(1)}%`);

Run that at hnsw.ef_search = 40 (the default) and again at 100, 200, 400. You'll see the curve from Part 5 with your own data. That's the honest version of the animation from Part 1's "demo that lied."

Retrieval recall, the one that needs labels

ANN recall is intrinsic. The harder question: of the chunks a human would mark as relevant for this query, how many came back?

The minimum viable labeled set is 20 to 50 representative queries with hand-curated relevantIds. Build it once. It pays for itself by the second tuning decision.

// evals/cases.ts
export interface EvalCase {
  query: string;
  relevantIds: number[]; // chunk IDs that should appear in top-k
}
 
export const evalCases: EvalCase[] = [
  {
    query: "how do I recover from a failed concurrent index build",
    relevantIds: [42, 43, 178], // 178 is the screenshot of the error
  },
  // ...
];

The runner scores each case against the live search path, the same one your users hit:

// evals/run.ts
import { search } from "../src/search"; // hybrid + optional rerank from Part 6
import { evalCases } from "./cases";
 
async function recallAtK(k: number) {
  let total = 0;
  for (const c of evalCases) {
    const results = await search(c.query, k);
    const retrieved = new Set(results.map((r) => r.id));
    const hits = c.relevantIds.filter((id) => retrieved.has(id)).length;
    total += hits / c.relevantIds.length;
  }
  return total / evalCases.length;
}
 
console.log(`Retrieval recall@5: ${(await recallAtK(5) * 100).toFixed(1)}%`);
console.log(`Retrieval recall@20: ${(await recallAtK(20) * 100).toFixed(1)}%`);

Recall@5 is what your generation actually sees. Recall@20 tells you whether a reranker has anything to work with. If recall@20 is 0.95 but recall@5 is 0.62, your reranker has a job. If recall@20 is 0.65, the reranker can't help; you need better retrieval, not better rerank.

A/B everything, with one click

The whole point of an eval set is that you can re-run it after any change. Tuning ef_search, swapping chunker, turning the reranker on, switching embedding models: all of these are A/B comparisons against the same eval set.

A small harness keeps you honest:

// evals/ab.ts
async function abTest(label: string, fn: () => Promise<number>) {
  const v = await fn();
  console.log(`${label.padEnd(40)} ${(v * 100).toFixed(1)}%`);
}
 
await abTest("recall@5 ef_search=40", () => withEf(40, () => recallAtK(5)));
await abTest("recall@5 ef_search=100", () => withEf(100, () => recallAtK(5)));
await abTest("recall@5 + reranker", () => withRerank(() => recallAtK(5)));

Every line runs the same eval set under a different configuration, so the numbers are directly comparable.

Answer faithfulness

For RAG-style applications, retrieval being right isn't enough. The generated answer also has to use the retrieved context correctly.

The bare-minimum sanity check from Part 6 (does the answer cite something) is a starting point. A real faithfulness eval goes further: for each claim the answer makes, is it actually supported by the cited chunk?

The pragmatic version uses an LLM as a judge. It's not perfect, but it's cheap, repeatable, and surfaces real problems:

// evals/faithfulness.ts
async function isClaimSupported(claim: string, chunk: string) {
  const r = await llm.chat({
    messages: [
      {
        role: "user",
        content: `Is the following claim DIRECTLY SUPPORTED by the context?
Answer "yes" or "no" only.
 
Claim: ${claim}
Context: ${chunk}`,
      },
    ],
  });
  return r.content.trim().toLowerCase().startsWith("yes");
}

Split the generated answer into sentences (a regex on .!? is fine for a v1), pair each sentence with its cited chunk, and compute the fraction that come back "yes." That's a faithfulness score per answer; average across the eval set is your headline number.

Make it run in CI

An eval that runs once is a thought. An eval that runs on every PR is a guardrail.

The simplest version: a job that runs the eval set against a known-state database snapshot and fails the build if recall@5 drops by more than 2 points or faithfulness drops by more than 5. Tune the thresholds to your appetite for noise.

# .github/workflows/eval.yml (sketched)
name: retrieval eval
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker compose up -d db && ./scripts/seed.sh
      - run: pnpm tsx evals/run.ts --threshold-recall5 0.85 --threshold-faith 0.90

Two things this will catch that nothing else will. First, a change to the chunker that improves average score but tanks one query class (because your eval set has that class in it). Second, a model upgrade that looks great in isolation but degrades on your specific corpus.

Numbers to actually aim for

Everyone wants the headline. Here's the honest version of "what should my numbers look like" for a docs-style multimodal RAG system, the kind we're building:

• ANN recall@10: 0.95 or higher. Below 0.90, raise ef_search. If you can't get above 0.85 with ef_search cranked, your index is broken or too small.
• Retrieval recall@5: 0.85 or higher. This is the one the generation actually sees. The labeled set is what defines relevant for you.
• Retrieval recall@20: 0.95 or higher. If this is low, your reranker has nothing to work with; fix retrieval, not rerank.
• Reranker p95 latency: under 500 ms for 20 candidates of mixed text and images, on Voyage's rerank-multimodal-2. Above that, you're feeding it too many candidates.
• Faithfulness: 0.90 or higher (LLM-judge, sentence-level). Below 0.85 and your grounding instructions need work, or your context is too noisy.

These aren't laws. They're the band where a system stops feeling broken and starts feeling trustworthy. Your corpus may push them. Calibrate on your data, then defend the calibration with the eval set.

Re-chunk, re-embed, re-eval

A policy I wish more teams wrote down: when do you actually re-chunk or re-embed? Pick one of each and post it on the wall.

• Re-chunk when the document structure changes (new doc source, added headings, new figure conventions) or when an eval reveals systematically bad chunks. Re-chunking is a backfill; treat it like one.
• Re-embed when the model changes (Voyage ships a new version, you switch providers) or when your eval shows a recall ceiling you can't raise any other way. Re-embedding is Part 8's whole story; you do it carefully.
• Re-eval every time the corpus or pipeline changes. Two-line PR, full eval run, accept or revert.

The eval is what makes any of these reversible. Without it, every change is a guess.

The honest cost of an eval set

This is the post where I should be most honest about cost. Building a 50-query labeled set takes a few hours and rarely more. Building one with genuine coverage of your query distribution takes a few iterations, because the first version you write is biased toward what you would ask. The second one, built from user query logs, is much better.

The eval set is a living artifact. New features ship, query patterns shift, the labels rot. Plan for a quarterly review. The cost of letting it rot is much higher than the cost of maintaining it.

From the trenches

Eval on production-scale data. Tuning against a 10k-row table tells you almost nothing about behavior at 5M rows. Use a snapshot.

Don't average across query classes. A mean recall of 0.85 can hide a specific class (one-word queries, queries containing version numbers) that's at 0.30. Bucket your eval set and report per-bucket numbers.

The LLM judge has its own biases. It tends to over-mark sentence-by-sentence faithfulness because it can't always see the big-picture inference. Treat the score as comparative, not absolute. A drop is meaningful; an absolute value is mostly vibes.

Resist the urge to optimize the metric. If you tune ef_search until recall@5 hits 0.95 and your latency tripled, you've optimized the metric, not the product. The eval is a signal, not a goal.

Build the labeled set first

If you take one thing from this post, take the labeled set. Twenty queries with hand-picked relevant chunks is enough to start, and every tuning decision after that gets a number instead of a guess.

Part 8 is the last one. The system is good, fast, and measured. Now you have to operate it: re-embed when the model changes, watch for drift, keep search trustworthy over months.