Every result scores 0.81. Your search is broken.

You ran the query from Part 2 on a few hundred rows. Half the results feel right, the rest feel wrong-but-similar, and you can't quite say why. That's normal at this stage, and it has specific causes. Relevance isn't binary, the model doesn't tell you when it's confused, and the only way out is measurement plus a few specific knobs.

This post is those knobs.

Three places relevance breaks

Before we touch anything, name the failure modes. There are basically three.

1. Chunking. The unit you embedded doesn't match the unit a user is asking about. The paragraph they want is buried inside a 1200-token megablock, or sliced across two chunks, or the chunk has so much surrounding unrelated text that the embedding got dragged toward the noise.

2. Modality mismatch. Your query is a sentence; the answer is a screenshot. Or the other way around. The multimodal model puts them in the same space, but proximity isn't symmetric in practice and you have to verify that with your own data.

3. The model is confused. It returns ten things that all sit at 0.81 similarity. They're not actually similar to the query, they're similar to each other, and the query happened to land near that cluster.

Each of those has a specific debug move. The rest of the post is them, in order.

Chunking: get the unit right

There is no universally correct chunk size. There are good defaults and there's an eval that tells you whether your default is good for your corpus.

The default I reach for first: 512 tokens, 50-token overlap, with chunk boundaries that respect paragraph breaks when possible. This is the same shape I'd use for general prose RAG; I went into it in more depth in the RAG-with-Postgres post.

// src/chunk.ts
import { encoding_for_model } from "tiktoken";
 
interface Chunk {
  body: string;
  chunkIndex: number;
}
 
const enc = encoding_for_model("text-embedding-3-small"); // good-enough tokenizer
 
export function chunkText(
  text: string,
  size = 512,
  overlap = 50,
): Chunk[] {
  const tokens = enc.encode(text);
  const chunks: Chunk[] = [];
  let i = 0;
 
  while (i < tokens.length) {
    const slice = tokens.slice(i, i + size);
    chunks.push({
      body: new TextDecoder().decode(enc.decode(slice)),
      chunkIndex: chunks.length,
    });
    i += size - overlap;
  }
 
  return chunks;
}

Use this as the starting point, then drag it on a real document and see which retrieved spans actually answer the query.

A few things you'll notice as you drag.

Too small (128 tokens, no overlap) and you lose the surrounding sentence that gave a phrase its meaning. The model is local, and "concurrent" alone is a different vector than "create index concurrently."

Too big (1500 tokens) and you blur. The chunk now covers three subtopics, and the embedding lands somewhere in the middle of them. The query lands near one subtopic, but the chunk's embedding is averaged across all three, and the nearest-neighbor score drops.

The sweet spot for prose is usually 400 to 700 tokens. For self-contained units (FAQ answers, error messages, function signatures), drop to 200 to 400. For long technical docs, push to 800 to 1000. The eval in Part 7 will tell you the right answer for your corpus; the numbers above just keep you in the right neighborhood while you build it.

A common community finding worth naming: simple sliding-window chunking (500 to 1000 tokens, 100 to 200 overlap) usually beats the fancy semantic splitters people reach for first. The win isn't in the splitter; it's in preserving document structure (headings, code blocks, list items) and adding the right context around each chunk. Which brings us to metadata.

Metadata is the lever you're not pulling

You can tune ef_search, swap embedding models, rewrite your chunker, and squeeze a few points of recall out. Or you can pre-pend each chunk with three lines of metadata and jump a tier.

// Before: naked chunk
"CREATE INDEX CONCURRENTLY builds the index without holding..."
 
// After: chunk with a metadata header
`Source: Postgres 16 docs / CREATE INDEX
Section: Concurrent index builds
Heading: Recovery from failure
 
CREATE INDEX CONCURRENTLY builds the index without holding...`

Same vector model, same chunk size, much better neighborhoods. Why: the embedding now lands near other chunks about concurrent index builds instead of near every chunk that happens to mention CREATE. The metadata is doing the work of disambiguation that the body alone can't.

The fancier version of this idea is Anthropic's Contextual Retrieval, where each chunk is prefixed with a one-sentence summary of where it sits in the document (generated once, at ingest, by a small LLM call). The hand-written version above gets you most of the benefit if your docs have useful structure to lift from. The LLM-generated version is the move when they don't.

Either way, metadata is the one part of relevance you can tune without re-embedding anything.

Multimodal: when the right answer is a screenshot

The promise of multimodal is "text query finds the image." The reality is messier. Same query, different modality, different ranking:

Two practical observations from running this on real corpora.

First, image embeddings are noisier than text embeddings, on average. This isn't a Voyage thing, it's true for every multimodal model I've tried. The implication: your top-1 might be a text chunk even when the best answer is a screenshot, because the text chunk's embedding sits more tightly on the query.

The fix isn't to weight modalities (resist this urge until Part 7 says you have to). The fix is to retrieve more, then rerank. Pull top-20, let the reranker in Part 6 do the work. We're laying the groundwork here.

Second, the screenshot needs context to embed well. A picture of a psql terminal showing an error embeds better if you ingest the surrounding caption with it. The multimodal model can mix text and image in a single input:

// One row, one vector, two modalities in the input.
const [vec] = await embed([
  [
    { type: "text", text: "psql output for CREATE INDEX CONCURRENTLY failure" },
    { type: "image_url", image_url: "https://.../screenshot.png" },
  ],
]);

That vector now lives somewhere between the caption's text neighborhood and the image's visual neighborhood. For docs with figures and captions, this is the move. For raw images with no context, ingest them alone and accept the noisier ranking until reranking earns its place.

The 0.81 problem

You'll see this within your first hundred queries. The top-10 all sit at similarity 0.78 to 0.83, the right answer isn't in there, and you can't tell from the scores that anything's wrong.

This is the single most useful diagnostic in vector search:

When the spread is tight (everything between 0.78 and 0.83), it almost always means one of two things. Either your query is too generic and many chunks are equally close, or your chunking is too uniform and every chunk's embedding looks like every other chunk's. Both are fixable; the fix is different.

For the generic-query case, you can't fix it inside retrieval. You either rewrite the query (often by feeding it through a small LLM that adds specificity), or you let hybrid search rescue you in Part 6 because the keyword side cares about specific terms in a way the vector side doesn't.

For the uniform-chunks case, you fix the chunker. Variable-length chunks that respect document structure (headings, code blocks, list items) produce embeddings with more spread, which means clearer separation between the near answers and the also-rans.

A quick way to spot the problem in code: log the histogram of top-10 similarities on a sample of queries.

import { search } from "./search";
 
const queries = ["create index concurrently failure", /* ... */];
 
for (const q of queries) {
  const results = await search(q, 10);
  const scores = results.map((r) => r.similarity);
  const min = Math.min(...scores);
  const max = Math.max(...scores);
  console.log(`${q}: spread ${(max - min).toFixed(3)}, top ${max.toFixed(3)}`);
}

If your spread is consistently under 0.05, you have the 0.81 problem.

A debugging workflow you'll actually use

Most relevance bugs in production look like this: a user reports a bad result, you re-run the query, the result is still bad, you stare at it. The loop below is the one I use, in the order I use it. It's not glamorous; it just works.

1. Eyeball the top-10 raw. Print the body (or image URL) of every returned row with its similarity score. Half the time the bug is obvious (your chunker put the whole document in one chunk; the screenshot you expected isn't in the table).

2. Check what the model would consider the ground truth. Take the known-correct answer text, embed it, and compute its similarity to the query. If that similarity is 0.95 but it wasn't in your top-10, you have a retrieval bug. If it's 0.71, the model genuinely doesn't think they're that similar and the bug is upstream (chunking, modality, or the query itself).

3. Look at neighbors of the known-correct answer. Run the query "find rows like this one" against the known-correct row. If its true semantic neighbors look right, your space is healthy and the query is the issue. If its neighbors look random, your chunking or your model choice is the issue.

4. Only then start changing settings. Chunk size, overlap, modality weights, retrieval depth, anything. With the first three steps you'll usually know what to change before you touch it.

From the trenches

Don't normalize by hand. Voyage returns L2-normalized vectors. Cosine distance on normalized vectors is monotonically related to L2 distance, which is monotonically related to dot product. The operators converge. This is why "they all give the same ranking" is true in practice: the embeddings are normalized. If you ever write code that re-normalizes Voyage output you're doing extra work for nothing.

Truncation is sneaky. Voyage's API silently truncates inputs over the token limit unless you pass truncation: false. For long doc chunks this is fine. For prompts you're carefully constructing for reranking later, you might want it to error so you find out. Pick on purpose.

Most "relevance bugs" are missing rows, not bad rankings. Before you spend an afternoon tuning, confirm the answer is actually in the table. The number of times I've debugged a ranking and the right chunk just wasn't ingested... too many.

Diagnose before you tune

My recommendation: run the four-step workflow on ten real queries before you change a single setting. It will tell you which of the three failure modes you actually have, and the fix for each one is in this post.

Speed comes next. The table is going to grow past the point where an exact scan is fine, and you'll need an index. Part 4 is choosing it.