Embeddings rot too. Running pgvector in production.

This is the last post in the series. The first seven were about making the system good. This one is about keeping it good. Most of the failure modes here are slow, silent, and specific to vector search. Embedding models ship new versions. Your data distribution shifts. The HNSW index quietly bloats from churn. Recall drifts down two points a month and nobody notices until a user complains.

What this post buys you: a re-embedding migration you can run during business hours, a recall-drift monitor that fires before your users feel it, and the small handful of maintenance habits that prevent the long tail of weird incidents.

Re-embedding without downtime

This is the operation people most want to avoid and most often have to do. Voyage ships a better model. You want to use it. Every vector in your table is in the wrong space. Naively, you take a maintenance window, drop the column, re-embed everything, and pray.

You don't have to.

The pattern is add a shadow column, backfill behind dual writes, verify, cut over read traffic, then drop the old column. Four phases, all online.

Phase 1: shadow column

Start by adding a column for the new model's vectors:

ALTER TABLE chunks
  ADD COLUMN embedding_v2 vector(1024);

No traffic uses this column yet. Adding a nullable column is fast and non-blocking.

Phase 2: dual write

Update your ingestion path to write both columns on insert:

const [vecV1, vecV2] = await Promise.all([
  embedV1([...]),
  embedV2([...]),
]);
 
await db.insert(chunks).values({
  // ... other fields
  embedding: vecV1[0],
  embedding_v2: vecV2[0],
});

From this point on, new rows have both vectors. Old rows have only the v1 vector. That's the gap the backfill fills.

Phase 3: backfill

Run a script that walks the table in batches, embeds the body (and image URL, where applicable) with the new model, and updates the shadow column. Throttle to whatever your Voyage account and your database can take.

// scripts/backfill-v2.ts
const BATCH = 64;
 
while (true) {
  const rows = await db
    .select({ id: chunks.id, body: chunks.body, metadata: chunks.metadata })
    .from(chunks)
    .where(isNull(chunks.embedding_v2))
    .limit(BATCH);
 
  if (rows.length === 0) break;
 
  const pieces = rows.map((r) =>
    r.metadata?.kind === "image"
      ? [{ type: "image_url" as const, image_url: r.metadata.url! }]
      : [{ type: "text" as const, text: r.body! }],
  );
 
  const vectors = await embedV2(pieces);
 
  await db.transaction(async (tx) => {
    for (let i = 0; i < rows.length; i++) {
      await tx
        .update(chunks)
        .set({ embedding_v2: vectors[i] })
        .where(eq(chunks.id, rows[i].id));
    }
  });
 
  console.log(`Backfilled ${rows.length}, sleeping...`);
  await new Promise((r) => setTimeout(r, 200)); // be a polite neighbor
}

Wrap it in a process supervisor so it survives restarts. Resume is implicit: the isNull filter only picks up un-embedded rows.

Phase 4: build the new index, verify, cut over

While the backfill runs, you can build the new index CONCURRENTLY. It will be incomplete until backfill finishes; that's fine, the live read path is still on embedding.

Once backfill is complete and embedding_v2 is non-null on every row, run the eval set from Part 7 against both columns. If the v2 results are meaningfully better (or at least not worse), cut over read traffic with a feature flag.

After a soak period (a week or two of live traffic, no regressions), drop the old column and its index:

DROP INDEX CONCURRENTLY chunks_embedding_hnsw_idx;
ALTER TABLE chunks DROP COLUMN embedding;
ALTER TABLE chunks RENAME COLUMN embedding_v2 TO embedding;

Zero user-visible downtime, the whole way through. The eval set is what gives you the confidence to cut over.

Recall drift, the silent failure

The slowest, ugliest failure mode in production vector search is recall drift. Your eval scores were 0.88 at launch. Today they're 0.82 and nobody noticed because nobody re-ran the eval.

Drift comes from a few specific sources.

1. Corpus drift. New documents whose distribution doesn't match the old. Common when you start ingesting a new source type without re-tuning chunking.

2. Query drift. Users started asking different things. Your eval set is now unrepresentative.

3. Index degradation. With high write/delete churn, HNSW graphs develop dead branches. Vacuum and incremental updates handle this most of the time; sometimes a periodic REINDEX CONCURRENTLY is the cleanest fix.

4. Silent dependency change. The embedding model provider deployed a new variant of the "same" model. Their docs say nothing changed; your recall says otherwise.

The mitigation is the same in every case: re-run the eval on a schedule, alert on a drop.

# .github/workflows/eval-nightly.yml
name: nightly retrieval eval
on:
  schedule:
    - cron: "0 6 * * *"
jobs:
  eval:
    # ... run the same eval as the PR job
    # On a drop of >3 points from the 7-day moving average, alert.

Plot the number over time. Alert when it crosses a threshold. The threshold doesn't have to be perfect; what matters is that there is one, and that it fires before users do.

Maintenance that's worth doing

A short list, in declining order of how often it matters.

VACUUM your vector table. Routine autovacuum is fine, but on a table with heavy deletes the HNSW index can accumulate tombstones. After a big delete event, run VACUUM (ANALYZE) chunks; and watch the index size drop.

Watch index bloat. SELECT pg_size_pretty(pg_relation_size('chunks_embedding_hnsw_idx')). If it's 2x what it should be for your row count, a REINDEX CONCURRENTLY will tighten it.

Keep effective_cache_size honest. As you add RAM, the planner needs to know. Get into the habit of revisiting this when the box changes.

Snapshot the eval set with the production DB. A backup of the DB without the eval set is missing a critical piece. They're co-located in spirit; treat them as one artifact.

Capacity, with a number

"How many vectors can pgvector handle" is the wrong question. The right ones are concrete.

Memory. HNSW index size in bytes is approximately rows × (dim × 4 + 4 × m × 2 + overhead). For 5M rows at 1024 dim and m = 16, that's ~25 GB. You want this in shared_buffers or the OS page cache. If your box has 64 GB, you're fine. If it has 16 GB, you're paging.

Build time. A 1M-row build at default m/ef_construction with parallel workers and decent maintenance_work_mem takes ~10 minutes on a modest box. Scale roughly linearly.

Query latency at scale. With the index in memory and ef_search around 60, p50 stays under 10ms up through ~10M rows. p99 is the honest number to track; it's the one that drifts with cache pressure.

If you're heading into the 50M-vector range with a sub-10ms p99 budget, that's the territory where a purpose-built vector database starts genuinely earning its keep. Below that, every story about pgvector "not scaling" I've investigated has been the team missing one of the three knobs from Part 5. Every one.

One stop worth knowing about before you escalate: pgvectorscale, Timescale's open-source companion extension. It layers a different graph index (StreamingDiskANN) on top of pgvector and moves query performance closer to the dedicated-DB band while you stay in Postgres. If you've outgrown vanilla pgvector at 10-50M vectors but the rest of your stack is happily Postgres, pgvectorscale is the next move before "buy a vector database" is. The series doesn't run on it, but it's the honest answer to what sits between pgvector and Pinecone.

From the trenches

The migration that bit me hardest. Voyage shipped a new model variant; I assumed it was backwards-compatible (same dims, same model family) and started using it on new rows without re-embedding old ones. The space had shifted just enough that mixed-vintage results made the top-5 look incoherent. The fix was a backfill, then a re-eval. The rule: any model change is a re-embed event, even when the dims match.

Backfills are also a corpus refresh. While you're re-embedding, it's the cheapest moment to fix bad chunks, drop dead pages, or re-normalize images. Build the backfill script with hooks for these because you'll want them.

Image URLs go stale. If you stored image_url and the URLs change over time, your image rows quietly become un-renderable. Either store the asset in your own object storage and reference that, or treat image rows as ephemeral and rebuild from source.

Postgres restarts cool the cache. A planned restart of a busy vector-search DB will produce a few minutes of degraded p99 while the working set warms back up. Either run a small warm-up script after restart, or schedule restarts when traffic is naturally lighter.

Where this leaves you

The operational loop is small. Treat any model change as a re-embed event and run it behind a shadow column. Re-run the eval on a schedule and alert on a drop. Vacuum after big deletes and keep the index in memory. Do that and the system stays as good as the day you shipped it.

That's the series.

You started with a similarity query that "worked," and you're ending with a multimodal search system you can tune, measure, operate, and trust. No new vendor. No second datastore. One Postgres you understand from the index up. That's the version of vector search I want more teams to have, and now you've built it.