By the end of this post, the project has a heartbeat: a Postgres you self-host, a Drizzle schema, a Voyage call that takes both text and an image, a table with a few hundred rows of real Postgres docs and screenshots, and one query that returns the right thing.
That sounds small. The reason it's a whole post is that almost every later decision (indexing, tuning, evals, re-embedding) is shaped by what you pick in the next twenty minutes. The dimension on your vector column, the extension version, the build flags on Postgres, the way you batch embedding calls. None of these throw if you get them wrong. They just make Part 5 miserable.
So we set up like the next six posts exist.
The project is a multimodal docs assistant. The corpus is Postgres' own documentation pages plus a curated set of screenshots (psql output, EXPLAIN plans, pgAdmin views). Real prose, real images, the kind of thing your users would actually search.
We're going to want this assistant to answer questions like "how do I
recover from a failed CREATE INDEX CONCURRENTLY?" and have it pull the
right doc paragraph and the screenshot of the relevant error message. Same
query, one index, both modalities.
Self-hosted is a deliberate choice. We need to see the build flags, the
config knobs, and what changing shared_buffers actually does. You can't
see those on a managed Postgres without indirection, and Parts 4 and 5
need them in the foreground. If you eventually run on Supabase or RDS, the
intuition transfers. The opposite direction (start managed, learn to tune)
mostly doesn't.
Use the official pgvector image so the extension is already in the binary:
# docker-compose.yml
services:
db:
image: pgvector/pgvector:pg16
environment:
POSTGRES_PASSWORD: dev
POSTGRES_DB: docs_assistant
ports:
Occasional notes on software, tools, and things I learn. No spam.
Unsubscribe anytime.
Save that as docker-compose.yml and start the container:
docker compose up -dThen enable the extension once. Drizzle Kit won't do this for you (and that quietly bites people on a fresh database):
CREATE EXTENSION IF NOT EXISTS vector;If this is your first time wiring pgvector into Drizzle, I wrote the
setup walkthrough
separately. The piece below assumes you've got drizzle-orm, drizzle-kit,
and postgres (or pg) installed.
One table for now. Splitting "documents" and "chunks" gets pulled out in Part 3 when chunking earns it; right now a single table keeps the cursor on the parts that matter.
// src/db/schema.ts
import {
pgTable,
serial,
text,
jsonb,
vector,
index,
} from "drizzle-orm/pg-core";
export const chunks = pgTable(
"chunks",
{
id: serial("id").primaryKey(),
source: text("source").notNull(),
body: text("body"),
// voyage-multimodal-3.5 returns 1024-dimensional vectors by default.
// You can request 256, 512, or 2048 instead. Whatever you pick, every
// row has to agree on it.
embedding: vector("embedding", { dimensions: 1024 }),
metadata: jsonb("metadata").$type<{
kind: "text" | "image";
url?: string;
pageId?: string;
chunkIndex?: number;
}>(),
},
(t) => [
index("chunks_embedding_hnsw_idx").using(
"hnsw",
t.embedding.op("vector_cosine_ops"),
),
],
);A few notes worth keeping in your head, because they all get cashed in later:
1. dimensions: 1024 is a one-way door. Every row has to agree on it.
You can re-embed the whole table, sure, but that's Part 8 with a careful
zero-downtime story. Pick on purpose.
2. The HNSW index uses vector_cosine_ops. That has to match the
operator you query with (<=>). If you query with <-> instead, the index
goes unused and you'll wonder why everything is slow. I went deeper on
what each pgvector operator actually means
in a separate post.
3. One column, both modalities. body is nullable because image rows
don't have text content. The metadata.kind field tells you which it is
when reading back results.
Run drizzle-kit generate and drizzle-kit migrate and you have the table.
Don't forget the CREATE EXTENSION above the first migration if this is a
brand new database.
Voyage's multimodal endpoint takes a list of inputs, each input is a list of content pieces, and each piece is either text or an image reference. The boring shape is on purpose: text and image go through the same call, and each input returns one vector in the same coordinate space.
There's an SDK but I prefer fetch for code that's going to be in front of
people. It's verifiable against the docs without grep:
// src/embed.ts
type VoyagePiece =
| { type: "text"; text: string }
| { type: "image_url"; image_url: string }
| { type: "image_base64"; image_base64: string };
export async function embed(
inputs: VoyagePiece[][],
): Promise<number[][]> {
const res = await fetch(
"https://api.voyageai.com/v1/multimodalembeddings",
{
method: "POST",
headers: {
Authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "voyage-multimodal-3.5",
input_type: "document",
inputs: inputs.map((content) => ({ content })),
}),
},
);
if (!res.ok) {
throw new Error(`Voyage ${res.status}: ${await res.text()}`);
}
const json = (await res.json()) as { data: { embedding: number[] }[] };
return json.data.map((d) => d.embedding);
}Two things worth pulling out of that. input_type: "document" tells Voyage
this is corpus content (not a query). When you embed a user's query at
search time, switch it to "query". The model treats them slightly
differently and using the right one helps relevance.
Second, image_url is a public URL Voyage's servers can fetch. If your
screenshots aren't public you can send image_base64 instead and skip the
URL altogether. We'll use both in this series; the URL version is cheaper
for big libraries, the base64 version is what you reach for during
development.
Here's the smallest honest ingestion loop. Two text chunks and one
screenshot, all hitting the same embed call, all landing in the same
table.
The text comes from the CREATE INDEX
page in the
PostgreSQL manual, specifically its "Building Indexes Concurrently"
section. The screenshot is the error-output panel from the pgAdmin query
tool
docs. Real
sources, so you can check a retrieved chunk against the page it came from
instead of trusting a paraphrase I wrote. Note the sourceUrl on every
record: you want it from day one, because the moment retrieval returns
something surprising the first question is always "where did this come
from?"
// src/ingest.ts
import { db } from "./db";
import { chunks } from "./db/schema";
import { embed } from "./embed";
// postgresql.org/docs/current/sql-createindex.html — "Building Indexes
// Concurrently". Excerpted verbatim so a retrieved chunk can be checked
// against the published page.
const CREATE_INDEX_URL =
"https://www.postgresql.org/docs/current/sql-createindex.html";
const examples = [
{
kind: "text" as const,
source: "postgres-docs",
sourceUrl: CREATE_INDEX_URL,
body:
"PostgreSQL will build the index without taking any locks that prevent " +
"concurrent inserts, updates, or deletes on the table; whereas a " +
"standard index build locks out writes (but not reads) on the table " +
"until it's done.",
metadata: { pageId: "sql-createindex", section: "concurrently", chunkIndex: 0 },
},
{
kind: "text" as const,
source: "postgres-docs",
sourceUrl: CREATE_INDEX_URL,
body:
"If a problem arises while scanning the table, such as a deadlock or a " +
"uniqueness violation in a unique index, the CREATE INDEX command will " +
"fail but leave behind an \"invalid\" index. The recommended recovery " +
"method in such cases is to drop the index and try again.",
metadata: { pageId: "sql-createindex", section: "concurrently", chunkIndex: 1 },
},
{
kind: "image" as const,
source: "pgadmin-docs",
sourceUrl: "https://www.pgadmin.org/docs/pgadmin4/latest/query_tool.html",
url:
"https://www.pgadmin.org/static/docs/pgadmin4-9.17-docs/_images/" +
"query_output_error.png",
metadata: { pageId: "query_tool", section: "error-output" },
},
];
const pieces: import("./embed").VoyagePiece[][] = examples.map((ex) =>
ex.kind === "text"
? [{ type: "text", text: ex.body }]
: [{ type: "image_url", image_url: ex.url }],
);
const vectors = await embed(pieces);
await db.insert(chunks).values(
examples.map((ex, i) => ({
source: ex.source,
body: ex.kind === "text" ? ex.body : null,
embedding: vectors[i],
metadata:
ex.kind === "text"
? {
kind: "text",
sourceUrl: ex.sourceUrl,
pageId: ex.metadata.pageId,
section: ex.metadata.section,
chunkIndex: ex.metadata.chunkIndex,
}
: {
kind: "image",
url: ex.url,
sourceUrl: ex.sourceUrl,
pageId: ex.metadata.pageId,
section: ex.metadata.section,
},
})),
);That's three rows, two modalities, one column. In Part 3 we replace the
hand-written examples array with a real ingestion pipeline (chunking,
batching, retries, image preprocessing) but the shape stays the same.
"How do I recover from a failed CREATE INDEX CONCURRENTLY?"
psql output: error message screenshot
Pull doc + screenshot from source
A doc and a screenshot, one pipeline. Same call, same column, same query.
A note on batching while it's fresh: Voyage's multimodal endpoint accepts a list of inputs per call. Batching is how you keep latency and cost sane. Start with 16 inputs per call for mixed text/image, smaller if your images are large. Wrap the call in a retry helper with exponential backoff; a 429 on a single batch shouldn't take down a backfill.
Embed the user's question with input_type: "query", then ask Postgres for
the nearest rows by cosine distance:
// src/search.ts
import { cosineDistance, sql } from "drizzle-orm";
import { db } from "./db";
import { chunks } from "./db/schema";
import { embed } from "./embed";
export async function search(query: string, k = 10) {
const [queryVector] = await embed(
[[{ type: "text", text: query }]],
// Tell Voyage this is a query, not a document. Set the type at the
// call site; we'll thread it through `embed` in Part 3.
);
const similarity = sql<number>`1 - (${cosineDistance(
chunks.embedding,
queryVector,
)})`;
return db
.select({
id: chunks.id,
source: chunks.source,
body: chunks.body,
metadata: chunks.metadata,
similarity,
})
.from(chunks)
.orderBy((t) => sql`${t.similarity} DESC`)
.limit(k);
}Call it with search("recovering from a failed concurrent index build") and
you get back a ranked mix of rows: the two paragraphs from the Postgres
docs and the screenshot of the error message, ordered by how close they sit
to the query in vector space. Same index, same query, both modalities. The
multimodal promise from Part 1 is no longer abstract.
(If you want to see the full retrieval-augmented version of this pattern end to end in TypeScript with hybrid search and an LLM, here's the post that does it. We do the proper version in Part 6, but the post is good context.)
A few things worth knowing now so they don't sting later.
Don't embed during a web request. Voyage calls take hundreds of milliseconds, sometimes more. Ingestion belongs in a background job or a script. Query-time embedding (one input) is fine to do inline; corpus embedding is not.
Your image format matters. PNG and JPEG work everywhere. HEIC and AVIF quietly fail in places you wouldn't expect. Normalize on ingest (sharp, ImageMagick, whatever you reach for) to one of the safe formats before sending to Voyage or storing the URL.
Pin your pgvector version in the image tag. pgvector/pgvector:pg16
floats; that's fine for a learn-along but pin to a specific tag
(pgvector/pgvector:0.8.1-pg16 or similar) for anything you operate. The
index file format has shifted between minor versions and an unexpected
upgrade can force a rebuild.
text-embedding-3-small is not in the same space. Just because both
models output 1536 or 1024 dims doesn't mean their vectors are comparable.
You cannot mix outputs from different models in one column. If you switch
models, you re-embed everything. Part 8 is the entire story of doing that
without downtime.
You have a self-hosted Postgres with pgvector enabled, and a Drizzle
schema with one vector column wide enough for voyage-multimodal-3.5 and
an HNSW index ready to grow into. The embed() function takes text or
images and returns vectors in the same space. The ingest script put three
rows in the table, and search() returns them ordered by relevance.
That's the heartbeat. Everything else in this series is making it good, making it fast, and making it survive contact with users. Part 3 starts with the part you'll trip on first: relevance.