Search text and screenshots with one pgvector column

By the end of this post, the project has a heartbeat: a Postgres you self-host, a Drizzle schema, a Voyage call that takes both text and an image, a table with a few hundred rows of real Postgres docs and screenshots, and one query that returns the right thing.

That sounds small. The reason it's a whole post is that almost every later decision (indexing, tuning, evals, re-embedding) is shaped by what you pick in the next twenty minutes. The dimension on your vector column, the extension version, the build flags on Postgres, the way you batch embedding calls. None of these throw if you get them wrong. They just make Part 5 miserable.

So we set up like the next six posts exist.

The project: a docs assistant for text and screenshots

The project is a multimodal docs assistant. The corpus is Postgres' own documentation pages plus a curated set of screenshots (psql output, EXPLAIN plans, pgAdmin views). Real prose, real images, the kind of thing your users would actually search.

We're going to want this assistant to answer questions like "how do I recover from a failed CREATE INDEX CONCURRENTLY?" and have it pull the right doc paragraph and the screenshot of the relevant error message. Same query, one index, both modalities.

Run Postgres on your own machine

Self-hosted is a deliberate choice. We need to see the build flags, the config knobs, and what changing shared_buffers actually does. You can't see those on a managed Postgres without indirection, and Parts 4 and 5 need them in the foreground. If you eventually run on Supabase or RDS, the intuition transfers. The opposite direction (start managed, learn to tune) mostly doesn't.

Use the official pgvector image so the extension is already in the binary:

# docker-compose.yml
services:
  db:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: docs_assistant
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data
volumes:
  pgdata:

Save that as docker-compose.yml and start the container:

docker compose up -d

Then enable the extension once. Drizzle Kit won't do this for you (and that quietly bites people on a fresh database):

CREATE EXTENSION IF NOT EXISTS vector;

If this is your first time wiring pgvector into Drizzle, I wrote the setup walkthrough separately. The piece below assumes you've got drizzle-orm, drizzle-kit, and postgres (or pg) installed.

The schema: one table, one vector column

One table for now. Splitting "documents" and "chunks" gets pulled out in Part 3 when chunking earns it; right now a single table keeps the cursor on the parts that matter.

// src/db/schema.ts
import {
  pgTable,
  serial,
  text,
  jsonb,
  vector,
  index,
} from "drizzle-orm/pg-core";
 
export const chunks = pgTable(
  "chunks",
  {
    id: serial("id").primaryKey(),
    source: text("source").notNull(),
    body: text("body"),
    // voyage-multimodal-3.5 returns 1024-dimensional vectors by default.
    // You can request 256, 512, or 2048 instead. Whatever you pick, every
    // row has to agree on it.
    embedding: vector("embedding", { dimensions: 1024 }),
    metadata: jsonb("metadata").$type<{
      kind: "text" | "image";
      url?: string;
      pageId?: string;
      chunkIndex?: number;
    }>(),
  },
  (t) => [
    index("chunks_embedding_hnsw_idx").using(
      "hnsw",
      t.embedding.op("vector_cosine_ops"),
    ),
  ],
);

A few notes worth keeping in your head, because they all get cashed in later:

1. dimensions: 1024 is a one-way door. Every row has to agree on it. You can re-embed the whole table, sure, but that's Part 8 with a careful zero-downtime story. Pick on purpose.

2. The HNSW index uses vector_cosine_ops. That has to match the operator you query with (<=>). If you query with <-> instead, the index goes unused and you'll wonder why everything is slow. I went deeper on what each pgvector operator actually means in a separate post.

3. One column, both modalities. body is nullable because image rows don't have text content. The metadata.kind field tells you which it is when reading back results.

Run drizzle-kit generate and drizzle-kit migrate and you have the table. Don't forget the CREATE EXTENSION above the first migration if this is a brand new database.

Voyage embeds text and images through one call

Voyage's multimodal endpoint takes a list of inputs, each input is a list of content pieces, and each piece is either text or an image reference. The boring shape is on purpose: text and image go through the same call, and each input returns one vector in the same coordinate space.

There's an SDK but I prefer fetch for code that's going to be in front of people. It's verifiable against the docs without grep:

// src/embed.ts
type VoyagePiece =
  | { type: "text"; text: string }
  | { type: "image_url"; image_url: string }
  | { type: "image_base64"; image_base64: string };
 
export async function embed(
  inputs: VoyagePiece[][],
): Promise<number[][]> {
  const res = await fetch(
    "https://api.voyageai.com/v1/multimodalembeddings",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.VOYAGE_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        model: "voyage-multimodal-3.5",
        input_type: "document",
        inputs: inputs.map((content) => ({ content })),
      }),
    },
  );
 
  if (!res.ok) {
    throw new Error(`Voyage ${res.status}: ${await res.text()}`);
  }
 
  const json = (await res.json()) as { data: { embedding: number[] }[] };
  return json.data.map((d) => d.embedding);
}

Two things worth pulling out of that. input_type: "document" tells Voyage this is corpus content (not a query). When you embed a user's query at search time, switch it to "query". The model treats them slightly differently and using the right one helps relevance.

Second, image_url is a public URL Voyage's servers can fetch. If your screenshots aren't public you can send image_base64 instead and skip the URL altogether. We'll use both in this series; the URL version is cheaper for big libraries, the base64 version is what you reach for during development.

Ingest a sliver of the corpus

Here's the smallest honest ingestion loop. Two text chunks and one screenshot, all hitting the same embed call, all landing in the same table.

The text comes from the CREATE INDEX page in the PostgreSQL manual, specifically its "Building Indexes Concurrently" section. The screenshot is the error-output panel from the pgAdmin query tool docs. Real sources, so you can check a retrieved chunk against the page it came from instead of trusting a paraphrase I wrote. Note the sourceUrl on every record: you want it from day one, because the moment retrieval returns something surprising the first question is always "where did this come from?"

// src/ingest.ts
import { db } from "./db";
import { chunks } from "./db/schema";
import { embed } from "./embed";
 
// postgresql.org/docs/current/sql-createindex.html — "Building Indexes
// Concurrently". Excerpted verbatim so a retrieved chunk can be checked
// against the published page.
const CREATE_INDEX_URL =
  "https://www.postgresql.org/docs/current/sql-createindex.html";
 
const examples = [
  {
    kind: "text" as const,
    source: "postgres-docs",
    sourceUrl: CREATE_INDEX_URL,
    body:
      "PostgreSQL will build the index without taking any locks that prevent " +
      "concurrent inserts, updates, or deletes on the table; whereas a " +
      "standard index build locks out writes (but not reads) on the table " +
      "until it's done.",
    metadata: { pageId: "sql-createindex", section: "concurrently", chunkIndex: 0 },
  },
  {
    kind: "text" as const,
    source: "postgres-docs",
    sourceUrl: CREATE_INDEX_URL,
    body:
      "If a problem arises while scanning the table, such as a deadlock or a " +
      "uniqueness violation in a unique index, the CREATE INDEX command will " +
      "fail but leave behind an \"invalid\" index. The recommended recovery " +
      "method in such cases is to drop the index and try again.",
    metadata: { pageId: "sql-createindex", section: "concurrently", chunkIndex: 1 },
  },
  {
    kind: "image" as const,
    source: "pgadmin-docs",
    sourceUrl: "https://www.pgadmin.org/docs/pgadmin4/latest/query_tool.html",
    url:
      "https://www.pgadmin.org/static/docs/pgadmin4-9.17-docs/_images/" +
      "query_output_error.png",
    metadata: { pageId: "query_tool", section: "error-output" },
  },
];
 
const pieces: import("./embed").VoyagePiece[][] = examples.map((ex) =>
  ex.kind === "text"
    ? [{ type: "text", text: ex.body }]
    : [{ type: "image_url", image_url: ex.url }],
);
 
const vectors = await embed(pieces);
 
await db.insert(chunks).values(
  examples.map((ex, i) => ({
    source: ex.source,
    body: ex.kind === "text" ? ex.body : null,
    embedding: vectors[i],
    metadata:
      ex.kind === "text"
        ? {
            kind: "text",
            sourceUrl: ex.sourceUrl,
            pageId: ex.metadata.pageId,
            section: ex.metadata.section,
            chunkIndex: ex.metadata.chunkIndex,
          }
        : {
            kind: "image",
            url: ex.url,
            sourceUrl: ex.sourceUrl,
            pageId: ex.metadata.pageId,
            section: ex.metadata.section,
          },
  })),
);

That's three rows, two modalities, one column. In Part 3 we replace the hand-written examples array with a real ingestion pipeline (chunking, batching, retries, image preprocessing) but the shape stays the same.

A note on batching while it's fresh: Voyage's multimodal endpoint accepts a list of inputs per call. Batching is how you keep latency and cost sane. Start with 16 inputs per call for mixed text/image, smaller if your images are large. Wrap the call in a retry helper with exponential backoff; a 429 on a single batch shouldn't take down a backfill.

Run the first real query

Embed the user's question with input_type: "query", then ask Postgres for the nearest rows by cosine distance:

// src/search.ts
import { cosineDistance, sql } from "drizzle-orm";
import { db } from "./db";
import { chunks } from "./db/schema";
import { embed } from "./embed";
 
export async function search(query: string, k = 10) {
  const [queryVector] = await embed(
    [[{ type: "text", text: query }]],
    // Tell Voyage this is a query, not a document. Set the type at the
    // call site; we'll thread it through `embed` in Part 3.
  );
 
  const similarity = sql<number>`1 - (${cosineDistance(
    chunks.embedding,
    queryVector,
  )})`;
 
  return db
    .select({
      id: chunks.id,
      source: chunks.source,
      body: chunks.body,
      metadata: chunks.metadata,
      similarity,
    })
    .from(chunks)
    .orderBy((t) => sql`${t.similarity} DESC`)
    .limit(k);
}

Call it with search("recovering from a failed concurrent index build") and you get back a ranked mix of rows: the two paragraphs from the Postgres docs and the screenshot of the error message, ordered by how close they sit to the query in vector space. Same index, same query, both modalities. The multimodal promise from Part 1 is no longer abstract.

(If you want to see the full retrieval-augmented version of this pattern end to end in TypeScript with hybrid search and an LLM, here's the post that does it. We do the proper version in Part 6, but the post is good context.)

From the trenches

A few things worth knowing now so they don't sting later.

Don't embed during a web request. Voyage calls take hundreds of milliseconds, sometimes more. Ingestion belongs in a background job or a script. Query-time embedding (one input) is fine to do inline; corpus embedding is not.

Your image format matters. PNG and JPEG work everywhere. HEIC and AVIF quietly fail in places you wouldn't expect. Normalize on ingest (sharp, ImageMagick, whatever you reach for) to one of the safe formats before sending to Voyage or storing the URL.

Pin your pgvector version in the image tag. pgvector/pgvector:pg16 floats; that's fine for a learn-along but pin to a specific tag (pgvector/pgvector:0.8.1-pg16 or similar) for anything you operate. The index file format has shifted between minor versions and an unexpected upgrade can force a rebuild.

text-embedding-3-small is not in the same space. Just because both models output 1536 or 1024 dims doesn't mean their vectors are comparable. You cannot mix outputs from different models in one column. If you switch models, you re-embed everything. Part 8 is the entire story of doing that without downtime.

You have a working multimodal search

You have a self-hosted Postgres with pgvector enabled, and a Drizzle schema with one vector column wide enough for voyage-multimodal-3.5 and an HNSW index ready to grow into. The embed() function takes text or images and returns vectors in the same space. The ingest script put three rows in the table, and search() returns them ordered by relevance.

That's the heartbeat. Everything else in this series is making it good, making it fast, and making it survive contact with users. Part 3 starts with the part you'll trip on first: relevance.