Counting GPT tokens

LLMs do not read characters or words. They read tokens, and they bill per token. An accurate GPT token count before you send a request tells you two things: what the call will cost, and whether the input still fits the context window.

A token is a chunk of text, not a word

A token is a piece of text that the model processes as a single unit. This could be as short as one character or as long as one word, depending on the language and the specific word. For example, the phrase "Hello, world!" might be broken down into the following tokens: "Hello", ",", "world", and "!". Tokens are the units the model reads and generates.

The encoding decides how your text splits

Encodings are the methods used to convert text into tokens that the model can understand. Different models use different encoding schemes, and this can affect how text is tokenized. For example, OpenAI models like GPT-4 and GPT-3.5 use the cl100k_base encoding. These encodings determine how efficiently the model processes the text and, ultimately, the cost of using the model. Other LLM providers, such as Hugging Face or Google's BERT, use their own unique tokenization methods, which can vary significantly in how they segment text into tokens. A count from one tokenizer is not valid for another.

Count GPT tokens in Node with tiktoken

tiktoken is the OpenAI tokenizer, published by OpenAI, so its counts match what the API bills. The npm package uses WebAssembly for efficient token counting. To get started, install it using:

npm install tiktoken

Here's a snippet to count tokens in a JSON file containing an array of items:

import { get_encoding } from "tiktoken";
import fs from "fs";
 
const filePath = process.argv[2];
const content = fs.readFileSync(filePath, "utf-8");
const parsedContent: any[] = JSON.parse(content);
 
const tokens: number[] = [];
const enc = get_encoding("cl100k_base");
 
for (const item of parsedContent) {
	const encodedContent = enc.encode(JSON.stringify(item));
	tokens.push(encodedContent.length);
}
 
const total = tokens.reduce((acc, cur) => acc + cur, 0);
const average = total / tokens.length;
const max = Math.max(...tokens);
const min = Math.min(...tokens);
 
console.log("Total", total.toLocaleString());
console.log("Average", average.toLocaleString());
console.log("Max", max.toLocaleString());
console.log("Min", min.toLocaleString());
console.log("Avg/Max average", ((average + max) / 2).toLocaleString());
 
enc.free();

Most newer models in the GPT family use cl100k_base encoding. This includes models like GPT-4, GPT-3.5, and even embedding models such as text-embedding-3-small and text-embedding-3-large.

Run the GPT token counter in your browser

For a one-off check, a script is more work than it is worth. I built a GPT token counter that runs in the browser. You can upload files or enter text and check total token count, max token count (if your input is an array of items) and average token count.

To understand the cost implications, you can check OpenAI's pricing here. Remember that token count can also impact the performance and accuracy of the responses generated by the models.

tl;dr

• Tokens: Basic units of text processed by LLMs.
• Encodings: Methods to convert text into tokens, differing across LLM providers.
• Token Counting: Use tiktoken for efficient counting with OpenAI models.
• Impact on Cost: Token count affects both cost and performance.