Models never see letters or words directly. A tokeniser splits text into tokens and maps each one to a number. Context windows, usage limits, speed and API prices are all measured in tokens, so understanding them helps you predict cost and fit more into a prompt.
How it works
Before a model processes text, a tokeniser breaks it into pieces from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Most modern tokenisers use methods such as byte-pair encoding, which starts with single characters and repeatedly merges the most common pairs found in training text. Frequent words like 'the' become one token, while rarer words are split: 'unbelievable' might become 'un', 'believ' and 'able'. Each token maps to an ID number, and the model converts each ID into an embedding it can work with. When generating, the model predicts one token at a time and adds it to the text. In English, a common rule of thumb is about four characters, or three quarters of a word, per token. Other languages, code and numbers often need more tokens for the same meaning. API providers usually price input tokens and output tokens separately, often per million tokens, with output costing more. Context windows range from a few thousand tokens in older models to hundreds of thousands, and some exceed a million.
Why it matters
Tokens are the meter that AI tools run on. If you use an API or a tool with usage limits, token counts decide your bill and how much text, such as a transcript or document, you can include at once. Knowing that output costs more than input, and that some languages use more tokens, helps you budget, trim prompts and choose the right model for high-volume jobs.
Where you will see it
- Estimating API costs before processing thousands of product descriptions.
- Checking whether a long transcript fits inside a model's context window.
- Setting maximum output length so answers do not run on.
- Comparing prices across AI providers on a cost-per-million-tokens basis.
Limits and common misconceptions
- A token is not a word: counts differ between models because each uses its own tokeniser.
- Token splitting explains odd failures, such as miscounting letters in a word, because the model sees chunks, not letters.
- Non-English text can cost noticeably more for the same content because it splits into more tokens.
How it differs from related terms
- Token vs Context window A token is one unit of text; the context window is the maximum number of tokens a model can handle in one go.
- Token vs Embedding A token is a chunk of text; an embedding is the list of numbers a model uses to represent its meaning.
Frequently asked questions
How many words is 1,000 tokens?
In typical English, 1,000 tokens is roughly 750 words, based on the rule of thumb of about four characters per token. Code, numbers, unusual names and many non-English languages use more tokens per word, so the ratio varies.
Do spaces and punctuation count as tokens?
Yes. Punctuation marks usually become their own tokens, and spaces are typically attached to the start of the following word. Line breaks, emojis and special formatting also use tokens, so heavy formatting adds to the count.
Why do output tokens cost more than input tokens?
Input tokens can be processed in parallel in one pass, while output tokens must be generated one at a time, each needing a fresh pass through the model. That makes generation slower and more expensive to serve.
Related terms
- Text to speech (TTS) Technology that converts written text into spoken audio, with modern neural systems producing voices that sound close to human speech.
- Text to video Generative AI that creates moving video clips from a written prompt, and often from a starting image as well.
- Training data The collection of examples, such as text, images, audio or code, that a machine learning model learns patterns from during training.
- Transformer A neural network architecture that uses self-attention to weigh how every token in a sequence relates to every other token.
- Upscaling Increasing the resolution of an image or video, with AI upscalers predicting realistic new detail rather than simply enlarging pixels.