Introduced by Google researchers in the 2017 paper 'Attention Is All You Need', the transformer replaced older sequence models for language tasks. It underpins large language models, and variants now power many image, audio and video models. Its design trains efficiently on GPUs, which allowed models to scale to billions of parameters.
How it works
A transformer first turns each token into an embedding and adds positional information, because the model otherwise has no sense of word order. The core operation is self-attention. For each token, the model computes three vectors, a query, a key and a value. Comparing one token's query with every other token's key gives attention scores that decide how much each token should draw on the others' values. In 'the trophy did not fit in the case because it was too big', attention helps link 'it' to 'trophy'. Multi-head attention runs several of these comparisons in parallel, so different heads can track grammar, meaning or long-range references. Each attention block is followed by a feed-forward layer, with residual connections and normalisation keeping training stable. Dozens of these blocks are stacked. The original design had an encoder and a decoder; encoder-only models such as BERT suit understanding tasks, while decoder-only models such as GPT generate text one token at a time. Unlike recurrent networks, a transformer processes all tokens at once during training. The catch is that standard attention cost grows with the square of the sequence length.
Why it matters
Almost every AI tool a creator or business uses today, from chatbots and writing assistants to transcription and many image generators, is built on transformers. Understanding the basics explains everyday behaviour: why models have context limits, why longer prompts cost more and run slower, and why a model predicts text one token at a time rather than looking facts up. That helps you set realistic expectations.
Where you will see it
- Powering chatbots and writing assistants built on large language models.
- Translating text between languages.
- Transcribing speech in recognition models such as Whisper.
- Processing image patches in vision transformers and some image and video generators.
Limits and common misconceptions
- Standard attention cost grows quadratically with input length, which makes very long contexts expensive.
- Attention weights are not a reliable explanation of why a model gave an answer.
- The architecture predicts plausible tokens; it has no built-in fact checking, so hallucinations remain possible.
How it differs from related terms
- Transformer vs Neural network A neural network is the broad family of models; a transformer is one specific architecture within it, built around attention.
- Transformer vs Large language model (LLM) An LLM is a large trained model for language; a transformer is the blueprint most LLMs are built on.
Frequently asked questions
What does GPT stand for?
GPT stands for Generative Pre-trained Transformer. Generative means it produces new text, pre-trained means it first learned from a large general dataset before any specialised training, and transformer is the neural network architecture it uses.
Why did transformers replace RNNs?
Recurrent neural networks read text one word at a time, which is slow to train and makes long-range links hard to learn. Transformers process the whole sequence in parallel and connect any two tokens directly through attention, so they train faster on GPUs and scale better.
Are transformers only used for text?
No. Vision transformers split images into patches and treat them like tokens. Transformers are also used in speech recognition, music generation, protein structure prediction and many image and video generation models.
Related terms
- Token The basic unit of text a language model reads and writes, usually a whole word, part of a word, a punctuation mark or a space.
- Training data The collection of examples, such as text, images, audio or code, that a machine learning model learns patterns from during training.
- Upscaling Increasing the resolution of an image or video, with AI upscalers predicting realistic new detail rather than simply enlarging pixels.
- Vector database A database built to store embeddings and quickly find the items whose vectors are most similar in meaning to a query.
- Voice cloning Using AI to create a synthetic voice that imitates a specific person's tone, accent and speaking style from recordings of them.