Training data determines what a model knows, the styles it can copy and the mistakes it tends to make. Large language models learn from vast amounts of text, while smaller specialised models may need only thousands of carefully labelled examples. Quality, diversity, size and legal rights all matter.
How it works
During training, a model is shown examples and adjusts its internal parameters to reduce its errors. For large language models, the first stage, called pre-training, uses enormous text collections drawn from public web pages, books, code repositories and other sources, measured in trillions of tokens. The task is self-supervised: the model learns by predicting the next token, so no human labelling is needed. Later stages use smaller, curated datasets, such as written examples of good answers for instruction tuning and human preference ratings for RLHF. Other models rely on labelled data, where each example carries a correct answer, for instance photos tagged 'cat' or 'dog'. Before training, data is cleaned: duplicates are removed, low-quality or harmful content is filtered, and personal information may be stripped. Teams also hold back separate validation and test sets to check the model works on examples it has never seen. Because training stops at a certain point, a model has a knowledge cutoff, and it will not know about later events unless it is given extra information, for example through search or RAG.
Why it matters
Training data explains much of a model's behaviour. If a tool is weak in your language, niche or industry, it probably saw little relevant data. For businesses, data questions are also legal ones: whether your uploads are used to train future models, whether outputs are safe to use commercially, and whether customer data is protected. Always check a tool's data-use settings before sharing sensitive material.
Where you will see it
- Pre-training large language models on web text, books and code.
- Fine-tuning a model on a company's support tickets to match its tone.
- Training an image classifier to spot product defects on a factory line.
- Building speech recognition systems from recorded audio with transcripts.
Limits and common misconceptions
- Biased or unbalanced data produces biased models, for example poorer accuracy for under-represented accents or groups.
- The use of copyrighted material in training is the subject of ongoing lawsuits and regulation in several countries.
- If test questions leak into training data, benchmark scores can overstate a model's real ability.
How it differs from related terms
- Training data vs Synthetic data Synthetic data is artificially generated, and it can be one source of training data alongside real-world examples.
- Training data vs Fine-tuning Fine-tuning is a later training step that uses a smaller, focused dataset to adapt an already trained model.
Frequently asked questions
Is my data used to train AI models?
It depends on the tool and plan. Many consumer AI services may use conversations to improve models unless you opt out in settings, while business and API plans often exclude customer data by default. Read the privacy policy and check the data controls.
How much training data does an AI need?
It varies enormously. Large language models are pre-trained on trillions of tokens. A narrow classifier built on top of an existing model may work well with a few hundred to a few thousand good examples. Quality and relevance often matter more than raw volume.
What is a knowledge cutoff?
It is the point in time after which a model has no training data. The model will not know about later events, products or prices unless it can search the web or is given documents in the prompt.
Related terms
- Text to video Generative AI that creates moving video clips from a written prompt, and often from a starting image as well.
- Token The basic unit of text a language model reads and writes, usually a whole word, part of a word, a punctuation mark or a space.
- Transformer A neural network architecture that uses self-attention to weigh how every token in a sequence relates to every other token.
- Upscaling Increasing the resolution of an image or video, with AI upscalers predicting realistic new detail rather than simply enlarging pixels.
- Vector database A database built to store embeddings and quickly find the items whose vectors are most similar in meaning to a query.