Training is the expensive, one-off process that sets a model's parameters. Inference is everything that happens afterwards, every time someone sends a prompt. Because it runs millions of times, inference drives most of the ongoing cost, speed and energy use of AI products.
How it works
During inference the model's parameters are fixed; nothing new is learned. Your input is converted into numbers, for a language model that means tokens, and passed through the network's layers in a forward pass. For text generation there are two phases. First, the model reads your whole prompt at once, often called prefill. Then it generates the answer one token at a time, feeding each new token back in, which is called decoding. That is why long replies take longer and why output tokens are usually priced higher than input tokens. Inference runs on specialised chips, mainly GPUs and other AI accelerators, because it involves huge numbers of matrix multiplications. Providers make it cheaper and faster with techniques such as batching many users' requests together, caching earlier calculations, quantisation, which stores numbers at lower precision, and smaller distilled models. Reasoning models spend extra inference compute thinking before they answer, which can improve accuracy but raises cost and latency. Inference can run in the cloud or, for smaller models, directly on a laptop or phone.
Why it matters
Inference is what you pay for. API pricing is usually quoted per million input and output tokens, and AI subscriptions and credits are ways of rationing inference. Understanding it helps you pick the right model for a job, keep prompts lean, and estimate costs before automating thousands of tasks. It also explains trade-offs between speed, price and quality, and why on-device AI can be more private.
Where you will see it
- Generating a chatbot reply each time a customer sends a message.
- Transcribing a podcast episode with a speech-to-text model.
- Running a small model on a phone to sort photos without uploading them.
- Batch-processing thousands of product descriptions overnight through an API.
Limits and common misconceptions
- Cheap per request is not cheap at scale; long prompts, big models and reasoning modes multiply costs quickly.
- Inference does not update the model; it does not learn from your chat unless the provider separately retrains it.
- Running models locally needs enough memory, and large models may be too slow or simply not fit.
How it differs from related terms
- Inference vs Fine-tuning Fine-tuning changes a model's parameters with extra training, while inference uses the parameters as they are to produce outputs.
- Inference vs GPU A GPU is the hardware that usually performs inference; inference is the task itself.
Frequently asked questions
What is the difference between training and inference?
Training teaches a model by adjusting its parameters using large amounts of data, and it happens before release. Inference is using the finished model to answer new inputs. Training is a huge upfront job; inference is smaller per request but happens constantly, so across a product's life it can cost more overall.
Why is AI inference expensive?
Each request involves billions of calculations on specialised, costly chips, and text is generated token by token, so longer answers need more work. Bigger models and reasoning modes require more computation per token. Providers recover these costs through per-token pricing, usage limits, subscriptions and credits.
Can I run AI inference on my own computer?
Yes, for smaller models. Open-weight models can run locally using free tools, especially on machines with a capable GPU or plenty of unified memory. Local inference keeps data private and avoids per-token fees, but the largest models need far more memory than typical laptops have.
Related terms
- Guardrails Controls placed around an AI system that check its inputs and outputs and block, change or flag anything unsafe, off-policy or off-topic.
- Hallucination When an AI model produces information that sounds confident and plausible but is false, unsupported or entirely made up.
- Large language model (LLM) A neural network, usually a transformer, trained on vast amounts of text to predict the next token, which lets it understand and generate human language.
- Lip-sync Making a person's or avatar's mouth movements in video match a voice track, either by animating a character or by editing real footage.
- Machine learning A branch of AI in which computers learn patterns from data to make predictions or decisions, instead of following rules a programmer wrote by hand.