During training the model learns to undo noise added to real examples. At generation time it runs that skill in reverse, guided by your text prompt. Diffusion powers most well-known image generators and many video and audio tools.
How it works
Training has two halves. In the forward process, real images are gradually corrupted with random noise over many steps until nothing recognisable remains. The model, typically a U-Net or a diffusion transformer, is trained to predict the noise that was added at each step, so it learns how to clean an image a little at a time. To generate, the process runs backwards: start from pure noise and ask the model to remove a bit of noise, again and again, often 20 to 50 steps with modern samplers. A text encoder turns your prompt into numbers that steer each step, and a setting called guidance scale controls how strictly the image follows the prompt. Most practical systems use latent diffusion, popularised by Stable Diffusion in 2022: they work in a compressed representation rather than full-size pixels, then decode the result into an image. A seed number fixes the starting noise, so the same seed and settings reproduce the same picture. Video models extend the idea across frames.
Why it matters
If you make thumbnails, product shots, ads or social graphics with AI, you are almost certainly using a diffusion model. Knowing how it works explains why prompts, seeds, steps and guidance change results, why text inside images used to come out garbled and why inpainting can fix one area without redrawing the rest. That understanding helps creators get consistent results faster and helps businesses brief designers and choose the right tool.
Where you will see it
- Generating thumbnails, illustrations and concept art from a text prompt.
- Inpainting: replacing or removing an object in part of a photo.
- Extending a photo beyond its edges (outpainting) for different aspect ratios.
- Generating short video clips or upscaling low-resolution images.
Limits and common misconceptions
- Can struggle with exact text, hands, counting objects and precise layouts, though newer models have improved.
- Training on scraped images raises copyright and consent questions; check each tool's licence for commercial use.
- Generation takes many steps, so it is slower and more compute-heavy than a single model pass.
How it differs from related terms
- Diffusion model vs Text to image Text-to-image is the task of making a picture from words; diffusion is the most common technique used to do that task.
- Diffusion model vs Transformer A transformer is a network architecture, while diffusion is a generation process; some diffusion models use a transformer as their denoising network.
Tools that use it
Frequently asked questions
Why is it called diffusion?
The name comes from physics, where diffusion describes particles spreading out, like ink in water. Adding noise to an image step by step is similar to that spreading. The model learns to reverse it, turning scattered noise back into a structured picture.
Is Midjourney a diffusion model?
Midjourney has not published full technical details of its models, but it is widely understood to use diffusion-based generation, like most leading image generators. Stable Diffusion is openly documented as a latent diffusion model, and many other tools follow similar methods.
What is the difference between diffusion models and GANs?
A GAN creates an image in one pass, trained by a generator competing against a discriminator. A diffusion model builds the image over many denoising steps. Diffusion is usually more stable to train and gives more varied results, which is why it replaced GANs in most image tools.
Related terms
- Deep learning Machine learning that trains neural networks with many stacked layers to learn patterns directly from raw data such as text, pixels or sound.
- Deepfake Synthetic video, image or audio, made with AI, that realistically shows a real person saying or doing something they never said or did.
- Dubbing (AI) Using AI to translate a video's speech into another language and re-voice it, often in a clone of the original speaker's voice.
- Embedding A list of numbers (a vector) that captures the meaning of text, images or audio, so that similar items end up close together mathematically.
- Fine-tuning Further training an already trained AI model on a smaller, targeted set of examples so it reliably adopts a specific style, format, task or domain.