RAG connects a language model to documents it was never trained on, such as company policies, product manuals or recent news. Instead of relying on memory, the model reads the retrieved passages and answers from them, often with citations. This keeps answers current and private data usable without retraining the model.
How it works
A RAG system has two stages. First, preparation: documents are split into chunks of a few hundred words, each chunk is turned into an embedding (a list of numbers capturing its meaning) by an embedding model, and these are stored in a vector database or search index. Second, answering: when a user asks a question, the question is also embedded, and the system finds the chunks whose embeddings are closest in meaning. Many systems combine this semantic search with keyword search, then use a reranker model to pick the best few results. Those chunks are inserted into the prompt alongside the question, with an instruction such as 'answer using only these sources and cite them'. The language model then writes the response. Quality depends heavily on the retrieval step: poor chunking, outdated documents or the wrong search results will produce a confident but wrong answer. Because only a handful of relevant passages are sent each time, RAG works even when the full document collection is far larger than the model's context window, and documents can be updated simply by re-indexing them.
Why it matters
Most useful business questions depend on information a general model does not have: your prices, policies, contracts or course notes. RAG lets you use that knowledge safely and cheaply, without the cost of training a custom model. Answers can link back to sources, so people can check them, which builds trust. For creators and educators, it powers assistants that answer from their own content library rather than the open web.
Where you will see it
- A support chatbot answering from the latest help articles and product manuals.
- An internal assistant that searches HR policies and cites the exact section.
- A study tool that answers questions only from uploaded lecture notes and readings.
- Search over a creator's past videos and transcripts to find where a topic was covered.
Limits and common misconceptions
- RAG reduces hallucinations but does not remove them; the model can still misread or ignore retrieved text.
- If retrieval finds the wrong or outdated passages, the answer will be wrong, so document quality and upkeep matter.
- Access control is easy to overlook: the system must not retrieve documents a user is not allowed to see.
How it differs from related terms
- RAG (retrieval-augmented generation) vs Fine-tuning Fine-tuning changes a model's behaviour by training it; RAG leaves the model unchanged and feeds it fresh information at question time.
- RAG (retrieval-augmented generation) vs Grounding Grounding is the general goal of tying answers to verifiable sources; RAG is the most common technique for achieving it.
Tools that use it
Frequently asked questions
Is RAG better than fine-tuning?
They solve different problems. RAG is best for giving a model facts and documents that change, with sources you can check. Fine-tuning is best for teaching a consistent style, format or narrow skill. Many production systems use both: a fine-tuned model that answers from retrieved documents.
Do I need a vector database for RAG?
Not always. A vector database is common because it finds passages by meaning, but small projects can use a simple in-memory index, and some systems use ordinary keyword search. Many tools now offer built-in document upload that handles retrieval for you, so you never manage the database directly.
Why use RAG when context windows are so large?
Large context windows let you paste in lots of text, but sending everything on every question is slow and costly, and models can miss details buried in very long inputs. RAG sends only the most relevant passages, scales to huge document collections and makes it easy to cite exact sources.
Related terms
- Prompt The text, files or other input you give an AI model to tell it what to do, including your question, context and instructions.
- Prompt engineering The practice of designing, testing and refining prompts and their surrounding context so an AI model gives accurate, consistent and useful output.
- Reasoning model A language model trained to generate an internal chain of reasoning before giving its final answer, improving results on multi-step problems.
- RLHF Reinforcement learning from human feedback: a training method where people rank AI responses, and the model is tuned to produce the preferred kind.
- Speech to text AI that converts spoken audio into written text, also called automatic speech recognition (ASR) or transcription.