RAG explained: handing AI a reference book

How do chatbots answer from your company's documents? Here's how RAG, a pattern widely used in AI products, works.

Key ideas

1. 1

   The limit: it only knows what it was trained on

   An LLM only knows what was in its training data. It can't know your company's leave policy or the price list you changed yesterday.

2. 2

   Hand it the reference material with the question

   RAG, or retrieval-augmented generation, finds relevant documents when a question comes in and passes them to the LLM along with the question.

   It's like an open-book exam. Because the answer comes from specific documents, you can show them as sources and check the answer.

   How an internal-docs chatbot handles one question (example)

   Step	What happens	Example
   1	Take the question	"How far ahead can I book a meeting room?"
   2	Find relevant documents	Finds the "Booking rooms" section of the facilities guide
   3	Send question and documents together	Passes that section's text and the question to the LLM
   4	Answer with sources	"You can book up to two weeks ahead" + a link to the source

3. 3

   Finding the right documents is half the job

   To find relevant documents, systems usually turn text into numbers that capture meaning — a technique called embedding — so documents with similar meaning match even when the words differ.

   Retrieve the wrong documents and you get the wrong answer. Most of a RAG system's quality is decided at this step.

   Check it

   • Open the source documents shown with an answer and check that the answer matches what they say.
   • If the wrong documents keep showing up as sources, fix how documents are organized or searched before tuning the answers.
4. 4

   How it differs from fine-tuning

   Fine-tuning trains the model itself further; RAG leaves the model alone and changes the reference material. For information that changes often, RAG is much easier to maintain.

   RAG and fine-tuning at a glance

   Compare	RAG	Fine-tuning
   What changes	The reference documents	The model itself
   When information changes	Swap the documents	Train it again
   Showing sources	Can show the documents it used	Hard to show where an answer came from
   Good for	Policies and product facts that change often	Learning a consistent tone or format