RAG explained: handing AI a reference book
How do chatbots answer from your company's documents? Here's how RAG, a pattern widely used in AI products, works.
About 16 min readingAs of Oct 2026
Lesson guide · goals & preparation
By the end of this guide
- Explain why an LLM doesn't know your internal documents or recent information.
- Understand how RAG works: question, document search, then answer.
- Judge whether RAG or fine-tuning fits your situation.
Who this is for
- Anyone considering a chatbot over internal documents
- Anyone who keeps seeing RAG in AI product plans
What you need
- Reading "What is an LLM?" first will help.
This lesson’s terms, in plain language
Read the short meanings here. Open a term for a fuller explanation and example.
- RAG
- Finding relevant documents first and giving them to the AI along with the question.
- embedding
- A list of numbers that captures the meaning of text or images, used to find similar things.
- fine-tuning
- Training an existing AI model further for a specific purpose.
This reading lesson explains how to use external AI tools. Read from theory to the workflow, examples and explanations, or use the contents to jump to a section.
1 / 4
Theory
Understand what to ask AI to do, how to ask, and the ideas behind the workflow.
“When can I book the meeting room?”
A newcomer asks when a meeting room can be reserved. A plausible account of typical workplace policy does not establish this room’s policy. We will use fictional study-space rules, not instructions for a real facility.
Place three cards on the desk: D01 says the library opens 10 a.m.–6 p.m.; D03 allows booking fourteen days ahead; D04 requires cancellation by the previous day. All numbers are invented. D03 answers our question; the others concern different questions.
Distinguish training knowledge, current conversation material and connected search tools. Services may receive recent information through context or tools. Ask what evidence actually reached this request rather than relying on a blanket training-date claim.
Place retrieved evidence beside the question
RAG retrieves relevant material and includes it when generating an answer. Retrieval brings D03; generation expresses its content in response to the question. Adding that context is not itself additional model training.
Figure: Selecting one relevant document and placing it beside the answer · AI Ping A source record needs a reopenable location as well as a title. Track what an ID points to after revisions or removal, and retain the source version used for the answer.
With a small collection, selecting D03 by hand already teaches the distinction between evidence retrieval and use. Larger systems find candidates from titles, text and aliases. IDs make sources reopenable, but citations alone do not establish accuracy.
An editorial target answer is “You can book fourteen days ahead [D03].” Adding “from 9 a.m., first come first served” invents details absent from the card. Check every claim against the source.
2 / 4
Step-by-step use
Learn what to prepare, how to request help and how to revise the result through examples and explanations.
Keep context when splitting documents
Splitting a handbook can keep requests focused, but a chunk saying “an exception applies here” is useless without its condition. Preserve headings, applicability and relevant neighboring text instead of optimizing only for character count.
Keep source IDs, revision dates, scope and permissions with each chunk. If D03 changes from fourteen to seven days, verify that obsolete chunks are not still retrieved. A later date alone does not resolve applicability.
If conflicting versions have no clear authority, neither average the numbers nor choose a convenient one. Report the conflict and ask the document owner to resolve it. Retrieval needs a source-maintenance process.
Distinguish keyword and semantic retrieval
Keyword search can find “meeting room” but may miss “a space for our group.” Embeddings represent text numerically to retrieve nearby candidates. Semantic proximity helps with paraphrases; it is not a correctness verdict.
Keywords can help with codes, dates and local abbreviations; semantic search can help with paraphrases. Hybrid approaches combine them. Compare methods using questions your readers actually ask.
Always returning three neighbors can produce meeting-room rules for a parking question. Relative rank is not sufficient relevance. Test absent-answer questions and provide a no-evidence path, with thresholds evaluated for the actual model and corpus.
Check evidence, permission and accuracy separately
Users should receive answers only from sources they may access. Exclude staff access-code documents from public retrieval candidates. Sending all documents and asking the model to keep secrets is not access control.
Figure: Separating the searchable document set from excluded documents · AI Ping A retrieved sentence telling the assistant to disclose another file remains source data, not new authorization. Combine instruction/data separation with enforced retrieval scope and separately controlled tool permissions.
Score retrieval and answer fidelity separately. A lucky correct number without its source is not grounded success. Retrieving D03 but changing fourteen days to fourteen hours is a generation failure.
3 / 4
Check & explain
Think about what you would ask AI, then read the explanation. Written submissions and external AI use are optional.
This is a learning check, not a test you must pass. Use the explanations to understand the ideas.
Q1D03 only says fourteen days ahead. Can an answer add “from 9 a.m., first come first served”?
Write your thoughts (optional)
Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.
Read the answer and explanation
No. Time and procedure are unsupported additions. The fourteen-day value itself belongs only to this fictional exercise.
This is your own review, not an automatic score.
Q2The correct source was retrieved, but the answer says fourteen hours. Which stage failed?
Write your thoughts (optional)
Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.
Read the answer and explanation
Generation introduced a unit error. Compare claims with the source and test that day/hour units survive.
This is your own review, not an automatic score.
Q3Is permission handled by sending staff documents and instructing the model not to reveal them?
Write your thoughts (optional)
Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.
Read the answer and explanation
No. Exclude inaccessible sources before model context is built. The application must enforce permissions derived from trusted identity information.
This is your own review, not an automatic score.
Common mistakes
- Expecting good results from dumping in every file without organizing them
- Reading the answer without checking the source documents
Your learning record
Completion is optional. Mark this lesson when you feel ready, or move to the next lesson without marking it.
Completion is saved in this browser.
4 / 4
Apply to your work
Consider how new material changes the workflow and checking criteria. Trying an external AI tool is optional.
Try it: evaluate with three cards
Write D01, D03 and D04 on paper. Ask about booking lead time, cancellation timing and parking. Use D03 and D04 for the first two; mark parking as absent. Check that no time, price or procedure was added.
Remove D03 and repeat. Even if you remember fourteen days, it is no longer supported by the retrieved set. This distinguishes remembered knowledge from grounded evidence. Restore D03 and verify the source returns.
Keep a three-column record: question, retrieved card and supported answer. When the parking answer says the source is missing, mark that as correct restraint. A fluent invented parking policy is a failed answer.
Fine-tuning adjusts model behavior from examples; RAG retrieves context at request time. They can be combined. These changing rules need maintained source records and repeated retrieval checks. The coding lesson turns this card-selection exercise into a small program.
Write a note (optional)
Use this only if you want to record a prompt or your understanding. This is a notebook, not a connected AI assistant.
Your writing is not sent anywhere. It disappears when you leave or reload this page, so copy anything you want to keep.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — arXiv (Lewis et al., 2020)
- Security filters for trimming search results — Microsoft Learn
