A minimal RAG chatbot with ten documents

Build a Python chatbot that searches ten internal policy documents with embeddings and answers only from what it finds, then test search and answers separately and fix what breaks.

Step by step

1. 1

   Prepare ten documents, one topic each

   Search works one document at a time, so keep one topic per document and give each an ID and a title. A long policy handbook gets cut by heading into pieces a few sentences long.

   In truth, with only ten documents you could put all of them in the prompt: Anthropic notes that a knowledge base under 200,000 tokens (about 500 pages) can go in whole, with no need for RAG. Treat this guide as practice, a small skeleton of RAG for when your documents grow.

   The ten internal policy documents used in this guide (a fictional company)

   ID	Title	Content
   D01	Annual leave	Request annual leave in the HR system at least 3 days ahead. Half days can be taken in the morning or the afternoon.
   D02	Working from home	You can work from home up to twice a week. Tell your manager by 6 p.m. the day before.
   D03	Booking meeting rooms	Meeting rooms can be booked in the room booking system from two weeks before the date. Bookings over 4 hours need approval from the facilities team.
   D04	Travel expenses	Hotels on domestic trips are covered up to $100 a night. Upload receipts within 7 days of getting back.
   D05	Work card expenses	Reconcile your work card spending in the expense system by the 5th of each month.
   D06	Taking laptops off site	Request approval from the security team before taking a company laptop off site.
   D07	Equipment requests	Request monitors, keyboards and other work equipment on the IT support page. It usually arrives within 3 working days.
   D08	Wellness credit	You get $500 in wellness credit a year, usable until 31 December of that year.
   D09	Training budget	Work-related training is covered up to $1,000 a year. Get your manager's approval before you enrol.
   D10	Health checkups	You can have one health checkup a year at a hospital the company designates.

   Check it

   • Check that each document covers one topic and doesn't overlap with another. If any contain personal data such as real names or contact details, take it out before you start.
2. 2

   Find the closest documents with embeddings

   An embedding turns the meaning of a piece of text into a list of numbers (a vector). Convert the question and the documents the same way, pick the closest, and you find documents with similar meaning even when the words differ.

   Anthropic doesn't offer its own embedding model; its docs use Voyage AI as the example. For ten or so documents you don't need a vector database: one array in memory is enough.

   I have ten internal policy documents in a docs list (id, title, text). Write Python that embeds them with voyageai's voyage-4 model into a numpy array, plus a search function that returns the three documents closest to a question.

   The key is embedding documents with input_type="document" and questions with input_type="query". Create the document embeddings once and reuse them.

   Example codeCopy

   import numpy as np import voyageai vo = voyageai.Client() # reads the VOYAGE_API_KEY environment variable docs = [ {"id": "D03", "title": "Booking meeting rooms", "text": "Meeting rooms can be booked in the room booking system from two weeks before the date."}, # add D01 to D10 in the same shape ] texts = [d["title"] + ": " + d["text"] for d in docs] doc_vecs = np.array(vo.embed(texts, model="voyage-4", input_type="document").embeddings) def search(question, k=3): q = vo.embed([question], model="voyage-4", input_type="query").embeddings[0] scores = doc_vecs @ np.array(q) top = np.argsort(scores)[::-1][:k] return [docs[i] for i in top]

   input_type="document"

   Creates document embeddings. Questions get input_type="query" instead; telling the two apart improves retrieval.

   doc_vecs @ np.array(q)

   Scores every document against the question. Voyage embeddings are normalized to length 1, so the dot product equals cosine similarity.

   np.argsort(scores)[::-1][:k]

   Picks the k highest-scoring documents.

   Check it

   • Before adding the LLM, run search("how early can I book a room?") and check that D03 comes first. If search is wrong, the answer will be too.
3. 3

   Answer only from what was found

   Put the three documents the search returns, with their IDs, in front of the question and send it to Claude. The system prompt sets the rules: answer only from the documents, cite the source ID, and say so when the documents don't cover it.

   Explicit permission to say it doesn't know cuts down on invented answers, and the IDs let a person check the source straight away.

   Write an answer function in which Claude answers from the three documents my search function returns. Use claude-opus-5-5.
   It should answer only from the documents, end with the source ID like [D03], and reply only “I couldn't find that in the documents.” when they don't cover the question.

   Bundle the search results into text tagged with document IDs, put it before the question, and keep the rules in the system prompt.

   Example output: Meeting rooms can be booked in the room booking system from two weeks before the date. [D03]

   Example codeCopy

   import anthropic client = anthropic.Anthropic() SYSTEM = ( "You are an internal policy assistant. Answer only from the documents below and end with the source ID, like [D03]. " "If the documents don't cover it, reply only “I couldn't find that in the documents.”" ) def answer(question): hits = search(question) context = "\n".join("[" + d["id"] + "] " + d["title"] + ": " + d["text"] for d in hits) response = client.messages.create( model="claude-opus-5-5", max_tokens=1024, system=SYSTEM, messages=[{"role": "user", "content": "Documents:\n" + context + "\n\nQuestion: " + question}], ) return next(b.text for b in response.content if b.type == "text") print(answer("How far ahead can I book a meeting room?"))

   search(question)

   Fetches the three relevant documents with the search function from step 2.

   "[" + d["id"] + "] "

   Tags each document with its ID so the answer can point to its source, like [D03].

   system=SYSTEM

   The rules: nothing beyond the documents, and say so when the answer isn't there.

   Check it

   • Open the document each answer's ID points to and check that they agree.
   • Ask something the documents don't cover (a parking allowance, say) and check that it replies “I couldn't find that in the documents.”
4. 4

   Write test questions

   To see whether search and answers work, you need questions whose answers you know. Don't just ask about the document titles: mix in the way people really talk, their shorthand, and questions the documents don't cover.

   Using the ten internal policy documents below, write five test questions for the chatbot. Four should be answerable from the documents, written the way staff really talk, shorthand included; one should be something the documents don't cover.
   Put them in a table: question / correct document ID ('none' if there isn't one)
   [Paste the ten documents]

   Here's a set you could use.

   Five test questions (example)

   Question	Correct document ID
   How far ahead can I book a meeting room?	D03
   What's the cap on hotels when I travel?	D04
   When do I have to say I'm working from home?	D02
   When are my P-card expenses due?	D05
   Is there a parking allowance?	none

   Check it

   • If you have real questions from staff, picking test questions from them is the best test of all. Strip out personal data first.
5. 5

   Score search and answers separately, then fix

   Record test results separately for search and for answers. If the right document isn't in the top three, no prompt change can make the answer right; only when search was right and the answer wrong is it time to fix the prompt.

   In this example, staff say "P-card" while the document says "work card," so search missed it. Embeddings are good at matching similar meanings but can miss shorthand and unique names. Add the other names to the document, as in "work card (P-card, company card)", and as your documents grow, consider adding keyword search (BM25) alongside embeddings.

   Results for the five test questions (example)

   Question	Correct	Search (top 3)	Answer
   How far ahead can I book…	D03	✓ (1st)	✓
   What's the cap on hotels…	D04	✓ (1st)	✓
   When do I have to say…	D02	✓ (2nd)	✓
   When are my P-card…	D05	✗ (not in top 3)	✗ (said it couldn't find it)
   Is there a parking allowance?	none	n/a	✓ (said it couldn't find it)
   Total		3 of 4	4 of 5

   Check it

   • Add the other names to D05, re-create the document embeddings and rerun the same five questions: search should go up to 4 of 4.
   • Rerun the full set of test questions every time you change a document, to make sure the fix didn't break another question.