How does a chatbot answer from a 500-page PDF?

1Why not paste the whole PDF into every question?

Pasting the whole PDF is slow and costs a lot. A 500-page PDF is about 3 lakh tokens, and the model reads every one of them for every question.

• 📄The 3 lakh is our estimate, at about 600 tokens a page. 600 tokens × 500 pages = 3,00,000 tokens.
• 🔁The model remembers nothing between questions (lesson 8), so the app sends the whole PDF again with every question. 10 questions send the same 3 lakh tokens 10 times.
• 💸You pay for every token you send (lesson 8). On Claude Opus 5.5, 3 lakh input tokens cost $1.20. 1,000 questions a day cost $1,200 a day, for the input alone.
• 🐢The answer starts late. The model reads all 3 lakh tokens before it writes the first word.
• 🚫It may not fit in the context window (lesson 8). Four such PDFs are 12 lakh tokens. Claude Opus 5.5 holds 10 lakh.

2What is RAG?

RAG is a way to answer from your own documents: your code finds the few pieces that match the question and puts them into the prompt, next to the question.

You already do this with a textbook. To answer "What is photosynthesis?", you find it in the index, open page 112 and read one page. You do not read the whole book again.

RAG stands for Retrieval-Augmented Generation:

• 🔍Retrieval: find the pieces of text that match the question. From a policy handbook, find the part about returning phones.
• ➕Augmented: add those pieces to the prompt. The prompt now holds 3 short pieces of the handbook, then the question.
• ✍️Generation: the LLM writes the answer from what it was given. "You can return a phone within 7 days."

Adding the pieces is easy: your code joins texts together. The hard part is the search: which pieces match the question?

3How does RAG answer a question, step by step?

RAG works in two parts. The setup runs once, when you add the document. The second part runs for every question a customer asks.

First, two new ideas: one list of numbers for a whole text, and how close two lists are.

• 📋In lesson 4, every token had its own list of numbers. The token for cat had one list, and the token for dog had another.
• 📄RAG gives one list to a whole piece of text. This list is also called an embedding. Chunk 1 of the store's handbook, about returning electronics, gets one list, like [0.12, −0.48, 0.91, …] (made-up numbers, used again in step 3).
• 🧲Texts about the same thing get lists that are close. The question "What's the return policy for electronic items?" and a chunk about returning phones get close lists. A chunk about free delivery gets a list far away.
• 📊Lesson 4 measured a gap: a small gap means close. Many RAG tools give a closeness number instead: a bigger number means closer, and 1 is the highest. In step 6 below, the returns chunk has a closeness of 0.91 and the delivery chunk 0.22 (made-up numbers).

🛠️ Setup, once:

1. ✂️ Cut the document into small pieces, called chunks.
2. 🔢 Turn each chunk into one list of numbers, its embedding.
3. 🗄️ Store every list, with its chunk's text, in a database.

💬 Every question:

1. 🔢 Turn the question into one list of numbers, the same way.
2. 🗄️ Find the stored lists closest to the question's list.
3. 🧑‍💻 Paste those chunks into the prompt, with the question.
4. 🤖 The LLM reads the prompt and writes the answer.

In the example below, a customer asks an online store's chatbot about returns. Use the buttons to go through it one step at a time.

4Who cuts the PDF into chunks?

Your own code does, with a small program. The LLM is not involved.

• ✂️Cut by size or by paragraph. Every 500 tokens, or one chunk per paragraph of the handbook.
• 🔁Repeat a little text at each cut, the overlap, so a sentence cut at the edge is still whole in one chunk. The last 50 tokens of chunk 1 are also the first 50 tokens of chunk 2.
• 🧮The overlap decides how many chunks you get. With 500-token chunks and a 50-token overlap, each new chunk starts 450 tokens after the last one. 3,00,000 tokens make 667 chunks.
• ⚖️Keep chunks small, but not too small. One sentence alone loses its meaning ("They must be unopened": what must?). A whole chapter mixes many topics into one list.

A simple splitter in JavaScript, which cuts by words:

function split(text, size, overlap) {
  const words = text.split(" ");
  const chunks = [];
  for (let start = 0; start < words.length - overlap; start += size - overlap) {
    chunks.push(words.slice(start, start + size).join(" "));
  }
  return chunks;
}

console.log(split("a b c d e f g h i j", 4, 1));
// [ 'a b c d', 'd e f g', 'g h i j' ]

Each chunk is 4 words, and "d" and "g" appear twice: that is the overlap of 1 word. This splitter counts words to keep the example short. Real splitters usually count tokens, like the 500 above. The idea is the same.

5Who makes the embeddings, and who does the search?

A separate embedding model, usually a small one, makes the lists. A vector database does the search. The LLM does neither.

An LLM does have embeddings inside it, but they are not the lists RAG uses:

• 🤖Inside the LLM: one list per token, used by the LLM while it reads (lesson 4). The token for cat has its own list inside the LLM.
• 🔢For RAG: one list for a whole chunk, made by a separate embedding model. All of chunk 1, "Phones, laptops and headphones can be returned within 7 days…", becomes one list.

The five workers, and what each worker does:

• 🧑‍💻Your code runs everything and passes text between the others. It reads the PDF, calls the splitter, sends chunks to the embedding model, and builds the final prompt.
• ✂️The splitter, a function in your code, cuts the text into chunks. 500 pages become 667 chunks.
• 🔢The embedding model turns one piece of text into one list of numbers. Voyage AI's voyage-4-lite turns any chunk into a list of 1,024 numbers by default.
• 🗄️The vector database stores the lists and finds the closest ones. Given the question's list, it returns chunks 1, 4 and 2.
• 🤖The LLM reads the prompt your code built and writes the answer. It sees 3 chunks and one question, never the other 664 chunks.

More about the embedding model and the database:

• 🧩The embedding model often comes from a different company than the LLM. Anthropic makes Claude but has no embedding model. Its own docs say so and name Voyage AI instead.
• 🔗The chunks and the question must use the same embedding model. voyage-4-lite gives 1,024 numbers and the open model all-MiniLM-L6-v2 gives 384. Lists from two models cannot be compared.
• 🗄️A vector database is a database that stores lists of numbers and quickly finds the lists closest to a given list. Pinecone, Qdrant and Chroma are built for it. Many teams use Postgres with the pgvector add-on.

6Does RAG use a smaller model, and is the LLM the only cost?

Yes, the embedding model is usually much smaller than the LLM, because its job is easier. No, the LLM is not the only cost: the embedding model costs money too, but far less.

Why a small model is enough:

• 🪶Turning text into a list is easier than writing an answer. The embedding model never writes a polite reply. It only has to make close lists for texts about the same thing.
• 📥It returns one list, not words, so it has no output tokens (lesson 8). An LLM writing a 100-token answer needs 100 rounds, one after another. The embedding model needs one round.
• 🎛️Fewer dials means less work per token (lesson 8). all-MiniLM-L6-v2, an open embedding model whose size is published, has 2.27 crore (22.7 million) dials. Llama 3 8B has 800 crore (8 billion), about 350 times more.

What costs money:

• 🔢The embedding model costs money too, because it runs on a computer like any model. voyage-4-lite costs $0.02 per 10 lakh tokens. Claude Opus 5.5 input costs $4 for the same 10 lakh tokens.
• 1️⃣The PDF is embedded once, not every question. 667 chunks of up to 500 tokens are 3,33,300 tokens, with the overlap. 3,33,300 × $0.02 ÷ 10,00,000 = about $0.0067, one time.

• ➗RAG sends 200 times fewer tokens, so each question's input costs 200 times less. 3,00,000 ÷ 1,500 = 200, and $1.20 ÷ $0.006 = 200.
• ✍️The answer's output tokens cost the same in both columns, so the table does not include them. The same 100-token answer is written either way.
• 🙋Each question is embedded too, but questions are short. 1,000 questions of about 10 tokens are 10,000 tokens a day: 10,000 × $0.02 ÷ 10,00,000 = $0.0002.

7When does RAG give a wrong answer?

RAG gives a wrong answer when the search misses the right chunk. The LLM can only use what your code puts into the prompt, so a missed chunk means a missing fact.

• 📉The right chunk is not among the closest. The rule for tablets is in a chunk titled "Gadget care". It comes 5th, your code takes only the top 3, and the LLM never sees it.
• 🎣Search always returns something, even when nothing matches. "Do you sell cars?" still returns the 3 closest chunks, all about other things.
• ✋A minimum closeness lets your code say "I don't know". With a minimum of 0.5, if no chunk reaches 0.5, your code replies "I don't know" and does not call the LLM.
• 🔤A meaning search can miss exact words, like a model number. The handbook has one line about the headphones model BX-40912. A question about BX-40912 can return chunks about other headphones, because their lists are close.
• 🔎Real systems often add a search for the exact words, next to the meaning search. A search for "BX-40912" finds only the chunks with exactly those characters.

So far, a model only reads text and writes text. Can it actually do things, like check the weather or send an email? Lesson 10 starts there.

← Lesson 8  ·  All lessons  ·  Next lesson →