How did a text-finisher become an assistant?

1What can a model do after reading the internet?

A model that has only read text can continue text. It cannot do anything else yet.

Pre-training is the first round of training: the model reads a huge amount of text and learns to guess the next word.

How pre-training works, with lesson 2's 🎯 guess, 📏 check and 🔧 nudge:

• 📥What goes in: text from web pages, books and more. Llama 3, a model from Meta, read over 15 trillion tokens (15 lakh crore).
• 🎯Guess: the model reads part of a sentence and gives every next word a chance. It reads I have a pet. At the start, dog gets only 0.01% (lesson 2's made-up numbers).
• 📏Check: the guess is compared with the word that really comes next in the text. The text says I have a pet dog, so the right word is dog.
• 🔧Nudge: the dials (parameters, lesson 2) move a tiny bit, so that dog gets a higher chance next time. One nudge moves dog from 0.01% to 0.012%. After all the nudges, dog has 38%, as in lesson 1.
• 🧩The trained model is called a base model. It continues any text you give it. Send it Once upon a time and it writes the next lines of a story.

A base model does not answer your question. It writes what usually comes after text like yours.

• 📄To the base model, your question looks like question 1 in a list of maths questions. It writes question 2, What is 1 + 1 + 1?, then question 3.
• ↪️It gives back more of your text. It does not give you the answer.

Two more rounds of training turn a base model into a chat model. Here are all three rounds:

1. Round 1📚 Read text

   Reads trillions of tokens (lakhs of crores) and learns to continue text.

2. Round 2💬 Learn to answer

   Trains on thousands of questions paired with good answers.

3. Round 3👍 Learn what people prefer

   Learns from people picking the better of two answers.

2How does a model learn to answer a question?

Instruction tuning is the second round of training: the base model trains on questions, each paired with a good answer.

It uses the same three steps as pre-training: 🎯 guess, 📏 check, 🔧 nudge. The example below is a training pair from lesson 1. Use the buttons to go through it one step at a time:

• 🔢Instruction tuning needs far less text than pre-training. Thousands of question-answer pairs here. Over 15 trillion tokens (15 lakh crore) for Llama 3 in section 1.
• 🧠The model still only guesses the next word. Only the chances changed. After What is the capital of India?, New now has a higher chance than 2.

3What does a chat look like to the model?

To the model, a whole chat is one long piece of text. Markers that you never see show who is speaking.

Here are the real markers of Phi-3-mini, the small open model from Microsoft that lessons 4 and 5 used.

<|user|>
What is 1 + 1?<|end|>
<|assistant|>
1 + 1 = 2.<|end|>

One short chat, written out the way the model reads it. The markers are Phi-3-mini's real ones. The question and the answer are our example.

What each marker means:

• 🙋<|user|> Your message starts here.
• 🤖<|assistant|> The reply starts here.
• ⏹️<|end|> A message ends here. It is the model's stop mark, like ⏹ end in lesson 1.

How one reply happens:

1. You type What is 1 + 1?
2. The chat app puts markers around it, and adds <|assistant|> at the end.
3. The model continues the text. After <|assistant|>, an answer is the likely next text.
4. Instruction tuning made it likely. Its training pairs were written with the answer after <|assistant|>.
5. When the answer is finished, the model writes <|end|>. The app stops asking for more words.
6. The app removes the markers and shows you only 1 + 1 = 2.

4How does a model learn which answers people prefer?

Preference tuning is the third round of training: people pick the better of two answers, and the model is nudged towards the kind of answer they pick.

Here is one common way to do it. Use the buttons to go through it one step at a time:

• 🔀Preference tuning is not the same as instruction tuning. Instruction tuning gives the model an answer to copy: New Delhi. Preference tuning only says which of two answers is better: A, not B.
• ✂️Some newer methods skip the scoring model. They use the pick, A over B, to nudge the chat model directly: answers like A become more likely, answers like B less likely.
• 🛑Most of a chat model's habit of saying no to harmful requests comes from this round. Asked how to use a neighbour's Wi-Fi without the password, people pick the answer that says no.
• ⚖️These two extra rounds can matter more than size. In a 2022 OpenAI test, people preferred the answers of a tuned model with 1.3 billion (130 crore) dials over GPT-3 with 175 billion (17,500 crore) dials, which had only pre-training.

5Should a company train a model on its own examples?

Sometimes. A company fine-tunes mostly to change how a model behaves, not what it knows.

Fine-tuning is more training for a model that is already trained, using your own examples.

How it works (the same way as instruction tuning in section 2):

• 📥You write requests, each with the answer you want. The card below: a support ticket, and the label billing.
• 🎯The model guesses an answer. It guesses refund.
• 📏The guess is checked against your answer. Your answer says billing, so refund is wrong.
• 🔧The dials are nudged. billing gets a higher chance after this ticket next time.

✅ Good for behaviour:

• 🗣️A fixed tone. Always short and polite, like your best support agent.
• 🏷️One narrow job. Sort support tickets into five types, like the billing example above.

❌ Bad for facts:

• 📅The model keeps old facts. Your return window changes from 7 days to 10. The tuned model still says 7 until you train it again.
• 🔍It can't show where an answer came from. Ask the tuned model how many days you have to return an item. It says 7, but it can't show you the policy page that says so. With RAG (section 6), the page itself is in the prompt.

6Which should you try first: a better prompt, RAG or fine-tuning?

Try a better prompt first, then RAG, then fine-tuning. RAG means your code finds the right pages in your own documents and puts them into the prompt. Each of the three takes more work than the one before.

1. ✍️ A better prompt. Say exactly what you want, and show an example. Example: Reply in 3 short points, like this: …
2. 📄 RAG, for your own documents. Example: the two pages about returns from a 500-page policy PDF. Lesson 9 shows how.
3. 🔧 Fine-tuning, for behaviour. Example: the ticket sorter from section 5.

Move to the next one only when the one before is not enough. The short rule: RAG for knowledge, fine-tuning for behaviour.

• 💻Knowledge that changes often belongs in the prompt, not in the dials. You want an assistant that knows your team's code. The code changes every day, so a fine-tuned model would soon hold old code. Coding tools like GitHub Copilot, Cursor and Claude Code read the right files and put that code into the prompt instead.

The training and every reply run on huge computers. Why does every token you send or get back cost money? The next lesson starts there.

← Lesson 6  ·  All lessons  ·  Next lesson →