Lesson 2
How does a model learn?
What a parameter is, how training turns billions (hundreds of crores) of them, and what a model keeps once training is over.
1How does a model learn which word comes next?
A model learns by guessing and being corrected. It is shown real sentences, and it has to guess each word before it sees it.
- 📚The model is shown a real sentence. I have a pet dog
- 🙈The last word is hidden, and the model guesses it. It sees I have a pet and gives a chance to every word that could come next.
- 👀Then it is shown the real word. dog
- 🔧If it gave the real word a low chance, numbers inside the model change. They change so that dog gets a higher chance next time.
This guessing and correcting is called training.
The numbers that change are called parameters. Section 2 shows what they are.
2What is a parameter?
A parameter is a number inside a model that changes while the model learns.
Example: the chai model below has 2 parameters, its guess for the price of 1 cup of chai and its guess for the price of 1 plate of samosa.
Think of each parameter as a dial. Turn it up or down, and the model's answer changes.
The chai model guesses the bill at a chai stall:
- 📥Input: the order, and only the order. Nobody tells the model any prices. 2 cups chai, 3 plates samosa.
- 🎛️Dials: the model's two guesses. Dial A is its guess for 1 cup of chai. Dial B is its guess for 1 plate of samosa. At the start, both dials are at ₹5.
- 📤Output: its guess for the bill. It multiplies the cups by dial A, multiplies the plates by dial B, and adds the two. 2 × ₹5 + 3 × ₹5 = ₹25.
- 📏Check: the stall's real bill for that order. The model sees it only after it has guessed. The real bill for 2 chai and 3 samosa is ₹70.
A made-up stall with made-up prices. Prices and bills are in rupees. We picked the size of each turn so the sums stay easy. Real training works out the size of each turn with maths.
- 🤫Nobody told the model the prices. It was only shown how far off each bill was. Round 1 said "too low by ₹45", not "chai costs ₹20".
- 🧾One order was not enough. Two prices need orders with different mixes of chai and samosa. Round 3 matched order 1 with the wrong prices, ₹14 and ₹14.
These two dials are the model's parameters. Training turned them from ₹5 and ₹5 to ₹20 and ₹10.
You will also hear parameters called weights. Both words mean these learned numbers. Example: "a model with 8 billion weights" means a model with 8 billion (800 crore) parameters.
3How does training turn the dials?
Training turns the dials by repeating three steps, again and again: make a guess, check how wrong it is, and nudge the dials. The chai model in section 2 worked the same way: it guessed a bill, checked it against the real bill, and turned its dials.
- 🎯Make a guess. The model is shown I have a pet. Partway through training, its top guess for the next word is "lion".
- 📏Check how wrong. The training sentence is I have a pet dog. The real next word is "dog", so "lion" is wrong.
- 🔧Nudge the dials a little. Each dial moves a tiny bit, in the direction that gives "dog" a higher chance next time. A dial that did not affect the guess does not move, like dial B in chai round 4.
What is the guess checked against?
- 🧾In the chai model: the real bill. ₹70.
- 📖In a language model: the word that really comes next in the training sentence. "dog" in I have a pet dog.
At the start the dials are random, so the guesses are nonsense.
- 🎲Before any training, the model's top guesses make no sense. I have a pet ___ → "the", "banana", "seven".
- 🤏Every word gets only a tiny chance. The model knows a huge number of words, and all their chances must add up to 100%, so each one gets a tiny share. "dog" gets about 0.01%.
One nudge barely changes anything. But the nudges add up.
- 0️⃣Before training: "dog" gets 0.01%.
- 1️⃣After 1 nudge: "dog" gets 0.012%.
- ♾️After billions of nudges (hundreds of crores): "dog" gets 38%.
These numbers are made up, to show the idea. The nudges add up because the model goes through all of its training text, and every word in that text is one more guess to check.
After training, the guesses make sense:
- 🐶"dog" and "cat" get a high chance after I have a pet. dog 38%, cat 31%.
- 🦁"lion" gets a low chance. lion less than 1%.
- ✍️Nobody wrote a rule saying pets are dogs or cats. It came out of the nudges.
4How many parameters does a real model have?
Billions (hundreds of crores). Llama 3, a model Meta released in 2024, comes in two sizes: one with 8 billion parameters (800 crore) and one with 70 billion (7,000 crore).
The dials sit in layers:
- 🔢Your words go into the model as numbers. Lessons 3 and 4 show how. Two numbers: 1 and 2.
- ✖️Each box in a layer multiplies those numbers by its own dials and adds them up. Box 1: 1 × 3 + 2 × 1 = 5. Box 2: 1 × 2 + 2 × 4 = 10.
- ➡️The results go into the next layer, which does the same with its own dials. 5 × 2 + 10 × 1 = 20.
- 🕸️Dials arranged in layers like this are called a neural network.
Your words go in as two numbers, 1 and 2. The layers have not worked yet.
Layer 1: each box multiplies 1 and 2 by its own two dials and adds. Out come 5 and 10.
Layer 2 takes 5 and 10, multiplies by its two dials and adds. Out comes 20. This network has 4 + 2 = 6 dials.
The numbers on orange are the dials, the parameters. Made-up numbers. A real layer does a little more than multiply and add, and it has far more boxes.
This tiny network has 6 dials. Llama 3 has 8 billion (800 crore) or 70 billion (7,000 crore), in many more layers.
A model has parameters. It is trained on text.
- 🎛️A model has parameters. They are its dials, and their count is the model's size. The chai model has 2. The small Llama 3 has 8 billion (800 crore).
- 📚A model is trained on text. That is what it reads while its dials are turned. The chai model was trained on 3 orders. Llama 3 was trained on over 15 trillion (15 lakh crore) tokens (pieces of words; lesson 3 explains).
- ❌"Trained on billions (hundreds of crores) of parameters" is a common mistake. It mixes the two up. Say instead: Llama 3 has 70 billion parameters (7,000 crore) and was trained on 15 trillion tokens (15 lakh crore).
5Are the chances for the next word stored in the model?
No. A model stores only its parameters, the numbers it learned in training, and once training is over they stop changing. The chances for the next word are worked out fresh, each time, from the sentence you type.
- 📥Your sentence goes in. The cat sat on the
- 🎛️It passes through all the dials, layer after layer. In the tiny network in section 4, the numbers 1 and 2 went in, and 20 came out.
- 📊A chance comes out for every word the model knows. All the chances add up to 100%. mat 34%, floor 18%, sofa 14%.
- 🔄Change the sentence and you get different chances from the same dials. The dog sat on the gives floor 29%, mat 16%.
The cat sat on the
Chance that each word comes next
- mat34%
- floor18%
- sofa14%
- lap9%
- rainbow<1%
The sentence goes into the model. Nothing is worked out yet.
The same dials work out a chance for every word that could come next.
The word with the highest chance goes into the blank.
Made-up chances. Only the top 5 words are shown. Tap the other sentence: same dials, different sentence, different chances.
The chai model works the same way. Its dials stay at ₹20 and ₹10, and every new order gets its own bill. Example: 1 chai and 4 samosas, an order it never saw, comes to 1 × ₹20 + 4 × ₹10 = ₹60.
6Does a model keep a copy of what it read?
No. The text a model was trained on is not part of the model. All that is kept is its parameters, the numbers it learned.
- 📦The small Llama 3 is a 16 GB download. 8 billion dials (800 crore) × 2 bytes for each dial = 16 GB.
- 📚It was trained on far more text than that. Over 15 trillion tokens (15 lakh crore), about 1,900 tokens for every dial.
- 🚫So there is no room for a copy. What is kept is the pattern that billions (hundreds of crores) of nudges built up.
Because the model keeps no copy:
- ✅It can handle sentences it has never seen. The chai model priced 1 chai and 4 samosas, an order it was never trained on.
- ⚠️It can make things up. There is no page to look up, so it writes the words that seem likely, and likely is not the same as true. Ask for the birthday of a person who is barely mentioned online. It may write a real-looking date that is wrong, and say it just as confidently as a right one.
- 🔁It can still repeat some text it saw many, many times. Give it the first line of a very famous text, like the opening of the Indian national anthem, and it can often write the next lines.
- 🔍But it is not a library you can search. Ask it for page 5 of a book it read. It cannot open the book and look. It can only guess words that seem likely.
Everything in this lesson works on numbers: dials multiply numbers and add them up. But you type words. How does a model read text? Lesson 3 starts there.
Where this comes from
- Training as guess, check, nudge: How Large Language Models Work (Raff, Farris, Biderman; Manning 2025), ch.4, PDF p.69: "Gradient descent—You need a mechanical process that tweaks the numeric values inside an algorithm to make the loss function score as small as possible." PDF p.73: the gradient is "the direction and amount that you need to change the parameters". PDF p.74: "Modern LLMs perform billions of parameter updates because they are trained on billions of tokens."
- Weights are the parameters, arranged in layers: Hands-On Large Language Models (Alammar, Grootendorst; O'Reilly 2024), ch.1, PDF p.28-29: "neural networks can have many layers where each connection has a certain weight depending on the input. These weights are often referred to as the parameters of the model."
- Llama 3, 8 billion (800 crore) and 70 billion (7,000 crore), over 15 trillion tokens (15 lakh crore): Meta, "Introducing Meta Llama 3" (18 April 2024): "pretrained and instruction-fine-tuned language models with 8B and 70B parameters"; "Llama 3 is pretrained on over 15T tokens that were all collected from publicly available sources." ai.meta.com/blog/meta-llama-3. The book gives the same scale: "LLM developers train models on 1 to 15 trillion tokens (e.g., Llama 3.1 used 15 trillion)" (How Large Language Models Work, ch.9, PDF p.174).
- 16 GB, 2 bytes per dial: Hugging Face, meta-llama/Meta-Llama-3-8B: 8,030,261,248 parameters stored as BF16 (2 bytes each), in four files totalling 16,060,556,376 bytes. huggingface.co/meta-llama/Meta-Llama-3-8B. 15 trillion (15 lakh crore) ÷ 8.03 billion (803 crore) ≈ 1,868 tokens per dial, rounded to 1,900.
- Not a copy, but some text word for word: Hands-On Large Language Models, ch.3, PDF p.134: "For an LLM to be successfully trained, it needs to memorize a lot of information. But it is not simply a large database." How Large Language Models Work, ch.4, PDF p.76-77: models are not trained to "Produce factually accurate text"; they are rewarded "for producing text that looks exactly like the training data".
- Made up on purpose: the chai stall, its prices, its orders and the size of each turn; the guesses and chances in sections 3 and 5; the numbers in the two-layer network. Each is labelled where it appears. All web pages accessed 2026-10-31.