Lesson 6

How does a model pick the next word?

How the model's chances become one word, why the same question can get a different answer, and the settings that change the pick.

1What does a model give before a word is written?

A chance for every token it knows. All the chances add up to 100%.

📊The model gives every token in its list a chance of coming next. After The cat sat on the: mat 34%, floor 18%, sofa 14%, as in lesson 1.
💯All the chances add up to 100%. GPT-4o's tokenizer has about 2 lakh tokens (200,000). Each one gets a chance, so most of them get a very small one.
🌈No token gets 0%. An odd word gets a very small chance, not none. After The cat sat on the, rainbow still gets a very small chance.
🧪In this lesson, we pretend the model knows only 8 words, so you can see every chance. mat 34%, floor 18%, sofa 14%, lap 12%, laptop 9%, roof 7%, moon 4%, rainbow 2%. 34 + 18 + 14 + 12 + 9 + 7 + 4 + 2 = 100.

2How is one word picked from the chances?

By a random draw. A small program around the model draws a random number, and a word with a bigger chance owns more of the numbers.

🤖The model only gives the chances. It does not pick a word. It hands back mat 34%, floor 18%, sofa 14% and the rest, and stops.
💻A small program around the model does the picking. Then it adds the word to the text and runs the model again, as in lesson 1. It picks mat, and The cat sat on the mat goes back into the model.
🎟️Each word owns a range of the numbers from 1 to 100, as wide as its chance. mat has 34%, so it owns 34 numbers: 1 to 34. floor owns the next 18 numbers: 35 to 52.
🎲The program draws a random number from 1 to 100. The word that owns that number is picked. The program draws 47. 47 is in floor's range, so floor is picked, even though mat has the bigger chance.
🔁Over many draws, each word is picked about as often as its chance. In 100 draws, mat is picked about 34 times and rainbow about 2 times.
❓So the same question can get a different answer. Each time, the program draws a new number. The first time it draws 12, so mat. The next time it draws 47, so floor.

Which numbers from 1 to 100 each word owns, after The cat sat on the

Made-up chances, for a model that knows only these 8 words, so they add up to 100%. The bar shows where each word's numbers sit between 1 and 100. Each press of "Draw a number" draws a new random number. Real programs draw a number with decimals between 0 and 1. Whole numbers from 1 to 100 work the same way here, because every chance is a whole percent.

3What does temperature do to the chances?

Temperature is a number you set that changes the chances before the draw. Below 1, the top word's chance grows and the smallest chances shrink. Above 1, the chances move closer together.

1️⃣Temperature 1 leaves the chances as the model gave them. mat 34%, floor 18%. mat is about 1.9 times floor.
⬇️Temperature 0.5: the program multiplies each chance by itself, then divides each result by the total of all 8, so they add up to 100% again. mat 34 × 34 = 1156. floor 18 × 18 = 324. mat is now about 3.6 times floor. After dividing: mat 58.7%, floor 16.4%.
⬆️Temperature 2: the program takes the square root of each chance, then divides each result by the total of all 8, so they add up to 100% again. The square root of 34 is 5.83, because 5.83 × 5.83 is about 34. mat 5.83, floor 4.24. mat is now only about 1.4 times floor. After dividing: mat 22.1%, floor 16.1%.
0️⃣Temperature 0: the program skips the draw and always takes the top word. mat every time, so the same question gets the same answer every time.
🧊Use a low temperature when the answer should not change from one try to the next. A reply to an email, or a fact.
🎨Use a high temperature when different answers help. Odd words are then picked more often, and some of them will not make sense. Ideas for the name of a new chai shop.

Chance that each word is picked after The cat sat on the, at 4 temperatures

Step 1 of 4

The chances at temperature 1 are the made-up ones from section 2. The other numbers are worked out from them by this page's script, then rounded to one decimal so they still add up to 100%. We show 0.5 and 2 because their maths is short: multiply by itself, and square root.

4How do top-k and top-p cut the list?

Top-k and top-p remove the words with the smallest chances before the draw. A removed word can't be picked at all.

Top-k keeps only the k words with the biggest chances. Top-p adds up the chances from the biggest down until the total reaches p, keeps those words and drops the rest.

✂️Top-k: you choose a number k, and only the top k words stay. k = 3 keeps mat, floor and sofa.
➕Top-p: you choose a percentage p. The program adds up the chances from the top until the total reaches p. p = 90%. The running total goes 34, 52, 66, 78, 87, 94. It passes 90% at roof, so mat to roof stay, and moon and rainbow are dropped.
🔢The words that stay get new chances that add up to 100% again, and then the program draws. With k = 3, the kept chances add up to 34 + 18 + 14 = 66. mat gets 34 out of 66, about 52%.
🛡️This stops a very unlikely word from being picked. With top-p 90%, rainbow can never be picked.

Which words stay, after The cat sat on the

wordchancerunning totaltop-k, k = 3top-p, p = 90%

The made-up chances from section 2. "Running total" adds each chance to the ones above it. Green "kept" means the word can still be picked. A crossed-out "cut" means it can't. The last lines show the kept words' new chances, rounded so they add up to 100%.

5Can you change temperature, top-k and top-p yourself?

Only in some tools. Temperature, top-k and top-p are settings of the program that does the picking, not part of the model.

🎛️They are settings, not parameters. Parameters are the dials the model learned in training (lesson 2). Setting temperature to 0.5 changes no dial inside the model. Only the draw changes.
🧰Hugging Face's transformers library, which runs open models, lets you set all three. You can ask for temperature 0.5 and top-p 90% in the same request.
🚫Anthropic's API, from its 21 October 2026 version, rejects any request that sets one of them, on every model. A request with temperature 0.5 gets an error (code 400) back, not an answer.

A model that only picks likely next words continues your text. Ask it What is 1 + 1? and it may continue with more questions, as in lesson 1. So how did a model that only continues text become an assistant that answers? Lesson 7 starts there.

← Lesson 5 · All lessons · Next lesson →

Where this comes from

A chance for every token in a fixed list: Hands-On Large Language Models (Alammar, Grootendorst; O'Reilly 2024), ch.3, PDF p.122-123: the model's last layer gives a probability score for every token in its vocabulary. How Large Language Models Work (Raff, Farris, Biderman; Manning 2025), ch.3, PDF p.63: "For each token in the vocabulary, compute the probability". The ladder from mat to rainbow follows Rahul's course blueprint (Module 1.6, Part 5).
About 2 lakh tokens in GPT-4o's tokenizer: o200k_base, the tokenizer behind GPT-4o, checked with OpenAI's tiktoken 0.14.0 for lesson 3 (about 200,000 pieces).
The model gives chances; a small program around it picks a word and runs the model again: Hands-On Large Language Models, ch.3, PDF p.118-120: "it actually generates one token at a time"; "Software around the neural network basically runs it in a loop"; PDF p.125: the decoding strategy is "the method of choosing a single token from the probability distribution". Rahul's blueprint, Part 3, Doubt 2.
Weighted dice; why not always the top token: How Large Language Models Work, ch.3, PDF p.63-64: "Randomly pick a token according to the probabilities calculated", shown as a weighted die. Hands-On Large Language Models, ch.3, PDF p.125-126: always taking the top token (greedy decoding) is what temperature 0 does, and it "doesn't tend to lead to the best outputs for most use cases"; a token with 40% is picked 40% of the time.
Settings, not parameters: Rahul's blueprint, Module 4.1 and Part 4, misconception 3 ("Temperature is a model parameter" is wrong). Parameters are the learned numbers from lesson 2.
Temperature: low sharpens, high flattens, 0 takes the top token; use cases: Hands-On Large Language Models, ch.6, PDF p.240-241 (temperature "defines how likely it is to choose tokens that are less probable"; 0 gives the same response every time; figure 6-4) and table 6-1, PDF p.242-244 (brainstorming high, email generation low). How Large Language Models Work, ch.3, PDF p.65-66.
Top-k and top-p: Hands-On Large Language Models, ch.6, PDF p.241-242: top_p "will consider tokens until it reaches their cumulative probability"; top_k keeps exactly k tokens, "e.g. the top 100". How Large Language Models Work, ch.3, PDF p.65: top-p sampling avoids the near-zero tokens.
Who lets you set them: Hands-On Large Language Models, ch.6, PDF p.240-242: Hugging Face transformers pipelines take do_sample, temperature and top_p. Anthropic's Claude API reference (claude-api skill, cached 2026-10-06): from API version 2026-10-21, a Messages request that sends temperature, top_p or top_k returns an error on every model; the newest models already reject them.
"What is 1 + 1?" from a text-finishing model: Hands-On Large Language Models, ch.12, figure 12-5 (PDF p.490).
Made up on purpose: the 8-word list and its chances after The cat sat on the (mat 34, floor 18, sofa 14 as in lesson 1; lap 12, laptop 9, roof 7, moon 4, rainbow 2), the draws 47 and 12, and the example uses (an email reply, names for a chai shop). The numbers 1 to 100 are a simplification of the real draw, which picks a number with decimals between 0 and 1; with whole-percent chances the two give the same picks. The temperature maths is real: each chance is raised to the power 1/temperature and the results are scaled back to 100% (the same as softmax of the scores divided by the temperature, when a score is ln(chance)), so 0.5 squares each chance and 2 takes its square root. Every temperature, running-total and share number on the page is worked out by the page script and was checked with a separate script.