From phone autocomplete to ChatGPT

1Does your phone guess your next word?

Yes. While you type, your phone shows three words above the keyboard. Those three words are its guesses for your next word.

• ⌨️You type a few words. I am going to
• 💡The phone shows its three guesses for the next word. be · the · work
• 👆You tap one. The phone adds it and guesses again. Tap the, and it offers market · office · gym.

Now play a game. Tap only the middle word, again and again, and type nothing yourself.

• 🔁It goes round in circles. I am going to the office and I am going to the office…
• 🧩Each word fits the word just before it. the office is fine. office and is fine.
• ❌But the whole sentence says nothing. You are never told where you are going, or why.

So the phone does guess. It just guesses badly, because it looks at very little of what you typed. Section 5 shows this.

2How does a program guess anything?

A program guesses with a model.

A model is a program that looks at what has happened so far and makes its best guess about what comes next.

Example: a weather model. It makes the forecast in your phone's weather app.

Is the guess always right? No. That is why the app says 70% chance of rain, and not it will rain.

3Can a model guess words?

Yes.

A model that guesses the next word is called a language model.

Try it yourself first: fill in the blank in I have a pet ___.

• 🐶You probably said dog or cat. Not lion.
• 🤔Nobody gave you a rule for this. You have seen "pet dog" and "pet cat" many times, and "pet lion" almost never.
• 🦁A pet lion is possible. It is just rare.

A language model learns the same way:

• 📚It reads a huge amount of text. GPT-3 read web pages, books and Wikipedia.
• 📊It notices which words often come after which. After "pet", it saw "dog" again and again.
• 🎯Give it some words, and it gives every possible next word a chance. dog 38%, cat 31%, lion less than 1%.

4How does ChatGPT write whole answers?

ChatGPT writes one word at a time. After each word, the whole text so far goes back into the model, and the model guesses the next word.

Each round has four steps:

• 📥The whole text so far goes into the model. The cat sat on the
• 📊The model gives every possible next word a chance. mat 34%, floor 18%, sofa 14%
• ✅One word is picked and added to the end. Here the top word is always picked. Lesson 6 shows how a lower word sometimes wins. "mat" makes The cat sat on the mat
• 🔁The whole new text goes back in, and the next round starts. The cat sat on the mat goes in, and the model picks "and".

When does it stop? Besides words, the model can also pick a stop mark. On this page: ⏹ end = the model's stop mark. When the model picks ⏹ end, it stops writing.

• 📱This is the same loop you did on the phone in section 1. Each tap added one word, and the phone guessed again.
• 🔢A long answer takes many rounds. A 300-word answer takes a few hundred rounds.

This loop is what people mean when they say an LLM "generates text".

LLM stands for large language model.

It is called large for two reasons:

• 📚It read a huge amount of text. GPT-3 read web pages, books and Wikipedia.
• 🔢It holds billions (hundreds of crores) of numbers that it learned while reading. Lesson 2 explains what these numbers are. GPT-3 holds 175 billion (17,500 crore).

5Why does the phone go round in circles when ChatGPT doesn't?

The phone keyboard and the model inside ChatGPT both guess the next word. The biggest difference is how much of your text each one reads before it guesses.

Look at this sentence, and guess the last word yourself.

The LLM could do this because of three changes:

• 🧠A better design (2017). It is called the Transformer. The model looks at all the earlier words and pays most attention to the ones that matter. In the sentence above, it paid attention to "dog", at the very start.
• 📈Much bigger (2020). GPT-3 holds 175 billion learned numbers (17,500 crore). If you wrote down one of those numbers every second, day and night, it would take you about 5,500 years.
• 💬
  Taught to reply (2022). OpenAI trained the model more on example questions with good answers. It was shown thousands of pairs like "Q: What is 1 + 1? A: 2" and "Q: What is the capital of India? A: New Delhi", each with a different question.

  • 👍People also marked which of two answers was better. Answer A says "2". Answer B says "1 + 1 + 1". A person marks A as better.
  • 🚀Then OpenAI put the model in a chat app, ChatGPT. One million people (10 lakh) used ChatGPT in its first five days.

And your phone today?

• 📱Today's keyboards do use small language models. Since iOS 17 (2023), the iPhone keyboard uses a Transformer.
• 🔋The model has to run on the phone itself. So it stays small and reads only a little of what you type. It must fit in the phone's memory next to your photos and apps, and show its guesses while you are still typing.

6Is ChatGPT just GPT-3 with a chat box?

No. GPT-3 is a model, and ChatGPT is an app. GPT-3 is like an engine. ChatGPT is like a whole car. The engine inside ChatGPT is newer, and it was trained more to reply.

• ⚙️GPT-3 (2020) is a model. Developers sent it text from their own code, and it sent back more text. A developer could send "Once upon a time" and get back the next lines of a story.
• ➡️GPT-3 continues your text, like autocomplete. It does not answer you.

Send the same question to both, and compare what comes back.

• 🔢Why does GPT-3 write "2." and "3."? To it, your question looks like question 1 in a list of maths questions. So it writes the next questions in the list. Question 2 is "What is 1 + 1 + 1?"
• 💬ChatGPT (30 November 2022) is an app: the chat box, your account, your old chats. You type "What is 1 + 1?" and it replies "2".
• 🧩Inside ChatGPT is a model trained to reply. In 2022, that model was built from GPT-3.5, a newer version of GPT-3.

The models keep getting newer, but the job is the one your phone does: guess the next word, again and again.

One question is still open: how does a model learn which word is likely in the first place? The next lesson starts there.

All lessons  ·  Next lesson →