How does a model read text?

1How does a model read the words you type?

A model can't read words. It only works with numbers, so your text has to be turned into numbers first.

• 🧮Inside, a model works only with numbers. Mostly it multiplies them and adds them up. 1 × 3 + 2 × 1 = 5, like the layers in lesson 2.
• ✂️So the text is cut up first. A tokenizer is a separate program that cuts your text into pieces. I have a pet becomes 4 pieces: I, have, a, pet.
• ␣On the pieces in this lesson, the mark ␣ means a space. The space is part of the piece that comes after it. ␣pet is one piece: a space, then the word pet.
• 🔢Then the tokenizer swaps each piece for a number. ␣pet becomes 5716.

2What is a token?

A token is one piece of text that a model reads. Each token has a number, and every piece with its number sits in one fixed list.

• 🧩Token: a piece of text. It can be a whole word, part of a word, or a mark like a comma. ␣pet is a whole word. nin is part of a word: the typo runnin is cut into run and nin. The full stop is a mark.
• 🔢Token id: the number for one token. In GPT-4o's tokenizer, ␣dog is 6446.
• 📖Vocabulary: the list of every token the tokenizer knows, each with its id. GPT-4o's vocabulary has about 2 lakh tokens (200,000).

Two things to know about token ids:

• 🔒The list never changes, so the same piece always gets the same id, in every sentence, for every user. ␣dog is 6446 in I have a pet dog. and also in My dog is big.
• 🤷The id says nothing about what the word means. ␣Dog with a capital D is 18018. Nothing in 6446 and 18018 says they are the same animal. The model has to learn that from text.

3Why are tokens pieces of words, not letters or whole words?

Pieces of words are the middle choice between single letters and whole words. Letters make every sentence a very long list, and whole words leave too much out.

• 🔤If every letter were a token, sentences would become very long lists. The word dog would be 3 tokens (d, o, g) instead of 1.
• ❓And a single letter means almost nothing on its own. The letter t could belong to tea, cat or tiger.
• 📕If every whole word were a token, no list could hold every word, name and typo. A word missing from the list could not be read at all. The typo runnin is in no dictionary.
• 🧩Pieces fix both problems. A common word is one token. Any other word is built from smaller pieces that are on the list. running is one token. The typo runnin is built from two: run + nin.

4Who decides how text is cut into pieces?

A separate program makes the list of pieces, before the model is trained. The program looks through a big pile of text. It finds the two pieces that sit side by side most often, and joins them into one new piece.

1. 1️⃣Start with single letters. sing is s, i, n, g.
2. 2️⃣Count how often each pair of pieces sits side by side in the pile. In the pile sing ring win pin, i and n sit side by side 4 times.
3. 3️⃣Join the most common pair into one new piece, and add it to the list. in joins the list.
4. 4️⃣Count again and join again, until the list is as big as its makers chose. Next, in and g sit side by side 2 times, so ing joins the list.

• 🔗A model does not choose its own tokens. It gets the list before training starts, and it must keep using that same list. GPT-4o learned that 6446 is ␣dog. With another list, 6446 would be some other piece, and what it learned about 6446 would be wrong.

5Why does the same sentence cost more in Hindi?

AI companies charge for each token, not for each word. Hindi usually needs more tokens than English to say the same thing, so the same sentence costs more in Hindi.

• 💰You pay for each token. I have a pet dog. has 5 words but 6 tokens, so you pay for 6 tokens.
• 📊The list was built by counting a big pile of text (section 4). Pieces that showed up often became whole tokens. Text that showed up less often gets cut into smaller pieces. The pile had less Hindi text than English, so Hindi words get cut into more pieces. ␣pet is 1 piece, but ␣पालतू (pet) is 3.

A token id is only a number: 6446 says nothing about what a dog is. So how does a model know that cat and dog are alike, while cat and car are not? Lesson 4 starts there.

← Lesson 2  ·  All lessons  ·  Next lesson →