Tokens, explained simply
Before a language model can work with text, the text is chopped into pieces called tokens. A token might be a whole short word, part of a longer word, or a punctuation mark.
Each token is then turned into numbers. The model works with those numbers, not the letters you see.
Tokens explain some odd AI behaviour. A model might struggle to count the letters in a word, because it may see the word as one or two chunks rather than individual letters.
Example: Chopping up a sentence
- "Unbelievable dragons!" might become: "Un" + "believ" + "able" + " dragons" + "!"
- Five tokens, each turned into numbers for the model.
- Real models each have their own way of splitting, so the exact pieces vary.
Try this
- Split long words into chunks that could be reused, like "un", "happy" and "ness".
- Invent a secret code where every common word gets a number. How is that like tokens?
- Think about why an AI might find it hard to spell a word backwards.