Tokens
Why AI reads in little pieces, and why that matters for its limits.
AI doesn't read whole words the way you do. It breaks text into small chunks called tokens: a short word, part of a longer word, or a punctuation mark. As a rough rule, 100 tokens is about 75 English words.
How it works
- 01A tokenizer chops your text into small, reusable pieces.
- 02Each piece gets a number, because the model works with numbers, not letters.
- 03The model writes back one token at a time, which is why answers seem to type themselves out.
Breaking a gingerbread cookie into toddler bites
You don't hand a toddler a whole gingerbread cookie; you break it into small pieces they can manage. A tokenizer does the same to your text. Common words like "mom" usually stay whole. Long or unusual words, like "Kinderbetreuung", get cut into several bites. Every bite counts toward how much the AI can handle at once, and for paid tools, toward the bill.
Where you'll use it
Why German uses more
Long German words often split into more tokens than English ones, so the same message takes up more of the AI's space.
Understanding pricing
Many AI tools for business charge per token, so tighter prompts and shorter answers cost less.
The typing effect
Answers appear piece by piece because the AI is writing one token at a time.
Your nap-time experiment
Ask the AI to show you its own bites.
Explain tokens to me like I'm a tired mom at 9pm, in 3 sentences. Then show me roughly how you'd split this sentence into tokens, with a | between each one: "My daughter's Kinderbetreuung starts on Wednesday!" Add one line saying this is an estimate.