Step by Step
T
Text in — the starting point
Before any NLP model can process text, that raw text must first be broken down into smaller pieces the model can actually work with.
Example: the raw sentence "I love unhappiness" as the starting input before any processing begins.
O
Output tokens — the Lego bricks
Tokenization splits text into tokens, which can be whole words, subwords, or even individual characters, depending on the specific tokenizer — like breaking a structure down into its individual Lego bricks.
Example: the word "unhappiness" might become three separate subword tokens: "un," "happy," "ness."
K
Keeping rare words manageable — why subwords matter
Subword tokenization specifically handles rare or entirely new words gracefully, since even an unfamiliar word can be broken down into smaller, more familiar subword pieces.
Example: a brand-new made-up word can still be tokenized by breaking it into recognizable subword pieces, rather than failing outright.
E
Everything measured in tokens — context, cost, limits
Once tokenized, everything about how an LLM operates — context window size, API costs, and various limits — is measured in tokens, not in words or characters.
Example: a typical paragraph translating to roughly 100-150 tokens, with API pricing and context window limits both calculated based on this token count rather than word count.
Applied Walkthrough
1
A user wants to understand why a paragraph they wrote costs a certain amount to process through an LLM API.
2
The paragraph first gets tokenized — broken down into Lego-brick-like tokens, which might be whole words for common terms or subword pieces for rarer words.
3
This same paragraph, once tokenized, translates into roughly 100-150 tokens, and it's this token count — not the raw word or character count — that determines the actual API cost and how much of the context window it occupies.
4
This is why understanding tokenization matters practically: every cost, limit, and context window discussion around LLMs is fundamentally about tokens, not the more intuitive units of words or characters.
Exam Application
Exams test whether you understand tokenization as the essential first step before any NLP model can process text, and whether you know why subword tokenization specifically helps handle rare and novel words gracefully — a key advantage over whole-word-only tokenization schemes.
⚠ Common Trap
The most common trap is assuming tokens are the same thing as words. A single word can become multiple tokens (like "unhappiness" splitting into three subword tokens), meaning token counts and word counts are meaningfully different — this distinction matters directly for cost and context window calculations.
✓ Quick Self-Check
1. What is tokenization, and when does it happen in the NLP process?
Splitting text into tokens (words, subwords, or characters); it's the essential first step before any NLP model can process text.
Tap to reveal / hide
2. Why does GPT use subword tokenization (BPE) rather than whole-word tokenization?
Because it handles rare and new words gracefully, breaking them into smaller, more familiar subword pieces rather than failing outright.
Tap to reveal / hide
3. How many tokens might "unhappiness" become under subword tokenization?
Three tokens: "un," "happy," "ness" (as one illustrative example).
Tap to reveal / hide
4. Roughly how many tokens does a typical paragraph translate to?
Roughly 100-150 tokens.
Tap to reveal / hide
5. What aspects of using an LLM are measured in tokens rather than words?
Context window size, API costs, and various usage limits.
Tap to reveal / hide