Step by Step
T
Tokenize — splitting raw text into subwords
Raw text is split into tokens, typically subwords, the fundamental units the rest of the pipeline operates on.
Example: splitting "unhappiness" into subword tokens like "un," "happy," "ness."
E
Embed — mapping tokens to dense vectors
Each token is mapped to a dense vector representation, capturing some initial notion of its meaning.
Example: converting the token "happy" into a dense vector of several hundred numbers.
E2
Encode — building contextual representations (BERT-style)
Tokens are run through Transformer layers to build rich, contextual representations that account for surrounding context, similar to how BERT processes text.
Example: encoding a full sentence so that each token's representation now reflects the meaning of its surrounding context, not just its isolated meaning.
D
Decode — generating output tokens one at a time (GPT-style)
For generative tasks, output tokens are generated one at a time, similar to how GPT-style models produce text.
Example: generating a response one token at a time, with each new token conditioned on everything generated so far.
G
Generate — applying a sampling strategy
A specific sampling strategy (greedy, beam search, or temperature-based sampling) is applied to actually select each output token from the model's predicted probability distribution.
Example: applying top-P nucleus sampling with a specific temperature setting to select each generated token, rather than always greedily picking the single most likely one.
Applied Walkthrough
1
A user reports two different problems with an NLP system: garbled, nonsensical output in one case, and repetitive, overly predictable output in another.
2
Ask: which pipeline stage is most likely responsible for the garbled output? Likely the tokenizer stage, since a tokenizer problem could scramble how text gets broken into meaningful units in the first place.
3
Ask: which pipeline stage is most likely responsible for the overly repetitive output? Likely the decoding/generation stage, since a decoding strategy like pure greedy decoding tends to produce repetitive text.
4
This diagnostic approach — mapping specific symptoms back to specific pipeline stages — is exactly why understanding this five-stage pipeline in detail is so valuable for troubleshooting real NLP system problems.
Exam Application
Exams test whether you can name and describe all five pipeline stages in order (tokenize, embed, encode, decode, generate) and whether you can diagnose which stage is likely responsible for a described failure symptom (garbled output → tokenizer; poor coherence → decoding strategy).
⚠ Common Trap
The most common trap is conflating the "encode" and "decode" stages, assuming they're the same step just described twice. Encode (BERT-style) builds contextual understanding of INPUT tokens; decode (GPT-style) generates OUTPUT tokens one at a time — these are distinct stages serving different purposes within the overall pipeline.
✓ Quick Self-Check
1. What happens during the Tokenize stage?
Raw text is split into tokens, typically subwords.
Tap to reveal / hide
2. What happens during the Embed stage?
Each token is mapped to a dense vector representation.
Tap to reveal / hide
3. What happens during the Encode stage, and what model style does it resemble?
Tokens are run through Transformer layers to build contextual representations, similar to BERT.
Tap to reveal / hide
4. What happens during the Decode stage, and what model style does it resemble?
Output tokens are generated one at a time, similar to GPT.
Tap to reveal / hide
5. If a system produces overly repetitive output, which pipeline stage is most likely responsible?
The decode/generate stage, likely due to the specific sampling strategy being used (like pure greedy decoding).
Tap to reveal / hide