Step by Step
B
BERT — encoder-only, bidirectional understanding
BERT sees the entire input sequence simultaneously (bidirectional), trained via masked language modeling, making it excellent for classification, named entity recognition, and question answering — but it cannot generate new text.
Example: BERT being used to classify whether a movie review is positive or negative, leveraging its bidirectional understanding of the full review text.
G
GPT — decoder-only, autoregressive generation
GPT sees only the left (preceding) context at each step, trained via next-token prediction, making it excellent for text generation, creative writing, code, and conversation.
Example: GPT generating a story one word at a time, only ever seeing the words that came before the current point in the generation.
T5
T5 / BART — encoder-decoder, best for transformation tasks
Encoder-decoder models like T5 and BART combine both approaches, making them particularly well suited for tasks that transform one sequence into another, like translation and summarization.
Example: using T5 to translate an English sentence into French, leveraging both its encoder's understanding of the source sentence and its decoder's generation of the target sentence.
Applied Walkthrough
1
A team needs to build a system that classifies customer support tickets into categories, and a separate system that generates helpful email responses.
2
For the classification task, BERT's bidirectional, encoder-only architecture is well suited, since it excels at understanding tasks like classification but cannot generate new text.
3
For the response-generation task, GPT's decoder-only, autoregressive architecture is the appropriate choice instead, since it excels specifically at generating coherent new text.
4
If the team instead needed to translate support tickets from one language to another, an encoder-decoder model like T5 would be the natural choice, combining understanding and generation in one architecture.
Exam Application
Exams test whether you can match the correct architecture (BERT, GPT, or T5/BART) to a described task, based on whether it requires understanding only, generation only, or transformation from one sequence to another.
⚠ Common Trap
The most common trap is assuming BERT and GPT are interchangeable general-purpose Transformer models. BERT fundamentally cannot generate text (it's encoder-only, bidirectional), and GPT cannot see the right/future context when processing a given position (it's decoder-only, autoregressive) — each is architecturally suited to different task types.
✓ Quick Self-Check
1. Can BERT generate new text?
No — BERT is encoder-only and bidirectional, excellent for understanding tasks but unable to generate text.
Tap to reveal / hide
2. What context can GPT see when generating a token at a given position?
Only the left (preceding) context — GPT is autoregressive and decoder-only.
Tap to reveal / hide
3. What kind of tasks is BERT particularly well suited for?
Classification, named entity recognition, and question answering.
Tap to reveal / hide
4. What kind of tasks is GPT particularly well suited for?
Text generation, creative writing, code generation, and conversation.
Tap to reveal / hide
5. What architecture type is best suited for translation and summarization, and why?
Encoder-decoder models like T5 or BART, since they combine both understanding (encoder) and generation (decoder).
Tap to reveal / hide