Step by Step
B
Bag of Words — simple word counts
Represents text as simple word counts, completely ignoring word order and context, producing sparse vectors with no semantic understanding.
Example: representing a document purely by how many times each word appears, with no regard for the order those words appeared in.
T
TF-IDF — weighting rare, informative terms higher
Term Frequency-Inverse Document Frequency weights rare, domain-specific terms more heavily than common words, still producing sparse vectors but with more useful weighting than raw Bag of Words.
Example: a rare, specific term like "myocardial" gets weighted much more heavily than a common word like "the" under TF-IDF.
W
Word2Vec / GloVe — dense semantic vectors, fixed per word
Produces dense vectors capturing semantic meaning as geometric relationships (like king - man + woman = queen), but assigns exactly one fixed vector per word regardless of context.
Example: the word "bank" gets the exact same Word2Vec vector whether it refers to a riverbank or a financial institution.
B
BERT — contextual embeddings, meaning changes with context
BERT and similar contextual embedding models give the same word different vectors depending on its surrounding context, representing the current, dominant generation of text representation used throughout modern NLP.
Example: BERT assigns "bank" a different embedding vector when it appears near "river" versus when it appears near "loan" or "finance."
Applied Walkthrough
1
A sentiment classifier needs to correctly distinguish the meaning of "bank" in "I sat by the river bank" versus "I deposited money at the bank."
2
Using Bag of Words or TF-IDF, both sentences would simply register "bank" as the same word count entry, with no way to distinguish the intended meaning.
3
Using Word2Vec, "bank" would still get the exact same fixed vector in both sentences, since Word2Vec assigns one vector per word regardless of context.
4
Only BERT's contextual embeddings would correctly assign "bank" two different vectors in these two sentences, reflecting its genuinely different meaning in each context — which is exactly why modern NLP has moved to contextual embeddings.
Exam Application
Exams test whether you can place these four representation methods in their historical progression and correctly identify their key distinguishing property — especially the critical difference between static (Word2Vec/GloVe, one fixed vector per word) and contextual (BERT, different vectors depending on context) embeddings.
⚠ Common Trap
The most common trap is assuming Word2Vec and BERT work the same way, since both produce "dense" semantic vectors. Word2Vec assigns exactly one fixed vector per word; BERT assigns different vectors to the same word depending on its surrounding context — this distinction is frequently and specifically tested.
✓ Quick Self-Check
1. What does Bag of Words represent, and what does it ignore?
Simple word counts, ignoring word order and context entirely.
Tap to reveal / hide
2. What does TF-IDF weight more heavily than common words?
Rare, domain-specific terms.
Tap to reveal / hide
3. Does Word2Vec give the same word different vectors depending on context?
No — Word2Vec assigns exactly one fixed vector per word, regardless of context.
Tap to reveal / hide
4. Does BERT give the same word different vectors depending on context?
Yes — BERT produces contextual embeddings, so the same word gets different vectors depending on its surrounding context.
Tap to reveal / hide
5. Which representation approach does virtually all modern NLP now rely on?
Contextual embeddings, like those produced by BERT.
Tap to reveal / hide