Step by Step
A
An address for every word — mapping to high-dimensional space
Word embeddings map each word to a specific point in high-dimensional space (often represented by hundreds of numbers), essentially giving every word its own precise "address."
Example: the word "king" might be represented by a vector of 300 numbers, its unique coordinate address in this high-dimensional space.
N
Neighborhoods — similar words live close together
Just like similar houses cluster together in the same neighborhood of a city, words with similar meanings end up positioned close together in this high-dimensional embedding space.
Example: "king" and "queen" end up positioned relatively close together in embedding space, reflecting their related meanings, while a completely unrelated word like "bicycle" would be positioned far away.
C
Coordinate arithmetic — capturing relationships mathematically
Remarkably, vector arithmetic on these coordinates captures meaningful relationships: king minus man plus woman approximately equals queen, and Paris minus France plus Italy approximately equals Rome.
Example: performing this vector arithmetic (king - man + woman) and finding that the resulting point in space lands very close to the actual "queen" vector.
⚠
No true understanding — just learned co-occurrence patterns
The model doesn't actually "understand" meaning in any human sense — it simply learns that words appearing in similar contexts throughout training end up with similar coordinates, purely as a statistical byproduct of the training process.
Example: "king" and "queen" end up close together not because the model understands royalty, but because these words tend to appear in similar surrounding contexts throughout the training text.
Applied Walkthrough
1
A student is confused about how a model could possibly know that "king minus man plus woman" should approximately equal "queen."
2
Ask: does the model actually understand concepts like royalty or gender? No — it never "understands" meaning in any human sense.
3
Instead, the model simply learned that words appearing in similar surrounding contexts throughout its training data end up with similar coordinate positions in the embedding space.
4
Because "king" and "queen" (and "man" and "woman") tend to appear in analogous contextual patterns, this purely statistical co-occurrence learning happens to produce vector arithmetic that mirrors meaningful relationships — without any genuine conceptual understanding involved.
Exam Application
Exams test whether you understand the geometric intuition behind word embeddings (similar words cluster together, vector arithmetic captures relationships) and specifically whether you know this all emerges from learned co-occurrence patterns, not genuine semantic understanding.
⚠ Common Trap
The most common trap is anthropomorphizing word embeddings, assuming the model genuinely "understands" concepts like royalty or gender because vector arithmetic happens to work. The model has simply learned statistical co-occurrence patterns from training data — the meaningful-looking arithmetic is an emergent byproduct, not evidence of genuine conceptual understanding.
✓ Quick Self-Check
1. What do word embeddings map each word to?
A specific point in high-dimensional space, essentially giving every word a coordinate "address."
Tap to reveal / hide
2. What happens to words with similar meanings in embedding space?
They end up positioned close together, like similar houses clustering in the same neighborhood.
Tap to reveal / hide
3. What famous vector arithmetic example illustrates embeddings capturing relationships?
King minus man plus woman approximately equals queen.
Tap to reveal / hide
4. Does the model genuinely "understand" meaning when producing these embeddings?
No — it simply learns that words appearing in similar contexts end up with similar coordinates, a purely statistical pattern.
Tap to reveal / hide
5. Name three approaches that all work on this word-embedding principle.
Word2Vec, GloVe, and modern contextual embeddings like BERT.
Tap to reveal / hide