Step by Step
D
Dense vectors — mapping discrete items to continuous space
Embeddings map discrete items (words, images, users) to dense numeric vectors, positioned so that similar items end up geometrically close together in that vector space.
Example: the word vectors for "cat" and "dog" end up close together in embedding space, reflecting their semantic similarity as animals.
S
Static vs. contextual embeddings
Word2Vec and GloVe produce static embeddings — the same word always gets the exact same vector regardless of context. BERT produces contextual embeddings — the same word gets a different vector depending on its surrounding context.
Example: the word "bank" gets the same Word2Vec vector whether it means a riverbank or a financial bank, but BERT would give it different vectors in each context.
C
Cosine similarity — measuring closeness
Cosine similarity measures the angle between two vectors, providing a standard way to quantify how similar two embeddings are, independent of their raw magnitude.
Example: computing the cosine similarity between the embeddings for two documents to determine how semantically similar their content is.
R
RAG — retrieval-augmented generation built on embeddings
Retrieval-Augmented Generation embeds documents into vectors, stores them in a vector database, and retrieves the most similar chunks (via cosine similarity or similar metrics) to provide as context to an LLM at query time.
Example: a customer support chatbot embedding its entire knowledge base, then retrieving the most relevant document chunks to include as context when answering a specific customer question.
Applied Walkthrough
1
The word "apple" needs to be represented in two very different contexts: as a fruit, and as the technology company.
2
Using a static embedding method like Word2Vec, "apple" gets the exact same vector in both contexts, since Word2Vec produces one fixed vector per word regardless of context.
3
Using BERT's contextual embeddings instead, "apple" gets two different vectors — one closer to other fruit words, and one closer to other technology company words — depending on the surrounding sentence.
4
This distinction between static and contextual embeddings is exactly why modern systems tend to prefer contextual embeddings for tasks requiring nuanced, context-sensitive understanding of meaning.
Exam Application
Exams test whether you understand the key distinction between static (Word2Vec/GloVe) and contextual (BERT) embeddings, and whether you know cosine similarity as the standard metric for comparing embedding vectors. Also expect a question connecting embeddings directly to RAG's retrieval mechanism.
⚠ Common Trap
The most common trap is confusing static embeddings with contextual embeddings, or assuming all embedding methods handle ambiguous words (like "bank" or "apple") the same way. Static embeddings assign exactly one fixed vector per word; contextual embeddings can assign different vectors to the same word depending on its surrounding context.
✓ Quick Self-Check
1. What do embeddings map discrete items into?
Dense numeric vectors, positioned so similar items are geometrically close together.
Tap to reveal / hide
2. What is the key difference between static and contextual embeddings?
Static embeddings (Word2Vec/GloVe) give the same word the same vector regardless of context; contextual embeddings (BERT) give the same word different vectors depending on context.
Tap to reveal / hide
3. What does cosine similarity measure?
The angle between two vectors, providing a standard way to quantify embedding similarity.
Tap to reveal / hide
4. What is a vector database used for?
Storing embeddings for efficient semantic search and retrieval.
Tap to reveal / hide
5. How are embeddings foundational to RAG (retrieval-augmented generation)?
Documents are embedded into vectors, and the most similar chunks are retrieved (via similarity metrics like cosine similarity) to provide as context to an LLM.
Tap to reveal / hide