Step by Step
B
BLEU — n-gram overlap for translation
BLEU measures n-gram overlap between a generated translation and a reference translation, ranging from 0 to 1, correlating reasonably well with human judgment but missing semantic equivalents that use different words.
Example: BLEU would score a translation using "automobile" lower against a reference using "car," even though they mean exactly the same thing, since BLEU only measures literal word overlap.
R
ROUGE — n-gram overlap for summarization
ROUGE similarly measures n-gram overlap, but specifically for summarization tasks, with variants including ROUGE-1 (unigram overlap), ROUGE-2 (bigram overlap), and ROUGE-L (longest common subsequence).
Example: ROUGE-L measuring how much of the longest common subsequence a generated summary shares with a reference summary.
P
Perplexity — how well a language model predicts text
Perplexity measures how well a language model predicts a test set of text — lower perplexity indicates the model finds the text less "surprising" and thus predicts it more accurately.
Example: a language model with lower perplexity on a test set is doing a better job predicting what comes next in that text than a model with higher perplexity.
B²
BERTScore — semantic similarity beyond n-grams
BERTScore uses embeddings to measure semantic similarity between generated and reference text, capturing meaning-based similarity that traditional n-gram metrics like BLEU and ROUGE can miss.
Example: BERTScore correctly recognizing that "automobile" and "car" are semantically similar, even though they share no n-gram overlap at all.
Applied Walkthrough
1
A translation model produces a technically excellent, fluent translation that uses "automobile" where the reference translation used "car."
2
Ask: will BLEU correctly recognize these as equivalent? No — BLEU only measures literal n-gram overlap, so it would penalize this translation despite its semantic correctness.
3
Using BERTScore instead, which measures semantic similarity via embeddings rather than exact word overlap, would correctly recognize "automobile" and "car" as very similar in meaning.
4
This illustrates why BLEU (and similarly ROUGE) can systematically underrate genuinely good outputs that simply use different but semantically equivalent wording, and why embedding-based metrics like BERTScore are considered an improvement.
Exam Application
Exams test whether you can match the correct metric to the correct task type (BLEU for translation, ROUGE for summarization, perplexity for language model quality, BERTScore for semantic similarity) and whether you understand BLEU/ROUGE's shared key limitation — missing semantically equivalent but lexically different text.
⚠ Common Trap
The most common trap is assuming a low BLEU or ROUGE score always means poor output quality. Both metrics can penalize genuinely excellent outputs simply because they used different words than the reference, even when the meaning is identical or even better — this is precisely why BERTScore was developed as a semantic alternative.
✓ Quick Self-Check
1. What does BLEU measure, and for what task?
N-gram overlap between generated and reference text, used primarily for evaluating translation quality.
Tap to reveal / hide
2. What does ROUGE measure, and for what task?
N-gram overlap (with variants like ROUGE-1, ROUGE-2, ROUGE-L), used primarily for evaluating summarization quality.
Tap to reveal / hide
3. What does perplexity measure?
How well a language model predicts a test set of text — lower is better.
Tap to reveal / hide
4. What key limitation do BLEU and ROUGE share?
They miss semantic equivalents that use different words, since they only measure literal n-gram overlap.
Tap to reveal / hide
5. What does BERTScore do differently from BLEU and ROUGE?
Uses embeddings to measure semantic similarity, rather than relying purely on literal n-gram overlap.
Tap to reveal / hide