Step by Step
A
The trap — a naive model can score deceptively high
If 99% of emails are legitimate, a model that predicts "not spam" for absolutely every single email achieves 99% accuracy — while catching zero actual spam, a complete failure at the actual task.
Example: a fraud detection model that always predicts "not fraud" would score 99%+ accuracy on typical data, despite never catching a single actual fraud case.
P
Use Precision — when false positives are costly
Precision should be checked specifically when false alarms (like flagging a good email as spam) carry a high cost.
Example: checking precision for a spam filter, since blocking a legitimate important email is a costly false positive.
R
Use Recall — when false negatives are costly
Recall should be checked specifically when missed detections (like missing a cancer diagnosis) carry a high cost.
Example: checking recall for a cancer screening test, since missing an actual cancer case is a costly false negative.
F1
Use F1 — when both matter and classes are imbalanced
F1 should be checked when both false positives and false negatives matter, especially on imbalanced datasets where accuracy alone would be misleading.
Example: checking F1 for a fraud detection system on a heavily imbalanced dataset, since both missing fraud and flagging legitimate transactions carry real costs.
Applied Walkthrough
1
A fraud detection model reports 99.2% accuracy, and initial excitement suggests it must be performing extremely well.
2
Ask: does 99.2% accuracy alone confirm strong real-world performance here? Not necessarily — if only 0.8% of transactions are actually fraudulent, always predicting "not fraud" would already achieve that same accuracy while catching zero fraud.
3
Checking precision, recall, and F1 reveals the model's actual recall is only 15%, meaning it misses 85% of actual fraud cases — a serious problem completely hidden by the impressive-looking accuracy number.
4
This is exactly why, on any meaningfully imbalanced real-world dataset, accuracy should never be reported alone — precision, recall, and F1 should always be checked alongside it.
Exam Application
This is one of the most heavily tested evaluation pitfalls across any ML course. Exams frequently present a high accuracy figure alongside an imbalanced dataset and ask you to identify why that accuracy figure might be misleading, expecting you to recommend checking precision, recall, and F1 instead.
⚠ Common Trap
The most common trap is treating a high accuracy figure as automatically good news without first checking the class balance of the dataset. On balanced datasets, accuracy is a reasonably fine metric; on real-world imbalanced data (which is extremely common), a high accuracy figure can mask a model that's completely failing at its actual task.
✓ Quick Self-Check
1. How can a model achieve 99% accuracy while completely failing at its task?
By always predicting the majority class on a severely imbalanced dataset — it achieves high accuracy while catching zero minority-class cases.
Tap to reveal / hide
2. When should you check precision instead of relying on accuracy?
When false positives (false alarms) carry a high cost.
Tap to reveal / hide
3. When should you check recall instead of relying on accuracy?
When false negatives (missed detections) carry a high cost.
Tap to reveal / hide
4. When should you check F1 instead of relying on accuracy?
When both false positives and false negatives matter, especially on imbalanced datasets.
Tap to reveal / hide
5. Is accuracy a fine metric to use on balanced datasets?
Yes — the accuracy trap specifically applies to imbalanced data; on balanced datasets, accuracy is a reasonably fine metric.
Tap to reveal / hide