Step by Step
C
The rain forecast test — does 70% actually mean 70%?
If a weather model says "70% probability of rain" many times, a well-calibrated model should actually see rain occur roughly 70% of those times — not more, not dramatically less.
Example: checking whether it actually rained on roughly 70 out of 100 days when the model predicted "70% chance of rain."
A
Reliability diagrams — visualizing the match (or mismatch)
A reliability diagram (calibration curve) plots predicted probability against actual observed frequency, with a perfect diagonal line representing perfect calibration.
Example: a reliability diagram revealing that the model's "90% confident" predictions are only actually correct 60% of the time, exposing serious miscalibration.
L
Why it's dangerous when poorly calibrated
A model that says "90% confident" but is only actually right 60% of the time is dangerously overconfident — a serious problem specifically for high-stakes applications like medical diagnosis or financial risk assessment, where decisions get made based on trusting that stated confidence level.
Example: a doctor trusting a model's stated 90% confidence in a diagnosis, when the model's true accuracy at that confidence level is actually only 60%, could lead to seriously flawed medical decisions.
⚡
Fixing calibration — Platt scaling and isotonic regression
Platt scaling (fitting a logistic regression on top of the model's raw outputs) and isotonic regression are two standard techniques used to correct poor calibration, bringing predicted probabilities back into better alignment with actual observed frequencies.
Example: applying Platt scaling to an overconfident model's outputs, recalibrating its probability estimates to better match its true accuracy.
Applied Walkthrough
1
A financial risk model states it's "90% confident" in a particular high-stakes lending decision.
2
A reliability diagram reveals that among all the times the model states 90% confidence, it's actually only correct about 60% of the time.
3
This is a serious calibration problem, since real financial decisions are being made based on trusting that stated 90% confidence level, when the model's true reliability at that stated level is significantly lower.
4
Applying Platt scaling or isotonic regression to recalibrate the model's probability outputs would bring its stated confidence levels back into better alignment with its actual real-world accuracy, making its confidence estimates trustworthy again.
Exam Application
Exams test whether you understand calibration as a distinct property from accuracy (a model can be accurate overall while still poorly calibrated on individual confidence estimates), and whether you know the two standard fixes: Platt scaling and isotonic regression.
⚠ Common Trap
The most common trap is assuming a model's stated confidence level can always be trusted at face value, especially if the model's overall accuracy seems reasonably good. Calibration specifically measures whether stated confidence matches actual correctness rate — a genuinely separate question from overall accuracy that must be checked independently, especially for high-stakes decisions.
✓ Quick Self-Check
1. What does the rain forecast test check for?
Whether a model's stated probability (like "70% chance of rain") actually matches the real-world frequency of that outcome occurring.
Tap to reveal / hide
2. What does a reliability diagram plot?
Predicted probability against actual observed frequency, with a perfect diagonal line representing perfect calibration.
Tap to reveal / hide
3. Why is poor calibration especially dangerous for medical diagnosis or financial risk?
Because real decisions get made based on trusting the model's stated confidence level, and if that confidence doesn't match true accuracy, those decisions can be seriously flawed.
Tap to reveal / hide
4. Name two standard techniques used to fix poor calibration.
Platt scaling (logistic regression on top of the model's outputs) and isotonic regression.
Tap to reveal / hide
5. Can a model be highly accurate overall while still being poorly calibrated?
Yes — accuracy and calibration are separate properties, and a model can excel at one while failing at the other.
Tap to reveal / hide