Step by Step
H
Historical bias — training data reflects past discrimination
Training data itself can reflect decades of past discriminatory practices, meaning a model trained on that data can simply learn to replicate historical unfairness.
Example: a lending model trained on decades of historical loan approval data that reflected discriminatory past lending practices.
S
Sampling bias — training data doesn't represent the deployment population
If the training data doesn't accurately represent the population the model will actually be used on, the model's performance and fairness can degrade badly for underrepresented groups.
Example: a facial recognition system trained primarily on lighter-skinned faces performing much worse on darker-skinned faces once deployed broadly.
M
Measurement bias — proxy variables carry demographic information
Seemingly neutral features can act as proxies that indirectly carry demographic information, even when protected characteristics themselves are explicitly excluded from the model.
Example: ZIP code correlating strongly with race, meaning a model using ZIP code as a feature can indirectly discriminate by race even without race ever being an explicit input.
F
Feedback loops — biased predictions create biased future data
A biased model's predictions can shape future real-world outcomes, which then become the training data for the next version of the model, compounding and reinforcing the original bias over time.
Example: a predictive policing model sending more patrols to a neighborhood, leading to more recorded arrests there, which then gets used as "evidence" to justify sending even more patrols in the future.
Applied Walkthrough
1
The COMPAS criminal risk-assessment tool was found to falsely flag Black defendants as high-risk at roughly twice the rate of white defendants.
2
Separately, examining accuracy alone, COMPAS also achieved similar overall predictive accuracy across racial groups.
3
Both of these seemingly conflicting bias claims were mathematically true simultaneously, illustrating that different fairness definitions (equal false positive rates vs. equal overall accuracy) can genuinely conflict with each other.
4
This case is frequently cited specifically because it demonstrates that "fairness" isn't a single, unambiguous mathematical property — different fairness metrics can produce contradictory conclusions about the very same model.
Exam Application
Exams test whether you can identify which specific source of bias (historical, sampling, measurement, or feedback loop) a described scenario represents, and whether you understand the COMPAS case as a landmark illustration of competing fairness definitions.
⚠ Common Trap
The most common trap is assuming excluding protected characteristics (like race) directly from a model's inputs is sufficient to eliminate bias. Measurement bias specifically shows that proxy variables (like ZIP code) can still carry substantial demographic information even when the protected characteristic itself is never an explicit input.
✓ Quick Self-Check
1. What is historical bias?
Training data reflecting decades of past discriminatory practices, which a model can learn to replicate.
Tap to reveal / hide
2. What is measurement bias, and give an example?
Seemingly neutral proxy variables carrying demographic information — like ZIP code correlating with race.
Tap to reveal / hide
3. What is a feedback loop in the context of algorithmic bias?
A biased model's predictions shaping future real-world data, which then reinforces and compounds the original bias in future model versions.
Tap to reveal / hide
4. What did the COMPAS case reveal about competing fairness definitions?
That different fairness metrics (equal false positive rates vs. equal overall accuracy) can be simultaneously true yet contradictory, showing fairness isn't a single unambiguous property.
Tap to reveal / hide
5. Does excluding race as an explicit input guarantee a model is unbiased by race?
No — proxy variables like ZIP code can still indirectly carry demographic information, a phenomenon called measurement bias.
Tap to reveal / hide