Step by Step
⚖
Regularization — a tax on model complexity
Regularization adds a penalty term to the loss function based on the size of the model's weights, discouraging the model from relying too heavily on any single feature and helping prevent overfitting.
Example: without regularization, a model might assign an extremely large weight to a noisy feature that happened to correlate with the training data by chance.
L1
L1 (Lasso) — kills unimportant features entirely
The penalty is the sum of the absolute values of the weights. This tends to drive unimportant weights all the way to exactly zero, effectively performing automatic feature selection.
Example: out of 100 candidate features, L1 regularization might drive 80 of their weights to exactly zero, leaving only the 20 most important features actively contributing to predictions.
L2
L2 (Ridge) — shrinks all features, rarely to zero
The penalty is the sum of the squared weights. This shrinks all weights toward zero somewhat but rarely forces any weight to become exactly zero, keeping all features contributing at least a little.
Example: L2 regularization might shrink all 100 feature weights toward smaller values, but nearly all of them remain nonzero, still contributing some small amount to predictions.
Applied Walkthrough
1
A dataset has 200 candidate features, but the data scientist strongly suspects only a handful actually matter for the prediction.
2
Ask: which regularization type is better suited here — L1 or L2? L1 (Lasso), since it drives unimportant weights to exactly zero, automatically performing feature selection.
3
Contrast: if instead the data scientist believed most of the 200 features contribute at least a small amount of genuine signal, L2 (Ridge) would be the better choice, since it shrinks weights without eliminating any entirely.
4
Elastic Net, which combines both L1 and L2 penalties, offers a middle-ground option when you're unsure which pure approach fits best.
Exam Application
Exams test whether you can match the correct regularization type to a described scenario: suspecting only a few features matter (favors L1/Lasso) vs. believing most features contribute something (favors L2/Ridge). Also expect questions on the lambda parameter, which controls how strong the regularization penalty is.
⚠ Common Trap
The most common trap is forgetting which regularization type can produce exactly-zero weights. Only L1 (Lasso) drives weights all the way to zero, enabling automatic feature selection; L2 (Ridge) only shrinks weights toward zero without eliminating them entirely. Mixing up which one "kills" vs. "shrinks" features is a frequent, heavily tested mistake.
✓ Quick Self-Check
1. What does regularization add to the loss function?
A penalty based on the size of the model's weights, discouraging overreliance on any single feature and helping prevent overfitting.
Tap to reveal / hide
2. What penalty does L1 (Lasso) regularization use?
The sum of the absolute values of the weights.
Tap to reveal / hide
3. What penalty does L2 (Ridge) regularization use?
The sum of the squared weights.
Tap to reveal / hide
4. Which regularization type can drive weights all the way to exactly zero, and why does this matter?
L1 (Lasso) — this effectively performs automatic feature selection by eliminating unimportant features entirely.
Tap to reveal / hide
5. When would you prefer L2 (Ridge) over L1 (Lasso)?
When you believe most or all features contribute at least some genuine signal, rather than suspecting only a few features matter.
Tap to reveal / hide