Step by Step
1
Blindfolded on a hill — you can't see the whole landscape
Imagine standing blindfolded on a hilly landscape representing the loss function. You can't see the whole terrain — you can only feel the slope right under your feet.
Example: the "landscape" represents all possible combinations of the model's weights, with height representing how wrong (high loss) or right (low loss) that combination is.
2
Feel the slope, step downhill — following the gradient
You feel which direction slopes downward beneath your feet and take a small step in that direction — this is exactly what the gradient (calculated via calculus) tells the model to do with its weights.
Example: the gradient calculation reveals that increasing one particular weight slightly would reduce the loss, so the model nudges that weight in that direction.
3
Repeat thousands of times — gradually reaching the valley
You repeat the feel-and-step process over and over, gradually descending toward the lowest point (minimum loss) you can find, exactly as the model repeats its weight updates thousands of times during training.
Example: after thousands of small downhill steps, the model's weights settle near a point where the loss can't easily be reduced further.
4
Learning rate — how big each step is
The learning rate controls the size of each step downhill. Too big a learning rate can cause overshooting past the lowest point; too small a learning rate makes progress agonizingly slow.
Example: an overly large learning rate might cause the model to leap right over the true minimum and end up on the opposite slope, bouncing back and forth instead of settling down.
Applied Walkthrough
1
A model's loss value oscillates wildly during training instead of steadily decreasing, sometimes even increasing.
2
Ask: using the hill metaphor, what does this behavior suggest? The steps downhill are too large — overshooting past the lowest point repeatedly.
3
This points to the learning rate being set too high, causing the model to leap past the minimum rather than settling smoothly into it.
4
The fix: reduce the learning rate, so each step downhill is smaller and more controlled, allowing the model to actually settle near the true minimum instead of bouncing past it.
Exam Application
Exams test whether you understand the tradeoff built into the learning rate: too large causes overshooting and unstable training; too small causes painfully slow convergence. Questions often describe a training symptom (oscillating loss, or extremely slow progress) and ask you to diagnose whether the learning rate is too high or too low.
⚠ Common Trap
The most common trap is assuming a smaller learning rate is always safer or better. While it avoids overshooting, an excessively small learning rate can make training so slow it effectively never converges in a reasonable amount of time — there is a real tradeoff, not a one-directional "smaller is always better" rule.
✓ Quick Self-Check
1. In the hill metaphor, what does the "lowest point" of the landscape represent?
The minimum loss — the best-fitting set of model weights.
Tap to reveal / hide
2. What does the gradient tell the model, in terms of the hill metaphor?
Which direction (which way to adjust the weights) will most reduce the loss — like feeling which way is downhill.
Tap to reveal / hide
3. What does the learning rate control?
The size of each step the model takes when updating its weights.
Tap to reveal / hide
4. What happens if the learning rate is set too high?
The model can overshoot past the minimum loss point, causing unstable or oscillating training.
Tap to reveal / hide
5. What happens if the learning rate is set too low?
Training becomes very slow, taking a very long time to reach the minimum loss.
Tap to reveal / hide