Step by Step
V
Vanishing gradients — early layers barely learn
In very deep networks, gradients can shrink exponentially as they flow backward through many layers, especially when using sigmoid or tanh activations, meaning the earliest layers receive almost no meaningful gradient signal and barely update at all.
Example: in a 50-layer network using sigmoid throughout, the first few layers might receive a gradient so tiny it's effectively zero, meaning those layers never really learn anything useful.
E
Exploding gradients — weights blow up uncontrollably
The opposite problem: gradients can grow exponentially large as they flow backward, causing weight updates to become unstable and the model's weights to blow up to extreme values.
Example: a network's weights suddenly becoming NaN (not a number) partway through training is a classic symptom of exploding gradients.
⭐
ResNet skip connections — the fix that enabled 100+ layer networks
Skip (residual) connections create a "gradient highway" that allows gradients to bypass layers directly, dramatically reducing the vanishing gradient problem and enabling networks with 100+ layers to train successfully.
Example: ResNet-152, with 152 layers, was made trainable specifically because of these skip connections allowing gradients to flow directly across many layers rather than vanishing.
Applied Walkthrough
1
A team trains a very deep network using sigmoid activations throughout, and notices the earliest layers' weights barely change at all during training, while later layers train normally.
2
Ask: what problem does this describe? Vanishing gradients — the gradient signal has shrunk so much by the time it reaches early layers that those layers barely update.
3
One fix would be switching to ReLU activations and adding batch normalization, but for very deep networks, adding ResNet-style skip connections is the more powerful, targeted solution.
4
Skip connections create a direct path for gradients to flow backward without being diminished by every single layer in between, which is precisely what enabled networks with 100+ layers to train successfully.
Exam Application
Exams test whether you can distinguish vanishing gradients (early layers don't learn, caused by sigmoid/tanh and many layers, fixed by ReLU/batch norm/skip connections) from exploding gradients (weights blow up, fixed by gradient clipping and careful initialization). Also expect a specific fact check: ResNet's skip connections are the landmark solution that enabled 100+ layer networks.
⚠ Common Trap
The most common trap is applying the wrong fix to the wrong problem — using gradient clipping (designed for exploding gradients) to address a vanishing gradient problem, or vice versa. These are opposite problems with different causes and different specific fixes.
✓ Quick Self-Check
1. What causes the vanishing gradient problem?
Gradients shrinking exponentially as they flow backward through many layers, especially with sigmoid/tanh activations, so early layers barely update.
Tap to reveal / hide
2. What causes the exploding gradient problem?
Gradients growing exponentially large as they flow backward, causing weights to blow up to unstable values.
Tap to reveal / hide
3. What technique specifically fixes exploding gradients?
Gradient clipping — capping the magnitude of gradients.
Tap to reveal / hide
4. What did ResNet's skip connections solve, and what did this enable?
They solved the vanishing gradient problem, enabling networks with 100+ layers to train successfully.
Tap to reveal / hide
5. Name two fixes for the vanishing gradient problem besides skip connections.
Using ReLU activations and adding batch normalization.
Tap to reveal / hide