Step by Step
B
Before ResNet — deeper networks degraded, not improved
Before ResNet's 2015 introduction, networks deeper than roughly 20 layers actually got WORSE as more layers were added, rather than better — a surprising degradation problem.
Example: a 30-layer network from before ResNet might have performed worse than a comparable 20-layer network, purely due to the difficulty of training such deep networks effectively.
S
Skip connections — output = F(x) + x
Residual connections add a direct shortcut: output = F(x) + x, where the network only needs to learn the residual (correction) F(x), rather than the full mapping — and even if F(x) learns nothing useful, the identity x still passes through unchanged.
Example: if a particular layer's transformation F(x) turns out not to help at all, the residual connection ensures the original input x still flows through unchanged, rather than being lost or degraded.
⚡
Why this works — gradients flow directly through skip connections
These skip connections let gradients flow directly backward through the network during training, keeping even the earliest layers actively learning, rather than suffering from vanishing gradients as in very deep pre-ResNet networks.
Example: ResNet-152, with 152 layers, trains successfully specifically because gradients can flow directly through its many skip connections, keeping early layers learning effectively.
Applied Walkthrough
1
Before 2015, a team tries training networks of increasing depth and finds that a 30-layer network actually performs worse than a comparable 20-layer network — deeper isn't helping.
2
ResNet introduces residual (skip) connections: output = F(x) + x, meaning each layer only needs to learn a correction on top of the input, rather than the full transformation from scratch.
3
This allows gradients to flow directly backward through the skip connections during training, keeping even very early layers actively learning, rather than suffering from vanishing gradients.
4
This breakthrough is exactly what enabled networks with 100+ layers (like ResNet-152) to train successfully, and residual connections remain a standard component in virtually all modern deep architectures, including the hundreds of Transformer blocks in modern LLMs.
Exam Application
Exams test whether you understand the specific pre-ResNet degradation problem (deeper networks getting worse, not better) and the residual connection formula (output = F(x) + x) that solved it by letting gradients flow directly through skip connections.
⚠ Common Trap
The most common trap is assuming the pre-ResNet degradation problem was simply due to overfitting. It was actually an optimization difficulty — very deep networks without residual connections were hard to train effectively at all, a fundamentally different problem from overfitting on already-well-trained networks.
✓ Quick Self-Check
1. What happened to networks deeper than roughly 20 layers before ResNet was introduced?
They degraded in performance — adding more layers made results worse, not better.
Tap to reveal / hide
2. What is the formula for a residual (skip) connection?
output = F(x) + x, where F(x) is the learned residual/correction and x is the original input passed through directly.
Tap to reveal / hide
3. What happens if F(x) learns nothing useful in a residual connection?
The identity x still passes through unchanged, so the layer doesn't actively hurt performance.
Tap to reveal / hide
4. Why do residual connections allow much deeper networks to train successfully?
They let gradients flow directly backward through the skip connections, keeping even early layers actively learning.
Tap to reveal / hide
5. Do modern LLMs, with their hundreds of Transformer blocks, use residual connections?
Yes — residual connections are a standard component throughout modern deep architectures, including Transformer blocks in LLMs.
Tap to reveal / hide