Step by Step
S
Sigmoid (Silly) — output layer only, for binary classification
Squashes any input to a range between 0 and 1, making it useful for binary classification output layers, but it suffers from vanishing gradients when used in hidden layers of deep networks.
Example: outputting 0.87, interpreted as an 87% probability the input belongs to the positive class.
T
Tanh (Tom) — zero-centered, but still vanishes in deep nets
Squashes input to a range between -1 and 1, centered at zero, which can help with certain training dynamics, but like sigmoid, it still suffers from vanishing gradients in very deep networks.
Example: an input far from zero gets squashed close to either -1 or 1, and the gradient in that flattened region becomes very small.
R
ReLU (Really) — the default choice for hidden layers
Computes max(0, x) — passing positive values through unchanged and zeroing out negative values. It's fast to compute and avoids the vanishing gradient problem for positive inputs, making it the standard default for hidden layers.
Example: an input of 4.4 passes through unchanged as 4.4, while an input of -2.1 becomes 0.
L
Leaky ReLU (Likes) — fixes the dying ReLU problem
Similar to ReLU, but instead of zeroing out negative inputs entirely, it allows a small negative slope, preventing neurons from getting permanently "stuck" at zero output (the dying ReLU problem).
Example: an input of -2.1 might become -0.021 instead of exactly 0, keeping a small gradient signal alive for that neuron.
Applied Walkthrough
1
A deep network's hidden layers use sigmoid activation throughout, and training progress stalls almost completely after a few layers.
2
Ask: what's the likely cause? Sigmoid activations in deep hidden layers cause vanishing gradients, since their gradient becomes tiny for inputs far from zero.
3
The fix: replace sigmoid with ReLU in the hidden layers, which avoids this vanishing gradient issue for positive inputs and trains much faster.
4
Sigmoid should be reserved specifically for the binary classification output layer, where its 0-1 probability output is actually needed — not scattered throughout the hidden layers.
Exam Application
Exams test whether you know which activation function belongs where: sigmoid at binary output layers only, softmax at multi-class output layers, and ReLU (or leaky ReLU) as the default for hidden layers. Using sigmoid or tanh throughout deep hidden layers is a classic tested mistake due to vanishing gradients.
⚠ Common Trap
The most common trap is using sigmoid or tanh throughout all hidden layers of a deep network, which causes vanishing gradients and stalls training. ReLU (or leaky ReLU) is the correct default for hidden layers precisely because it avoids this problem for positive inputs.
✓ Quick Self-Check
1. What range does sigmoid squash its output to?
Between 0 and 1.
Tap to reveal / hide
2. Where should sigmoid typically be used in a network?
Only at the output layer for binary classification, not throughout hidden layers.
Tap to reveal / hide
3. What formula defines ReLU?
max(0, x) — passing positive values through unchanged and zeroing out negative values.
Tap to reveal / hide
4. What problem does Leaky ReLU fix compared to standard ReLU?
The dying ReLU problem, where neurons can get permanently stuck outputting zero; Leaky ReLU allows a small negative slope instead.
Tap to reveal / hide
5. Why do sigmoid and tanh cause problems in deep hidden layers?
They suffer from vanishing gradients — their gradient becomes very small for inputs far from zero, stalling training in deep networks.
Tap to reveal / hide