I
Input layer — one neuron per feature, no computation
Simply holds the raw feature values fed into the network; it performs no computation of its own, just passes values into the first hidden layer.
Example: an image classifier's input layer might have one neuron per pixel value in the image.
H
Hidden layers — where representations are actually learned
The layers between input and output where the network progressively learns increasingly abstract representations of the data.
Example: in an image classifier, early hidden layers might learn to detect edges, middle layers combine edges into shapes, and later layers combine shapes into recognizable objects.
O
Output layer — one neuron per class, or one for regression
Produces the network's final prediction — one neuron per class for classification tasks (often paired with softmax), or a single neuron for a regression task predicting a continuous value.
Example: a 10-class digit classifier has 10 output neurons, one for each possible digit 0-9.
Applied Walkthrough
1
An image classifier's hidden layers are examined to understand what each layer has learned.
2
The earliest hidden layers are found to detect simple edges and color gradients.
3
Middle hidden layers combine those edges into basic shapes and textures.
4
The deepest hidden layers combine those shapes into recognizable objects and even abstract concepts — illustrating exactly why depth enables this edges-to-shapes-to-objects-to-concepts hierarchy that a shallow network could not learn as effectively.
Exam Application
Exams test whether you understand the distinct role of each layer type (input performs no computation, hidden layers learn representations, output produces the final prediction) and the Universal Approximation Theorem's precise claim — that even a single hidden layer with enough neurons can theoretically approximate any continuous function, even though deeper networks are typically far more practical and efficient for learning hierarchical representations.
⚠ Common Trap
The most common trap is confusing "depth" (number of hidden layers) with "width" (number of neurons per layer) — these are two independent architectural choices that affect a network's capacity differently. A second trap is assuming the Universal Approximation Theorem implies a single hidden layer is always sufficient in practice — theoretically possible, but often requiring an impractically large number of neurons compared to a deeper architecture.
✓ Quick Self-Check
1. How much computation does the input layer perform?
None — it simply holds and passes through the raw feature values.
Tap to reveal / hide
2. What happens in the hidden layers of a network?
They progressively learn representations of the data, often in a hierarchical fashion from simple to complex.
Tap to reveal / hide
3. How many output neurons would a 10-class classification task typically have?
10 — one neuron per class.
Tap to reveal / hide
4. What does the Universal Approximation Theorem claim?
That even a single hidden layer with enough neurons can theoretically approximate any continuous function.
Tap to reveal / hide
5. What is the difference between a network's "depth" and its "width"?
Depth refers to the number of hidden layers; width refers to the number of neurons within each layer — two independent architectural choices.
Tap to reveal / hide