The Neural Network Training Loop Memory Trick

Step by Step

Forward Pass — GUESS

Input data flows through the network's layers, each doing a calculation, producing a prediction at the end. This is the network's current "guess" based on its current weights.

Example: an image goes in, and the network outputs "73% confident this is a cat."

Loss Calculation — GRADE

The prediction is compared to the actual correct answer using a loss function, producing a single number that measures how wrong the guess was.

Example: if the true answer was "dog" but the network guessed "73% cat," the loss function outputs a high error score.

Backpropagation — BLAME

The error is sent backward through the network, calculating exactly how much each individual weight contributed to the mistake — assigning "blame" mathematically using calculus (the chain rule).

Example: backprop determines that a specific weight in layer 3 contributed heavily to the wrong guess, while a weight in layer 1 barely mattered.

Weight Update — FIX

An optimizer (like gradient descent) nudges each weight slightly in the direction that would have reduced the error, using a learning rate to control the step size.

Example: the misbehaving weight gets adjusted down slightly; this whole 4-step cycle then repeats with the next batch of data, thousands or millions of times.

Applied Walkthrough

A network is shown a training image of a dog and predicts "85% cat" — a bad guess.

The loss function compares "85% cat" to the true label "dog" and produces a high error value.

Backpropagation traces that error backward through every layer, calculating each weight's share of the blame.

The optimizer updates the weights slightly to reduce that error next time, and the entire GUESS → GRADE → BLAME → FIX cycle repeats with the next training image — over and over, often millions of times, until the loss gets small.

Exam Application

Exam questions often ask you to put the steps of training in the correct order, or to identify which step a description matches. Forward pass always comes first (producing a prediction), loss calculation is always second (comparing to truth), backpropagation is always third (assigning blame backward through the network), and the weight update is always last (actually changing the weights). Memorize this order — it never changes.

⚠ Common Trap

The most common trap is confusing backpropagation with the weight update itself. Backpropagation only CALCULATES how much each weight contributed to the error (the gradients) — it does not change any weights by itself. The optimizer (gradient descent or a variant) is the separate step that actually uses those calculated gradients to update the weights. Many students merge these into one step and lose points for imprecision.

✓ Quick Self-Check

1. What happens during the forward pass?

Input data flows through the network's layers and produces a prediction — the network's current "guess."

Tap to reveal / hide

2. What does the loss function measure?

How wrong the network's prediction was compared to the true, correct answer — expressed as a single error number.

Tap to reveal / hide

3. Does backpropagation change the network's weights directly?

No. Backpropagation only calculates how much each weight contributed to the error (the gradients); the optimizer uses those gradients to actually update the weights.

Tap to reveal / hide

4. What is the correct order of the four training loop steps?

Forward pass (guess), loss calculation (grade), backpropagation (blame), weight update (fix) — always in that order.

Tap to reveal / hide

5. Roughly how many times does this training loop cycle repeat during training?

Thousands to millions of times, depending on the dataset size and number of training epochs.

Tap to reveal / hide