Backpropagation and gradient learning: The idea
In plain words
The idea explained without jargon
A network makes a guess, and the guess is compared with the right answer to give an error. Backpropagation works out, for every connection weight, how much the error would change if that weight changed a little. It does this by passing the error backwards from the output through each layer, using the chain rule of calculus, so one backward sweep covers every weight at once. Each weight is then nudged in the direction that shrinks the error, and the process repeats over many examples. The weights of the connections are adjusted again and again to minimise the difference between actual and desired output, and the hidden units in the middle come to stand for useful features of the task [1].
as of 2026-11-12
Key equation
The equation that sums it up, and what each symbol means
- Equation
- w_new = w_old - eta * dE/dw
- Symbols
- {"w":"one connection weight in the network","E":"the error: how far the network's output is from the desired output","dE/dw":"how much the error changes per small change in w, found by passing the error backwards with the chain rule","eta":"the learning rate: how big a step to take"}
- Note
- This is gradient descent. Backpropagation is the method for finding dE/dw for all weights at once [1].
as of 2026-11-12
An analogy
A picture from everyday life that makes the idea easier to hold
Imagine a kitchen line where a dish comes out too salty. The head chef tells the last cook how far off it was, the last cook works out how much each earlier station added to the problem, and every station adjusts a little. The picture stops being true because the stations are numbers, the blame is worked out exactly by calculus rather than by judgement, and the whole line can settle on a mediocre recipe if the steps are badly chosen.
as of 2026-11-12
Sources
- Learning representations by back-propagating errors Nature (D. E. Rumelhart, G. E. Hinton and R. J. Williams), via the SAO/NASA Astrophysics Data System, 1986, vol. 323, October 1986