Backpropagation and gradient learning
Errors at a network's output can be passed back to adjust every weight, so deep networks can learn.
Mathematics and information · Machine learning · No. 130
- Digital age
- Science
The mathematics behind backpropagation appeared in a 1970 master's thesis on rounding errors, and a 1974 PhD thesis laid its foundation.
Chart: Illustration: Backpropagation and gradient learning
Where it comes from
Chart: Map of where it comes from
Seppo Linnainmaa described the reverse mode of differentiation in a 1970 master's thesis. Paul Werbos's 1974 Harvard doctoral thesis laid the foundation of backpropagation. Rumelhart, Hinton and Williams made it known in 1986.
Three things to know
- The 1986 Nature abstract says backpropagation can create useful new features, which earlier methods such as the perceptron-convergence procedure could not.
- Hidden units, which are neither input nor output, come to represent important features of the task.
- In 1989 a backpropagation network read handwritten digits on US Postal Service zip codes with a 1% error rate and about a 9% reject rate.
The journey
- 1 ErrorA network's output is compared with the right answer
- 2 BlameThe error is passed backwards, layer by layer
- 3 UpdateEvery weight is nudged to shrink the error
- 4 PaperRumelhart, Hinton and Williams in Nature, 1986
- 5 UseHandwritten digits on zip codes, 1989, and then deep networks