# Backpropagation and gradient learning

Errors at a network's output can be passed back to adjust every weight, so deep networks can learn.

Source page: https://www.materialinsights.org/science/backpropagation/

Mathematics and information · Machine learning · No. 130

- Digital age
- Science
 Mathematics and information · The science behind technology

The mathematics behind backpropagation appeared in a 1970 master's thesis on rounding errors, and a 1974 PhD thesis laid its foundation.

Chart: Illustration: Backpropagation and gradient learning

##### Where it comes from

Chart: Map of where it comes from

Seppo Linnainmaa described the reverse mode of differentiation in a 1970 master's thesis. Paul Werbos's 1974 Harvard doctoral thesis laid the foundation of backpropagation. Rumelhart, Hinton and Williams made it known in 1986.

##### Three things to know

1. The 1986 Nature abstract says backpropagation can create useful new features, which earlier methods such as the perceptron-convergence procedure could not.
2. Hidden units, which are neither input nor output, come to represent important features of the task.
3. In 1989 a backpropagation network read handwritten digits on US Postal Service zip codes with a 1% error rate and about a 9% reject rate.

##### The journey

1. 1 **Error**A network's output is compared with the right answer
2. 2 **Blame**The error is passed backwards, layer by layer
3. 3 **Update**Every weight is nudged to shrink the error
4. 4 **Paper**Rumelhart, Hinton and Williams in Nature, 1986
5. 5 **Use**Handwritten digits on zip codes, 1989, and then deep networks

##### Go deeper pick a tab to open it in full

## Other sections

- [The idea](https://www.materialinsights.org/science/backpropagation/sci-idea/index.md)
- [Discovery](https://www.materialinsights.org/science/backpropagation/sci-discovery/index.md)
- [Technologies](https://www.materialinsights.org/science/backpropagation/sci-technologies/index.md)
