Material Insights · Understanding the Material World

Backpropagation and gradient learning

Errors at a network's output can be passed back to adjust every weight, so deep networks can learn.

Mathematics and information · Machine learning · No. 130

  • Digital age
  • Science
Mathematics and information · The science behind technology

The mathematics behind backpropagation appeared in a 1970 master's thesis on rounding errors, and a 1974 PhD thesis laid its foundation.

Chart: Illustration: Backpropagation and gradient learning

Where it comes from

Chart: Map of where it comes from

Seppo Linnainmaa described the reverse mode of differentiation in a 1970 master's thesis. Paul Werbos's 1974 Harvard doctoral thesis laid the foundation of backpropagation. Rumelhart, Hinton and Williams made it known in 1986.

Three things to know
  1. The 1986 Nature abstract says backpropagation can create useful new features, which earlier methods such as the perceptron-convergence procedure could not.
  2. Hidden units, which are neither input nor output, come to represent important features of the task.
  3. In 1989 a backpropagation network read handwritten digits on US Postal Service zip codes with a 1% error rate and about a 9% reject rate.
The journey
  1. 1 ErrorA network's output is compared with the right answer
  2. 2 BlameThe error is passed backwards, layer by layer
  3. 3 UpdateEvery weight is nudged to shrink the error
  4. 4 PaperRumelhart, Hinton and Williams in Nature, 1986
  5. 5 UseHandwritten digits on zip codes, 1989, and then deep networks
Go deeper pick a tab to open it in full