Backpropagation and gradient learning: Discovery
The story
How the idea was found and came to be accepted
Earlier methods such as the perceptron-convergence procedure could not create useful new features inside a network [1]. Backpropagation repeatedly adjusts the connection weights to shrink the gap between the actual and the desired output, and the hidden units in between come to represent important features of the task [1]. The mathematical core was already known: in 1970 Seppo Linnainmaa, in a master's thesis in Helsinki, described what is now called the reverse mode of differentiation, as a tool for estimating rounding errors [2, 3]. Paul Werbos's 1974 Harvard doctoral thesis laid the foundation of backpropagation [4]. In 1986 Rumelhart, Hinton and Williams showed in Nature that backpropagation could train networks with a hidden layer, so that the hidden units learn useful features [1, 2]. In 1989 LeCun and colleagues used it to read handwritten digits from US Postal Service zip codes [5]. Because several people reached the idea separately, credit is shared rather than given to one person.
as of 2026-11-12
People
The people behind it, their years and what each contributed
| Name | Years | Contribution |
|---|---|---|
| David Rumelhart | 1942 to 2011 | Co-author of the 1986 paper, credited with a powerful algorithm for learning internal representations in multi-layered neural networks [1, 6] |
| Geoffrey Hinton | born 1947 | Co-author of the 1986 paper and a 2024 Nobel Prize in Physics laureate [1, 7] |
as of 2026-11-12
Milestones
The key moments, year by year
| Year | Event |
|---|---|
| 1970 | Seppo Linnainmaa's master's thesis at the University of Helsinki describes what is now called the reverse mode of differentiation [2, 3] |
| 1974 | Paul Werbos's Harvard doctoral thesis lays the foundation of backpropagation [4] |
| 1986 | Rumelhart, Hinton and Williams show in Nature that backpropagation can train networks with a hidden layer [1, 2] |
| 1989 | LeCun and colleagues use a backpropagation network to read handwritten US Postal Service zip code digits, with a 1% error rate [5] |
as of 2026-11-12
Sources
- Learning representations by back-propagating errors Nature (D. E. Rumelhart, G. E. Hinton and R. J. Williams), via the SAO/NASA Astrophysics Data System, 1986, vol. 323, October 1986
- Scientific Background to the Nobel Prize in Physics 2024 Royal Swedish Academy of Sciences, 2024
- Who Invented the Reverse Mode of Differentiation? Documenta Mathematica, Extra Volume ISMP (Andreas Griewank), copy hosted by the Bau Lab, 2012, pp. 389 to 400
- The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting John Wiley and Sons (P. J. Werbos), product page, 1994
- Handwritten Digit Recognition with a Back-Propagation Network Advances in Neural Information Processing Systems 2 (NIPS 1989) (Y. LeCun, B. E. Boser, J. S. Denker and others), 1989
- Remembering David E. Rumelhart (1942-2011) - Association for Psychological Science - APS Association for Psychological Science
- Press release: The Nobel Prize in Physics 2024 The Royal Swedish Academy of Sciences, NobelPrize.org, 2024