Large language models: Science
Principles
The science it rests on, with why each principle matters to it
Relies on- Attention and the transformer Self-attention models the relationships between all words in a passage at once, the core of the transformer [1].
- Backpropagation and gradient learning The models' weights are trained by backpropagation, the method set out in 1986 [6].
- Information theory Shannon measured the information in English by asking how well the next letter can be predicted [5]; language models are trained to make the same prediction, and their error is measured in the same terms.
as of 2026-11-11
Discoveries
The discoveries that made it possible: when, by whom and what was found
| Year | Who | What |
|---|---|---|
| 1951 | Claude Shannon, Bell Labs | Estimated the entropy and redundancy of English from how well the next letter can be predicted [5] |
| 1986 | David Rumelhart, Geoffrey Hinton and Ronald Williams | Backpropagation for training networks with hidden layers [6] |
| 2000 | Yoshua Bengio | A neural probabilistic language model with word embeddings [6] |
| 2017 | Google researchers | The Transformer, built on self-attention [1] |
as of 2026-11-11
The science explained
The science behind it explained for a curious reader
Shannon showed that English is very predictable: given the start of a sentence, people can often guess the next letter, and he put its redundancy at roughly 75% [5]. A language model plays the same guessing game with whole word pieces, billions of times over. To guess well across every kind of text, it must pick up spelling, grammar, facts and the shape of arguments. The transformer makes this possible at scale. Instead of reading words one by one, each word attends to every other word in the passage, like a reader who can see the whole page at once and draw lines between the words that belong together [1]. Stacking many such layers and training them on vast text gives the model its fluency.
as of 2026-11-11
Sources
- Transformer: A Novel Neural Network Architecture for Language Understanding Google Research Blog (Jakob Uszkoreit), 2017
- Prediction and Entropy of Printed English Nokia Bell Labs, publications (C. E. Shannon, Bell System Technical Journal), 1951
- Fathers of the Deep Learning Revolution Receive ACM A.M. Turing Award Association for Computing Machinery, news release distributed on EurekAlert!, 2019