Large language models
Transformer networks trained on vast text predict words and can write, answer and reason.
Computing and AI · AI and data · No. 293
- Networked and AI age
- Readiness 9 of 9
- Platform
Models trained to guess the next word now power ChatGPT, which OpenAI says has 800 million users every week.
Chart: Illustration: Large language models
Where it comes from
Chart: Map of where it comes from
Neural language models go back to Bengio's work of 2000. The transformer of 2017 made them scale, OpenAI sold GPT-3 access from June 2020, and ChatGPT launched in November 2022.
Three things to know
- The transformer design, published by Google researchers in August 2017, links every word in a passage to every other word at once.
- The cost of output as good as GPT-3.5 fell from 20 US dollars per million tokens in November 2022 to 7 cents by October 2024.
- Claude Shannon measured how well people could guess the next letter of English text in a paper of 1951, finding English about 75% redundant.
The journey
- 1 TextHuge collections of books, web pages and code
- 2 TokensText is cut into word pieces turned into numbers
- 3 Pre-trainingA transformer learns to predict the next token, on many GPUs
- 4 Fine-tuningExamples and human feedback teach it to follow instructions
- 5 UseChat assistants, coding tools and search answers