Material Insights · Understanding the Material World

Large language models

Transformer networks trained on vast text predict words and can write, answer and reason.

Computing and AI · AI and data · No. 293

  • Networked and AI age
  • Readiness 9 of 9
  • Platform
Computing and AI · Technology

Models trained to guess the next word now power ChatGPT, which OpenAI says has 800 million users every week.

Chart: Illustration: Large language models

Where it comes from

Chart: Map of where it comes from

Neural language models go back to Bengio's work of 2000. The transformer of 2017 made them scale, OpenAI sold GPT-3 access from June 2020, and ChatGPT launched in November 2022.

Three things to know
  1. The transformer design, published by Google researchers in August 2017, links every word in a passage to every other word at once.
  2. The cost of output as good as GPT-3.5 fell from 20 US dollars per million tokens in November 2022 to 7 cents by October 2024.
  3. Claude Shannon measured how well people could guess the next letter of English text in a paper of 1951, finding English about 75% redundant.
The journey
  1. 1 TextHuge collections of books, web pages and code
  2. 2 TokensText is cut into word pieces turned into numbers
  3. 3 Pre-trainingA transformer learns to predict the next token, on many GPUs
  4. 4 Fine-tuningExamples and human feedback teach it to follow instructions
  5. 5 UseChat assistants, coding tools and search answers
Go deeper pick a tab to open it in full