# Large language models

Transformer networks trained on vast text predict words and can write, answer and reason.

Source page: https://www.materialinsights.org/technologies/large_language_models/

Computing and AI · AI and data · No. 293

- Networked and AI age
- Readiness 9 of 9
- Platform
 Computing and AI · Technology

Models trained to guess the next word now power ChatGPT, which OpenAI says has 800 million users every week.

Chart: Illustration: Large language models

##### Where it comes from

Chart: Map of where it comes from

Neural language models go back to Bengio's work of 2000. The transformer of 2017 made them scale, OpenAI sold GPT-3 access from June 2020, and ChatGPT launched in November 2022.

##### Three things to know

1. The transformer design, published by Google researchers in August 2017, links every word in a passage to every other word at once.
2. The cost of output as good as GPT-3.5 fell from 20 US dollars per million tokens in November 2022 to 7 cents by October 2024.
3. Claude Shannon measured how well people could guess the next letter of English text in a paper of 1951, finding English about 75% redundant.

##### The journey

1. 1 **Text**Huge collections of books, web pages and code
2. 2 **Tokens**Text is cut into word pieces turned into numbers
3. 3 **Pre-training**A transformer learns to predict the next token, on many GPUs
4. 4 **Fine-tuning**Examples and human feedback teach it to follow instructions
5. 5 **Use**Chat assistants, coding tools and search answers

##### Go deeper pick a tab to open it in full

## Other sections

- [Overview](https://www.materialinsights.org/technologies/large_language_models/tech-overview/index.md)
- [Science](https://www.materialinsights.org/technologies/large_language_models/tech-science/index.md)
- [Evolution](https://www.materialinsights.org/technologies/large_language_models/tech-evolution/index.md)
- [Energy footprint](https://www.materialinsights.org/technologies/large_language_models/tech-energy-footprint/index.md)
- [Industries](https://www.materialinsights.org/technologies/large_language_models/tech-industries/index.md)
