# Attention and the transformer

A network that weighs every part of its input against every other learns language and other sequences at scale.

Source page: https://www.materialinsights.org/science/attention_transformer/

Mathematics and information · Machine learning · No. 131

- Networked and AI age
- Science
 Mathematics and information · The science behind technology

A 2017 translation paper dropped recurrent layers and used attention alone; the design is now the standard for language tasks.

Chart: Illustration: Attention and the transformer

##### Where it comes from

Chart: Map of where it comes from

Earlier translation systems squeezed a sentence into one fixed-length vector. Bahdanau, Cho and Bengio let the decoder search the source words instead in 2015, and in June 2017 Vaswani and co-authors used attention alone.

##### Three things to know

1. The 2017 paper called the transformer the first sequence model based entirely on attention.
2. Its best English-to-French translation model trained for 3.5 days on eight GPUs.
3. In 2021 a pure transformer applied to image patches was shown to do well at image classification.

##### The journey

1. 1 **Bottleneck**Translators squeeze a sentence into one fixed-length vector
2. 2 **Attention**The decoder learns to look at the relevant source words, 2015
3. 3 **Transformer**Attention replaces recurrence altogether, 2017
4. 4 **Pre-training**BERT trains on unlabelled text, 2018
5. 5 **Spread**Image patches treated like words, 2021

##### Go deeper pick a tab to open it in full

## Other sections

- [The idea](https://www.materialinsights.org/science/attention_transformer/sci-idea/index.md)
- [Discovery](https://www.materialinsights.org/science/attention_transformer/sci-discovery/index.md)
- [Technologies](https://www.materialinsights.org/science/attention_transformer/sci-technologies/index.md)
