Attention and the transformer
A network that weighs every part of its input against every other learns language and other sequences at scale.
Mathematics and information · Machine learning · No. 131
- Networked and AI age
- Science
A 2017 translation paper dropped recurrent layers and used attention alone; the design is now the standard for language tasks.
Chart: Illustration: Attention and the transformer
Where it comes from
Chart: Map of where it comes from
Earlier translation systems squeezed a sentence into one fixed-length vector. Bahdanau, Cho and Bengio let the decoder search the source words instead in 2015, and in June 2017 Vaswani and co-authors used attention alone.
Three things to know
- The 2017 paper called the transformer the first sequence model based entirely on attention.
- Its best English-to-French translation model trained for 3.5 days on eight GPUs.
- In 2021 a pure transformer applied to image patches was shown to do well at image classification.
The journey
- 1 BottleneckTranslators squeeze a sentence into one fixed-length vector
- 2 AttentionThe decoder learns to look at the relevant source words, 2015
- 3 TransformerAttention replaces recurrence altogether, 2017
- 4 Pre-trainingBERT trains on unlabelled text, 2018
- 5 SpreadImage patches treated like words, 2021