Material Insights · Understanding the Material World

Attention and the transformer

A network that weighs every part of its input against every other learns language and other sequences at scale.

Mathematics and information · Machine learning · No. 131

  • Networked and AI age
  • Science
Mathematics and information · The science behind technology

A 2017 translation paper dropped recurrent layers and used attention alone; the design is now the standard for language tasks.

Chart: Illustration: Attention and the transformer

Where it comes from

Chart: Map of where it comes from

Earlier translation systems squeezed a sentence into one fixed-length vector. Bahdanau, Cho and Bengio let the decoder search the source words instead in 2015, and in June 2017 Vaswani and co-authors used attention alone.

Three things to know
  1. The 2017 paper called the transformer the first sequence model based entirely on attention.
  2. Its best English-to-French translation model trained for 3.5 days on eight GPUs.
  3. In 2021 a pure transformer applied to image patches was shown to do well at image classification.
The journey
  1. 1 BottleneckTranslators squeeze a sentence into one fixed-length vector
  2. 2 AttentionThe decoder learns to look at the relevant source words, 2015
  3. 3 TransformerAttention replaces recurrence altogether, 2017
  4. 4 Pre-trainingBERT trains on unlabelled text, 2018
  5. 5 SpreadImage patches treated like words, 2021
Go deeper pick a tab to open it in full