Material Insights · Understanding the Material World

Attention and the transformer: Discovery

The story

How the idea was found and came to be accepted

Early neural translators encoded a source sentence into a single fixed-length vector and decoded the translation from it; Bahdanau, Cho and Bengio argued that this vector was a bottleneck and let the decoder search for the source words relevant to each target word, in a paper at ICLR 2015 [3]. Those systems still read word by word, which prevents parallel training within an example [2]. In June 2017 Vaswani and seven co-authors proposed the Transformer, based entirely on attention, and reported 28.4 BLEU on English-to-German and 41.8 on English-to-French translation, the latter after 3.5 days on eight GPUs [1, 2]. BERT followed in 2018, with new best results on eleven language tasks [5], and in 2021 a pure transformer applied to image patches did well on image classification [4].

as of 2026-11-12

People

The people behind it, their years and what each contributed

NameYearsContribution
Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengioactive in 2015Added a learned, soft search over the source words to neural translation [3]
Ashish Vaswani and co-authorsactive in 2017Proposed the Transformer, built entirely on attention, at Google Brain, Google Research and the University of Toronto [1, 2]

as of 2026-11-12

Milestones

The key moments, year by year

YearEvent
2015Bahdanau, Cho and Bengio's paper on jointly learning to align and translate appears at ICLR [3]
2017'Attention Is All You Need' introduces the Transformer; 28.4 BLEU on English-to-German translation [1]
2018BERT is posted and sets new state-of-the-art results on eleven language tasks [5]
2021The Vision Transformer, presented at ICLR 2021, applies a pure transformer to sequences of image patches [4]

as of 2026-11-12

Sources

  1. Attention Is All You Need arXiv:1706.03762 (A. Vaswani and seven co-authors), 2017
  2. Attention Is All You Need ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017
  3. Neural Machine Translation by Jointly Learning to Align and Translate ICLR 2015, arXiv:1409.0473 (D. Bahdanau, K. Cho and Y. Bengio), 2015
  4. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ICLR 2021, arXiv:2010.11929 (A. Dosovitskiy and co-authors), 2021
  5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding arXiv:1810.04805 (J. Devlin and co-authors), 2018