Attention and the transformer: Discovery
The story
How the idea was found and came to be accepted
Early neural translators encoded a source sentence into a single fixed-length vector and decoded the translation from it; Bahdanau, Cho and Bengio argued that this vector was a bottleneck and let the decoder search for the source words relevant to each target word, in a paper at ICLR 2015 [3]. Those systems still read word by word, which prevents parallel training within an example [2]. In June 2017 Vaswani and seven co-authors proposed the Transformer, based entirely on attention, and reported 28.4 BLEU on English-to-German and 41.8 on English-to-French translation, the latter after 3.5 days on eight GPUs [1, 2]. BERT followed in 2018, with new best results on eleven language tasks [5], and in 2021 a pure transformer applied to image patches did well on image classification [4].
as of 2026-11-12
People
The people behind it, their years and what each contributed
| Name | Years | Contribution |
|---|---|---|
| Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio | active in 2015 | Added a learned, soft search over the source words to neural translation [3] |
| Ashish Vaswani and co-authors | active in 2017 | Proposed the Transformer, built entirely on attention, at Google Brain, Google Research and the University of Toronto [1, 2] |
as of 2026-11-12
Milestones
The key moments, year by year
| Year | Event |
|---|---|
| 2015 | Bahdanau, Cho and Bengio's paper on jointly learning to align and translate appears at ICLR [3] |
| 2017 | 'Attention Is All You Need' introduces the Transformer; 28.4 BLEU on English-to-German translation [1] |
| 2018 | BERT is posted and sets new state-of-the-art results on eleven language tasks [5] |
| 2021 | The Vision Transformer, presented at ICLR 2021, applies a pure transformer to sequences of image patches [4] |
as of 2026-11-12
Sources
- Attention Is All You Need arXiv:1706.03762 (A. Vaswani and seven co-authors), 2017
- Attention Is All You Need ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017
- Neural Machine Translation by Jointly Learning to Align and Translate ICLR 2015, arXiv:1409.0473 (D. Bahdanau, K. Cho and Y. Bengio), 2015
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ICLR 2021, arXiv:2010.11929 (A. Dosovitskiy and co-authors), 2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding arXiv:1810.04805 (J. Devlin and co-authors), 2018