# Attention and the transformer: Discovery

Source page: https://www.materialinsights.org/science/attention_transformer/sci-discovery/

#### The story

How the idea was found and came to be accepted

Early neural translators encoded a source sentence into a single fixed-length vector and decoded the translation from it; Bahdanau, Cho and Bengio argued that this vector was a bottleneck and let the decoder search for the source words relevant to each target word, in a paper at ICLR 2015 ^[3]. Those systems still read word by word, which prevents parallel training within an example ^[2]. In June 2017 Vaswani and seven co-authors proposed the Transformer, based entirely on attention, and reported 28.4 BLEU on English-to-German and 41.8 on English-to-French translation, the latter after 3.5 days on eight GPUs ^[1, 2]. BERT followed in 2018, with new best results on eleven language tasks ^[5], and in 2021 a pure transformer applied to image patches did well on image classification ^[4].

as of 2026-11-12

#### People

The people behind it, their years and what each contributed

| Name | Years | Contribution |
|---|---|---|
| Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio | active in 2015 | Added a learned, soft search over the source words to neural translation ^[3] |
| Ashish Vaswani and co-authors | active in 2017 | Proposed the Transformer, built entirely on attention, at Google Brain, Google Research and the University of Toronto ^[1, 2] |

as of 2026-11-12

#### Milestones

The key moments, year by year

| Year | Event |
|---|---|
| 2015 | Bahdanau, Cho and Bengio's paper on jointly learning to align and translate appears at ICLR ^[3] |
| 2017 | 'Attention Is All You Need' introduces the Transformer; 28.4 BLEU on English-to-German translation ^[1] |
| 2018 | BERT is posted and sets new state-of-the-art results on eleven language tasks ^[5] |
| 2021 | The Vision Transformer, presented at ICLR 2021, applies a pure transformer to sequences of image patches ^[4] |

as of 2026-11-12

#### Sources

1. [Attention Is All You Need](https://arxiv.org/abs/1706.03762) arXiv:1706.03762 (A. Vaswani and seven co-authors), 2017
2. [Attention Is All You Need](https://ar5iv.labs.arxiv.org/html/1706.03762) ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017
3. [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/pdf/1409.0473) ICLR 2015, arXiv:1409.0473 (D. Bahdanau, K. Cho and Y. Bengio), 2015
4. [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/pdf/2010.11929) ICLR 2021, arXiv:2010.11929 (A. Dosovitskiy and co-authors), 2021
5. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805) arXiv:1810.04805 (J. Devlin and co-authors), 2018

## Other sections

- [Summary](https://www.materialinsights.org/science/attention_transformer/index.md)
- [The idea](https://www.materialinsights.org/science/attention_transformer/sci-idea/index.md)
- [Technologies](https://www.materialinsights.org/science/attention_transformer/sci-technologies/index.md)
