Attention and the transformer: Technologies
Technologies
The technologies that rely on it, and how each one uses it
Relied on by- LLM Large language models Self-attention models the relationships between all the words in a passage at once, the core of the transformer; its cost grows with the square of the passage length [2].
- DL Deep learning The transformer is a deep network built from stacks of attention layers; the base encoder stacks six identical layers [2].
- Srch Web search engine Google applies BERT, a transformer model, to ranking search results, and said it would help Search understand one in ten English searches in the US [6].
as of 2026-11-12
Sources
- Attention Is All You Need ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017
- Understanding searches better than ever before Google, The Keyword blog (Pandu Nayak), 2019