Material Insights · Understanding the Material World

Attention and the transformer: Technologies

Technologies

The technologies that rely on it, and how each one uses it

Relied on by
  • LLM Large language models Self-attention models the relationships between all the words in a passage at once, the core of the transformer; its cost grows with the square of the passage length [2].
  • DL Deep learning The transformer is a deep network built from stacks of attention layers; the base encoder stacks six identical layers [2].
  • Srch Web search engine Google applies BERT, a transformer model, to ranking search results, and said it would help Search understand one in ten English searches in the US [6].

as of 2026-11-12

Sources

  1. Attention Is All You Need ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017
  2. Understanding searches better than ever before Google, The Keyword blog (Pandu Nayak), 2019