# Attention and the transformer: The idea

Source page: https://www.materialinsights.org/science/attention_transformer/sci-idea/

#### In plain words

The idea explained without jargon

A transformer reads a passage by letting every word look at every other word. Each word is turned into three lists of numbers: a query (what am I looking for), a key (what do I offer) and a value (what I pass on). A word's query is compared with every key to give scores, the scores are turned into weights that add up to one, and the word takes a weighted mix of the values. A word like 'it' can therefore draw on the noun it refers to, however far away. Because no word has to wait for the one before it, the work can be done in parallel ^[2]. Several such 'heads' run side by side so the model can attend to different kinds of relation at once ^[2].

as of 2026-11-12

#### Key equation

The equation that sums it up, and what each symbol means

**Equation**
: Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

**Symbols**
: {"Q":"the queries, one list of numbers per word","K":"the keys, one per word","V":"the values, one per word","d_k":"the length of the key lists","softmax":"the function that turns scores into positive weights that add up to one"}

**Note**
: Dividing by sqrt(d_k) keeps the scores from growing large, which would push the softmax into regions where it learns very slowly ^[2].

as of 2026-11-12

#### An analogy

A picture from everyday life that makes the idea easier to hold

A meeting where everyone writes a question on a card, reads the label on every other person's badge, and then listens most closely to the people whose badges match the question best. Older models were like passing a note along a line of people, one at a time. The picture stops being true because the questions and badges are lists of numbers learned in training, and the cost of everyone reading every badge grows quickly as the room gets bigger.

as of 2026-11-12

#### Sources

1. [Attention Is All You Need](https://ar5iv.labs.arxiv.org/html/1706.03762) ar5iv (arXiv Labs), full text of arXiv:1706.03762, 2017

## Other sections

- [Summary](https://www.materialinsights.org/science/attention_transformer/index.md)
- [Discovery](https://www.materialinsights.org/science/attention_transformer/sci-discovery/index.md)
- [Technologies](https://www.materialinsights.org/science/attention_transformer/sci-technologies/index.md)
