Attention, step by step

After the embedding step, every token is a vector, but each vector only knows about its own token. The vector for bank is the same in "river bank" and "bank account". Attention is the mechanism that lets each position look at the other positions and pull in what it needs, so that by the end of the network, bank near river means something different from bank near account.

The problem attention solves

Think of each position as asking a question about its context:

• A pronoun asks: which noun earlier in the text do I refer to?
• An adjective's noun asks: what's describing me?
• A token in code asks: where was this variable defined?

The answer is a weighted mix of information from other positions. The weights should depend on content, not fixed distance, and they should be computed fresh for every input. Attention does exactly that, with three learned projections.