Foundations / Attention and transformers.

2503 words | Esment

Attention and transformers.

The mechanism that lets every word look at every other word.

╌╌╌╌

Read this sentence: “The animal didn’t cross the street because it was too tired.” You knew at once that “it” means the animal. Change one word, “because it was too wide”, and “it” now means the street. You did not run a grammar rule. You looked back, found the words that could matter, and weighed them.

For decades this was exactly what language software could not do well. Then, in 2017, a short paper with a cheeky title described a way for every word in a passage to look at every other word and decide which ones matter to it. The mechanism is called attention1, the architecture built around it is the transformer15, and almost every AI model you have used since runs on it.

Attention weights drawn as a grid under a glass plateTHECATSATONTHEMATKEYSQUERIES

Before attention

The previous chapter ended with a problem. An embedding6 gives “bank” one vector, whatever sentence it is in. To understand language, a model has to let words change each other, and for a long time the best way to do that was to read like a person reading aloud: one word at a time, left to right.

These models were called recurrent neural networks12. They kept a running summary, a single vector, and updated it with each new word. By the end of a sentence, the summary was supposed to hold everything that mattered. A refined version from 1997, the LSTM, was better at holding on to things, and by the mid-2010s these networks were translating text well enough to be used in real products.

An older network reads one word at a time and squeezes everything into one running summaryThecatthatthedogchasedsatONE SUMMARY, PASSED FROM WORD TO WORDBY “SAT”, LITTLE OF “CAT” IS LEFT

They had two deep problems. The first was the bottleneck: everything had to squeeze through that one summary, and early words faded as later ones arrived. The second was speed. Word ten could not be processed until word nine was done, so the work could not be spread across the thousands of cores of a GPU7. Training was slow, and long passages were out of reach.

In 2014, researchers working on translation tried a patch. Instead of relying only on the final summary, the translator could glance back at every word of the original sentence and choose which ones to focus on while writing each word of the translation. They called it attention. It worked so well that, three years later, a team at Google asked what would happen if you kept the attention and threw away everything else. Their paper was called “Attention Is All You Need”.

The idea of attention

Attention lets every word build a new version of itself out of the other words around it. For each word, the model asks: which of the other words are relevant to me, and how much? Then it blends their information into its own vector in those proportions.

The word it looks back at every earlier word; most of its attention goes to animal62%18%THEANIMALDIDN’TCROSSTHESTREETBECAUSEITWASTOOTIRED

In the example above, “it” spends most of its attention on “animal”. After this step, the vector for “it” contains a good deal of the vector for “animal”, so the layers above know what “it” refers to. In the same pass, “bank” can pull in “river”, and “tired” can pull in “animal” too. Every word does this at the same moment, which is the second half of the breakthrough: there is no waiting for the previous word, so the whole passage can be processed in parallel.

Query, key and value

How does a word decide what is relevant? The mechanism borrows an idea from databases. Each word is turned into three different vectors, each by multiplying its embedding by a different matrix9 of learned weights:

  • a query11, which describes what this word is looking for (“I am a pronoun; I need a noun that could be tired”);
  • a key8, which describes what this word has to offer (“I am a noun, an animal, the subject of the sentence”);
  • and a value17, which is the information it will actually hand over if someone pays attention to it.
One word's vector is multiplied by three matrices to give a query, a key and a valueITWQQUERYWHAT AM I LOOKING FOR?WKKEYWHAT DO I OFFER?WVVALUEWHAT DO I PASS ON?

The wording in quotes is only for intuition. Nobody tells the model what to put in its queries and keys. Like everything else, the three matrices start random and are shaped by training, through the same backpropagation3 as in the first chapter, until the queries and keys they produce happen to match when matching is useful.

Scores and softmax

To decide how much “it” should attend to “animal”, the model takes the query of “it” and the key of “animal” and computes their dot product, the same multiply-and-add we used for cosine similarity5 in the last chapter. If the query and key point the same way, the score is large. It does this for every pair of words.

Raw scores are awkward. Some are negative, and they do not add up to anything in particular. So the scores for each word pass through a function called softmax13, which turns any list of numbers into positive weights that add up to exactly one. Large scores take most of the share, small ones get almost nothing.

Raw scores, some negative, become weights between zero and one that add up to oneRAW SCORES q · k0.4THE3.0ANIMAL-0.6CROSS1.6STREET-0.2TIREDSOFTMAXWEIGHTS (ADD UP TO 100%)5%THE72%ANIMAL2%CROSS18%STREET3%TIRED

The last step is a weighted average. Each word’s new vector is the sum of every word’s value, multiplied by the share of attention it received. That is the whole of attention, and it fits in one line:

output = softmax( Q · Kᵀ ÷ √d ) · V

attention

Q, K and V are the queries, keys and values of every word stacked into matrices, so the entire calculation is a handful of matrix multiplications, which GPUs do extremely fast. The division by √d, the square root of the number of entries in each key, keeps the scores from growing so large that softmax hands all the attention to a single word.

No peeking

Models like ChatGPT and Claude are trained to predict the next word. That training would be pointless if each word could simply look ahead and see the answer. So in these models attention is given a causal mask4: a word may attend to itself and to every word before it, but the scores for every word after it are set to minus infinity, which softmax turns into exactly zero.

Drawn as a grid, with one row for each word and one column for each word it might look at, the mask cuts away the whole upper triangle. You will see it hatched in the next figure. It is also why these models are called decoders, and why they write strictly from left to right.

Many heads

A single round of attention can only blend words in one way at a time. But a word usually needs several kinds of context at once: the word just before it, the subject of the sentence, the thing a pronoun refers to, the opening of a quotation. So transformers run several attentions side by side, each with its own query, key and value matrices. Each is called a head2.

Three attention heads looking for three different relationships in the same sentenceHEAD 1THE WORD BEFOREHEAD 2THE SUBJECTHEAD 3ITSELFROWS: EACH WORD · COLUMNS: THE WORDS IT CAN SEE · HATCHED: THE FUTURE, HIDDEN

When researchers inspect trained models, they find heads with recognisable habits like these, alongside many more whose job is hard to describe in words. The original transformer used 8 heads; large models today use dozens. Their outputs are joined together and mixed by one more matrix before moving on.

Where each word sits

There is a strange gap in everything so far. Attention compares every word with every other word, but nothing in that comparison says which word came first. To attention on its own, “dog bites man” and “man bites dog” are the same bag of words.

The fix is to add a positional encoding10 to each embedding before attention starts: a pattern of numbers that is different for every position. The original paper built these from waves of different lengths, like the hands of a clock that each turn at a different speed. Read together, they give every position its own fingerprint, and nearby positions get similar ones.

Waves of different lengths give every position its own pattern of numbers01234567POSITIONDIM 1DIM 2DIM 3DIM 4

Most recent models use a refinement that rotates the queries and keys by an angle that depends on position, rather than adding a pattern to the embeddings. The goal is the same: to let attention know not just what each word is, but where it is.

The transformer block

Attention is the heart of a transformer, but not the whole of it. Each layer, or block16, has two halves. First comes attention, where words exchange information. Then each word’s vector passes on its own through a small ordinary neural network, the same kind as in the first chapter, which is where much of the model’s factual knowledge seems to be stored. Around both halves are shortcuts that add each half’s input back to its output, so that information can flow past a layer untouched when it has nothing to add.

Layers of a transformer block pulled apartEMBEDDINGATTENTIONADD & NORMFEED FORWARDADD & NORMOUTPUTSELF-ATTENTION8 HEADSMLP · 4× WIDTHNEXT LAYERTOKEN VECTORS

Then the block is repeated, the output of one becoming the input of the next. The original transformer stacked six. GPT-3, in 2020, stacked 96. Each pass refines the vectors a little further: early layers tend to sort out grammar and nearby words, later ones handle meaning, references and the shape of the whole passage.

Writing one token at a time

After the last block, the vector at the final position is compared with every row of a table much like the embedding table, giving a score for every token14 in the vocabulary. Softmax turns those scores into probabilities. The model picks one, usually one of the most likely but not always the very top, adds it to the end of the text, and runs the whole stack again.

After the cat sat on the, the model gives every possible next word a probability and picks oneThe cat sat on the matEVERY TOKEN IN THE VOCABULARY GETS A PROBABILITYmat41%floor17%sofa12%bed8%windowsill5%roof3%… AND 49,994 MORE, MOSTLY NEAR 0%PICK ONE,ADD IT,REPEAT

That is all a chatbot’s answer is: this loop, run once per token, hundreds of times. It feels like the model is planning a reply, and in a sense the planning is real, smeared across billions of weights. But mechanically, it only ever decides one token, then reads everything again to decide the next.

The price of looking everywhere

Attention’s great strength, that every word can look at every other word, is also its great cost. A passage of 1,000 tokens has a million pairs to score. Double the passage and the pairs quadruple. That quadratic growth is why every model has a limit on how much text it can consider at once, and why that limit has been one of the most fought-over numbers in the field.

That limit is called the context window. It decides what a model can see while it answers you, which means it decides what the model can “remember” in a conversation. It is the subject of the next chapter.

Glossary

1Attention — The mechanism that lets each word in a passage build a new version of itself from the other words, weighting each one by how relevant it is.

2Attention head — One of several attentions run side by side in the same layer, each with its own query, key and value matrices, so that each can track a different kind of relationship.

3Backpropagation — The method for working out how much each weight contributed to an error, by passing the error backwards through the network one layer at a time using the chain rule of calculus.

4Causal mask — The rule that a word may only attend to itself and the words before it. It stops a model that is learning to predict the next word from simply reading it.

5Cosine similarity — A measure of how closely two vectors point in the same direction, from −1 (opposite) through 0 (unrelated) to 1 (the same direction).

6Embedding — A list of numbers that stands for a piece of text, arranged so that texts with similar meanings get similar numbers.

7GPU — Graphics processing unit. A chip built to do thousands of simple calculations at the same time, originally for drawing pixels and now the workhorse of neural networks.

8Key — In attention, the vector each word offers to be matched against: a description of what the word has to give. A query that points the same way as a key gets a high score.

9Matrix — A rectangular grid of numbers. Multiplying a vector by a learned matrix turns it into a different vector, such as a query, a key or a value.

10Positional encoding — A pattern of numbers, different for every position in the text, mixed into each word’s vector so that attention can tell word order apart.

11Query — In attention, the vector a word uses to ask what it is looking for. It is compared with every other word’s key.

12Recurrent neural network — A network that reads a sequence one item at a time, carrying a running summary from step to step. The standard approach to language before transformers.

13Softmax — A function that turns any list of numbers into positive weights that add up to one, giving most of the share to the largest numbers.

14Token — One of the pieces a tokenizer cuts text into: a whole common word, a fragment of a rarer one, or a single character. Models read and write tokens, not words.

15Transformer — The neural network architecture introduced in 2017 that is built almost entirely from attention. Nearly every large language model is a transformer.

16Transformer block — One layer of a transformer: an attention step where words exchange information, followed by a small feed-forward network applied to each word on its own.

17Value — In attention, the information a word hands over to the words that attend to it, blended in proportion to their attention weights.

Get the next chapter by email.

One email per new chapter. Nothing else.

0.00