Foundations / What is an embedding?

3162 words | Esment

What is an embedding?

How meaning becomes a position in space, and why nearby means similar.

╌╌╌╌

A neural network only understands numbers. The last chapter got away with that easily: a photo is already a grid of brightness values, so you can pour it straight into the first layer. Words are harder. There is nothing numerical about the word “cat”, and the obvious ways of turning it into a number throw away the one thing we care about, which is what it means.

The solution is called an embedding7: a list of a few hundred numbers for every word, chosen so that words with similar meanings get similar lists. It sounds like a modest bookkeeping trick. It turns out to be one of the most useful ideas in the whole field, and it is also how most AI memory systems find things, so it is worth understanding properly.

An embedding table: one row of numbers per token; the row for cat is lifted out“CAT” IS TOKEN 9,246SO: FETCH ROW 9,246EMBEDDING TABLEONE ROW PER TOKENITS ROW: 8 NUMBERS(768 IN A REAL MODEL)50,000 ROWSONE PER TOKEN

From text to tokens

Before anything else, text has to be cut into pieces. You might expect those pieces to be words, but they are usually a little smaller. A tokenizer12 splits text into tokens11: common words stay whole, rarer words are broken into frequent fragments, and anything truly unusual falls back to single letters. “The” is one token. “Remembering” is three.

The word remembering split into three tokens, each with an idremem18604TOKEN 1ber1941TOKEN 2ing278TOKEN 311 LETTERS → 3 TOKENS → 3 NUMBERS

The full list of pieces a model knows is its vocabulary14, typically somewhere between 30,000 and a couple of hundred thousand entries. Each one has a fixed position in that list, so a tokenizer’s real output is not text at all but a row of whole numbers. As a rule of thumb, in English a token is about three quarters of a word, which is why the limits you will meet in the context window chapter are quoted in tokens rather than words.

Fragments are a sensible compromise. A vocabulary of whole words would need millions of entries and would still meet words it had never seen. A vocabulary of letters would be tiny, but every sentence would become a very long string of symbols that each mean almost nothing. Pieces like “remem”, “ber” and “ing” are common enough to learn well and meaningful enough to be worth learning.

Why a number is not enough

So “cat” has become, say, token 9,246. Could we just feed 9,246 into the network? We could, but it would be nonsense. The numbers are positions in a list, nothing more. Token 9,247 is not a slightly bigger cat; it is whatever happened to come next when the vocabulary was built. Any arithmetic the network did on those numbers would be arithmetic on accidents.

The classic fix is a one-hot8 vector: a list as long as the vocabulary, all zeros except for a single 1 in the word’s own slot. Nothing is accidental any more, but nothing is meaningful either. Every word is now exactly as different from every other word. “Cat” is as far from “dog” as it is from “carburettor”, and a list of fifty thousand numbers, all but one of them zero, is a very wasteful way to say so.

A word as one switched-on slot among fifty thousand, and as eight meaningful numbersONE-HOT: ONE SLOT PER WORD… × 50,000CATDOGEVERY WORD IS EXACTLY AS FAR FROM EVERY OTHERDENSE: 8 NUMBERS THAT CARRY MEANINGCAT0.8-0.30.60.1-0.70.40.9-0.2DOG0.7-0.20.50.3-0.60.50.8-0.1SIMILAR NUMBERS,SIMILAR WORDS

What we want instead is a short, dense4 list, a few hundred numbers, all of them doing some work, arranged so that “cat” and “dog” come out similar and “carburettor” comes out different. Nobody can write those numbers by hand. The question is where they could possibly come from.

Words are known by their company

In 1957 the British linguist J. R. Firth summed up an old idea in a line that is now quoted in almost every course on the subject: “You shall know a word by the company it keeps.” Linguists call it the distributional hypothesis6: words that turn up in the same kinds of sentences tend to mean similar things.

Three sentences with the same gap: cat and dog both fitSENTENCE 1The cat/dog chased the mouse across the kitchen.SENTENCE 2I fed the cat/dog before I left for work.SENTENCE 3The cat/dog fell asleep on the warm windowsill.WORDS THAT FIT THE SAME GAPS END UP WITH SIMILAR VECTORS

You can feel this yourself. Cover up one word in a sentence and you can usually guess not the exact word but its family. “I fed the ___ before I left for work” wants an animal, probably a pet. You did not need a definition of “cat” to know it fits; you only needed to have seen it in sentences like this before.

That is the whole trick. If we can find numbers for every word such that words appearing in similar sentences get similar numbers, the meaning will come along for free. We never have to explain to the computer what a cat is. We only have to show it an enormous amount of text and ask it to be good at guessing the gaps.

A map of meaning

It helps to think of each list of numbers as a position. Two numbers give you a point on a flat map; three give you a point in a room. An embedding with 768 numbers is a point in a space with 768 dimensions5. Nobody can picture that, and nobody has to. Everything you know about distance on a map still works, and that is all we will need.

Words as points in a three dimensional space, grouped by meaningANIMALSFOODPLACES“CAT”QUERYDIM 1DIM 2DIM 3

Once words are positions, meaning becomes geography. Animals cluster in one region, foods in another, cities in a third. Within the animal region, pets sit closer to each other than to farm animals. None of these regions were drawn by a person. They appear on their own, because words in the same neighbourhood kept turning up in the same kinds of sentences.

It is tempting to ask what each dimension means. Is dimension 41 “furriness”? Almost never. The meaning is spread across all of them at once, the same way the knowledge in the last chapter’s network was smeared across all of its weights. Individual axes are rarely readable. Directions and distances are.

Measuring closeness

If meaning is position, similarity should be distance. In practice, people usually measure the angle between two vectors rather than the gap between their tips, because the angle ignores how long a vector is and only asks which way it points. The measure is called cosine similarity3.

similarity(a, b) = (a · b) ÷ (|a| × |b|)

cosine similarity

The dot on top multiplies the two lists number by number and adds up the results. Dividing by the lengths keeps the answer between −1 and 1. Two vectors pointing the same way score 1. Two at right angles, which in this world means “unrelated”, score 0.

Cat and dog point in nearly the same direction; car points somewhere elseCATDOGCAR16°56°COSINE SIMILARITYCAT · DOG 0.96CAT · CAR 0.561 = SAME DIRECTION0 = UNRELATED

Notice what this measure does not care about. “Cat” and “car” look almost identical as strings of letters, and a spell checker would call them neighbours. In embedding space they are not close at all, because they do not keep the same company. This is the first sign of what embeddings are good for: they compare meaning, not spelling.

Directions that mean something

In 2013 a team at Google led by Tomas Mikolov released word2vec16, a fast way of learning embeddings from billions of words of news text. The vectors were good, but what made the paper famous was something nobody had designed. Directions in the space carried meaning too.

The step from France to Paris is the same as the step from Italy to RomeFRANCEPARISITALYROME“CAPITAL OF”PARIS − FRANCE + ITALY ≈ ROME

Take the vector for “Paris”, subtract “France”, add “Italy”, and the nearest word to the result is “Rome”. The step from a country to its capital is, roughly, the same arrow wherever you start. So is the step from a verb to its past tense, or from a singular noun to its plural. Analogies had become arithmetic.

These results are real but easy to oversell. They work best on clean, frequent relationships and fail quietly on many others; in some of the famous examples the answer only comes out right once the original words are excluded from the search. The lasting lesson is less about party tricks and more about structure: the space is not a jumble. It has consistent directions in it, and a network built on top of it can use them.

Where the numbers come from

So who decides that “cat” should be 0.8, −0.3, 0.6…? Nobody. The embedding table is just another set of weights15, trained exactly like the ones in the previous chapter. It starts as random noise. The network is given a task, such as predicting a missing word from the words around it, and every time it guesses wrong, backpropagation1 works out how each number in each row contributed to the mistake and nudges it.

Over billions of examples, the nudges add up. Every time “cat” and “dog” appear in similar sentences, their rows receive similar pushes, so they drift towards each other. Words that never share company drift apart. The map in figure 5 is not designed; it is sediment, laid down one gradient step at a time.

In a large language model the embedding table is the very first layer. Each token’s id selects one row, and that row, not the id, is what flows into the rest of the network. A model with a vocabulary of 100,000 tokens and 4,096 numbers per row keeps more than 400 million weights in this table alone.

One word, many meanings

Word2vec and its cousins gave every word exactly one vector. That is a problem for words like “bank”, which might mean the side of a river or the place that holds your savings. A single fixed vector has to sit somewhere between the two meanings, close to neither.

The word bank starts as one point and moves towards river words or money words depending on the sentenceRIVERMONEYwatershorefishboatloancashaccountsavingsbankONE VECTOR, NO CONTEXTriver bankbank loan

Modern models fix this by treating the lookup as only a starting point. The table still gives “bank” one vector, but the layers above adjust it using the rest of the sentence. By the time it has passed through them, the “bank” in “we sat on the bank and watched the boats” has moved towards water, and the “bank” in “the bank refused the loan” has moved towards money. These adjusted vectors are called contextual embeddings2, and they appeared around 2018 in models such as ELMo and BERT.

The machinery that does the adjusting, letting every word look at every other word and decide which ones matter to it, is called attention. It is the subject of the next chapter, and it is the reason the models of the last few years are so much better than the ones before them.

Nothing stops us embedding more than a word. A model can read a whole sentence, or a paragraph, and condense it into a single vector that captures what it is about. These sentence embeddings10 behave just like word vectors: passages about similar things end up near each other, whatever words they happen to use.

That makes a new kind of search possible. Store a vector for every note, document or memory you have. When a question arrives, embed it too, and fetch the stored vectors that sit closest to it. This is semantic search9, and it finds things by meaning rather than by matching words.

A question becomes a point; the three memories closest to it are returned123QUESTIONWhat do I usuallydrink in the morning?NEAREST MEMORIES1Oat milk flat white, no sugar2Green tea after lunch3Cutting down on caffeineEVERYTHING ELSE STAYS PUT:DENTIST, FLIGHTS, STAND-UPS…

Ask “what do I usually drink in the morning?” and a keyword search looks for “drink” and “morning”. A semantic search finds “oat milk flat white, no sugar”, because in embedding space coffee orders live near questions about morning drinks. At scale, the stored vectors go into a vector index13 built to find the nearest neighbours among millions of points in a few milliseconds. Almost every system that gives an assistant a memory, or lets it answer questions about your documents, has one of these at its heart.

What embeddings get wrong

Embeddings measure similarity, and similarity is not the same thing as relevance. A few failure modes come up again and again.

Negation is weak. “I love coffee” and “I don’t drink coffee” are about the same topic and use most of the same words, so they often land close together. A system that retrieves by similarity alone can fetch the opposite of what you meant.

Exact details blur. Embeddings are good at gist and poor at specifics. Two invoices for different amounts, or two meetings on different dates, can look nearly identical. Names, numbers and codes are usually better matched by plain old keyword search, which is why good systems use both.

They inherit their training text. If the text a model learned from associates certain jobs with certain genders, the geometry will too. The space is a mirror of the writing it was built from, flaws included.

Each model has its own map. Vectors from two different embedding models cannot be compared, any more than coordinates from two different maps with different origins. Change the model and every stored vector has to be computed again.

Here is where we stand. Text becomes tokens, tokens become rows in a table, and the rows are arranged so that position means meaning. What is still missing is a way for words to change each other: for “bank” to notice the river, and for “it” to work out what it refers to. That mechanism is attention, and it is next.

Glossary

1Backpropagation — The method for working out how much each weight contributed to an error, by passing the error backwards through the network one layer at a time using the chain rule of calculus.

2Contextual embedding — A word’s vector after the layers of a model have adjusted it using the rest of the sentence, so that “bank” near “river” and “bank” near “loan” end up in different places.

3Cosine similarity — A measure of how closely two vectors point in the same direction, from −1 (opposite) through 0 (unrelated) to 1 (the same direction). The usual way to compare embeddings.

4Dense vector — A fairly short list of numbers in which every number carries some information, as opposed to a long list that is almost all zeros.

5Dimension — One of the numbers in a vector, thought of as one direction in space. An embedding with 768 numbers is a point in a 768-dimensional space.

6Distributional hypothesis — The idea that words appearing in similar contexts have similar meanings. It is the reason embeddings can be learned from plain text.

7Embedding — A list of numbers that stands for a piece of text (a token, a word, a sentence), arranged so that texts with similar meanings get similar numbers.

8One-hot vector — A vector as long as the vocabulary, all zeros except for a single 1 in one word’s slot. It identifies a word without saying anything about its meaning.

10Sentence embedding — A single vector for a whole sentence or passage, so that passages about similar things sit near each other whatever words they use.

11Token — One of the pieces a tokenizer cuts text into: a whole common word, a fragment of a rarer one, or a single character. Models read and write tokens, not words.

12Tokenizer — The program that splits text into tokens and replaces each one with its number in the vocabulary, before the model sees anything.

13Vector index — A data structure that stores many embeddings and can quickly find the ones nearest to a new vector, even among millions.

14Vocabulary — The fixed list of every token a model knows, usually tens of thousands to a couple of hundred thousand entries. A token’s id is its position in this list.

15Weight — A number the network learns during training. The rows of an embedding table are weights like any other.

16word2vec — A method published by a team at Google in 2013 for learning word embeddings quickly from very large amounts of text, famous for vectors whose directions encode analogies.

Get the next chapter by email.

One email per new chapter. Nothing else.

0.00