Machine Memory · 54 terms

Glossary

Every term the manual defines, in one place. Each entry links to the chapter that explains it properly.

A

Activation function
The small function a neuron applies to its weighted sum before passing it on. It bends the straight line of the sum, which is what lets stacked layers learn curved, complicated patterns.
→ How does a neural network work?
Attention
The mechanism that lets each word in a passage build a new version of itself from the other words, weighting each one by how relevant it is.
→ Attention and transformers.→ The context window.
Attention head
One of several attentions run side by side in the same layer, each with its own query, key and value matrices, so that each can track a different kind of relationship.
→ Attention and transformers.

B

Backpropagation
The method for working out how much each weight contributed to an error, by passing the error backwards through the network one layer at a time using the chain rule of calculus.
→ How does a neural network work?→ What is an embedding?→ Attention and transformers.
Bias
An extra number added to a neuron’s weighted sum. It shifts the point at which the neuron switches on, independently of its inputs.
→ How does a neural network work?

C

Causal mask
The rule that a word may only attend to itself and the words before it. It stops a model that is learning to predict the next word from simply reading it.
→ Attention and transformers.
Compaction
Replacing the oldest part of a long conversation with a shorter summary written by a model, so that the rest still fits in the context window.
→ The context window.
Context window
The maximum amount of text, measured in tokens, that a model can read at once. Everything the model uses to write its reply has to fit inside it.
→ The context window.
Contextual embedding
A word’s vector after the layers of a model have adjusted it using the rest of the sentence, so that “bank” near “river” and “bank” near “loan” end up in different places.
→ What is an embedding?
Cosine similarity
A measure of how closely two vectors point in the same direction, from −1 (opposite) through 0 (unrelated) to 1 (the same direction). The usual way to compare embeddings.
→ What is an embedding?→ Attention and transformers.

D

Dense vector
A fairly short list of numbers in which every number carries some information, as opposed to a long list that is almost all zeros.
→ What is an embedding?
Dimension
One of the numbers in a vector, thought of as one direction in space. An embedding with 768 numbers is a point in a 768-dimensional space.
→ What is an embedding?
Distributional hypothesis
The idea that words appearing in similar contexts have similar meanings. It is the reason embeddings can be learned from plain text.
→ What is an embedding?

E

Embedding
A list of numbers that stands for a piece of text (a token, a word, a sentence), arranged so that texts with similar meanings get similar numbers.
→ What is an embedding?→ Attention and transformers.
Epoch
One full pass of training over every example in the training data. Networks are usually trained for many epochs.
→ How does a neural network work?

G

GPU
Graphics processing unit. A chip built to do thousands of simple calculations at the same time, originally for drawing pixels and now the workhorse of neural networks.
→ How does a neural network work?→ Attention and transformers.
Gradient
For every weight, the direction and steepness of the slope of the loss. Following the gradient downhill is how a network improves.
→ How does a neural network work?
Gradient descent
Repeatedly nudging every weight a small step in the direction that lowers the loss, like walking downhill in fog by feeling the slope under your feet.
→ How does a neural network work?

H

Hidden layer
Any layer of neurons between the input and the output. Nobody tells it what to represent; it learns whatever intermediate features are useful.
→ How does a neural network work?

K

Key
In attention, the vector each word offers to be matched against: a description of what the word has to give. A query that points the same way as a key gets a high score.
→ Attention and transformers.
KV cache
The stored keys and values of every token read so far, kept in GPU memory so they do not have to be recomputed for each new token. It grows with the length of the context.
→ The context window.

L

Learning rate
How big each step of gradient descent is. Too small and training crawls; too large and it overshoots the bottom of the valley.
→ How does a neural network work?
Loss
A single number measuring how wrong the network’s answers are on a batch of examples. Training is the search for weights that make it small.
→ How does a neural network work?
Lost in the middle
The tendency, described in a 2023 study, for models to use information near the start or end of a long context better than information in the middle.
→ The context window.

M

Matrix
A rectangular grid of numbers. A whole layer’s weights fit in one matrix, so running a layer is one matrix multiplication.
→ How does a neural network work?→ Attention and transformers.

N

Neuron
In a neural network, a tiny function: multiply each input by a weight, add them up with a bias, and pass the result through an activation function.
→ How does a neural network work?

O

One-hot vector
A vector as long as the vocabulary, all zeros except for a single 1 in one word’s slot. It identifies a word without saying anything about its meaning.
→ What is an embedding?
Overfitting
When a network memorises its training examples, noise included, instead of learning the pattern behind them, so it does worse on anything new.
→ How does a neural network work?

P

Parameter
Any number the network learns during training: its weights and biases. Model sizes are quoted in parameters.
→ How does a neural network work?
Perceptron
Frank Rosenblatt’s 1958 learning machine: a single layer of artificial neurons whose weights adjust themselves from examples.
→ How does a neural network work?
Positional encoding
A pattern of numbers, different for every position in the text, mixed into each word’s vector so that attention can tell word order apart.
→ Attention and transformers.

Q

Query
In attention, the vector a word uses to ask what it is looking for. It is compared with every other word’s key.
→ Attention and transformers.

R

Recurrent neural network
A network that reads a sequence one item at a time, carrying a running summary from step to step. The standard approach to language before transformers.
→ Attention and transformers.
ReLU
Rectified linear unit. The activation function that outputs zero for negative inputs and the input itself otherwise. Simple, fast, and the default in most networks.
→ How does a neural network work?
Retrieval
Fetching only the stored pieces of text relevant to the current question and placing them in the context window, instead of including everything.
→ The context window.

S

Sentence embedding
A single vector for a whole sentence or passage, so that passages about similar things sit near each other whatever words they use.
→ What is an embedding?
Sigmoid
An S-shaped activation function that squashes any number into the range 0 to 1.
→ How does a neural network work?
Softmax
A function that turns any list of numbers into positive weights that add up to one, giving most of the share to the largest numbers.
→ Attention and transformers.
Stateless
Keeping nothing between requests. A language model does not hold a conversation in between turns; the whole conversation is sent again each time.
→ The context window.
System prompt
Instructions written by whoever built an app, placed at the start of the context window before the conversation. Users usually do not see it.
→ The context window.

T

Token
One of the pieces a tokenizer cuts text into: a whole common word, a fragment of a rarer one, or a single character. Models read and write tokens, not words.
→ What is an embedding?→ Attention and transformers.→ The context window.
Tokenizer
The program that splits text into tokens and replaces each one with its number in the vocabulary, before the model sees anything.
→ What is an embedding?
Training data
The labelled examples a network learns from, such as thousands of images of handwritten digits paired with the digit each one shows.
→ How does a neural network work?
Transformer
The neural network architecture introduced in 2017 that is built almost entirely from attention. Nearly every large language model is a transformer.
→ Attention and transformers.
Transformer block
One layer of a transformer: an attention step where words exchange information, followed by a small feed-forward network applied to each word on its own.
→ Attention and transformers.
Truncation
Dropping the oldest parts of a conversation so that the rest fits in the context window.
→ The context window.

V

Validation set
Examples held back from training and used only to check how well the network does on data it has never seen.
→ How does a neural network work?
Value
In attention, the information a word hands over to the words that attend to it, blended in proportion to their attention weights.
→ Attention and transformers.
Vector
An ordered list of numbers. Images, words and sounds all have to become vectors before a network can work with them.
→ How does a neural network work?
Vector index
A data structure that stores many embeddings and can quickly find the ones nearest to a new vector, even among millions.
→ What is an embedding?
Vocabulary
The fixed list of every token a model knows, usually tens of thousands to a couple of hundred thousand entries. A token’s id is its position in this list.
→ What is an embedding?

W

Weight
A number on a connection between two neurons that says how strongly the first one’s output should count towards the second. Learning is changing weights.
→ How does a neural network work?→ What is an embedding?
word2vec
A method published by a team at Google in 2013 for learning word embeddings quickly from very large amounts of text, famous for vectors whose directions encode analogies.
→ What is an embedding?