2990 words | Esment
How does a neural network work?
From a single artificial neuron to a network that reads handwriting: weights, layers, and the small piece of calculus that makes them learn.
╌╌╌╌
There is a slightly embarrassing secret at the centre of modern AI: the basic idea is old, simple and fits on a napkin. The chatbots, the image generators and the thing that finishes your sentences are all built from the same unit, an artificial neuron12, repeated billions of times and tuned by a procedure you could explain to a curious twelve year old.
What makes them powerful is not any single clever part. It is scale, patience, and a piece of calculus that lets a pile of numbers correct itself. By the end of this chapter you should be able to look at a picture like the one below and know what every layer is doing.
A brief history of the artificial neuron
In 1943 a neurophysiologist, Warren McCulloch, and a young logician, Walter Pitts, published a paper describing the brain’s nerve cells as tiny switches. Each one adds up the signals arriving from its neighbours and, if the total crosses a threshold, fires a signal of its own. Wire enough of these switches together, they argued, and you can compute anything a logic circuit can.
Their neurons could not learn; someone had to set every connection by hand. That changed in 1958, when Frank Rosenblatt built the perceptron15. The Mark I Perceptron was a cabinet of wires fed by a 20 by 20 grid of light sensors, with its connection strengths stored on motor-driven dials. Show it a shape, tell it whether it guessed right, and the dials would turn a little. After enough examples it could tell simple shapes apart without anyone programming the rule.
The excitement did not last. In 1969 Marvin Minsky and Seymour Papert showed that a single layer of perceptrons cannot learn some embarrassingly simple patterns, and funding dried up for more than a decade. Everyone suspected that stacking several layers would fix the problem. Nobody knew how to train a stack.
The answer, backpropagation2, was popularised in 1986, and the ideas in this chapter were essentially complete by then. What was missing was data and computing power. Both arrived around 2012, when a network trained on two consumer graphics cards crushed the competition at recognising objects in photos. Everything since has been, to a surprising degree, the same recipe made bigger.
One neuron
Forget biology for a moment. An artificial neuron is a very small function. It takes in a few numbers, multiplies each one by its own weight21, adds the results together with one extra number called the bias3, and passes the total through a final squashing step.
z = w₁·x₁ + w₂·x₂ + b → output = σ(z)
the whole neuron
The weights are the interesting part. A large positive weight means “this input matters, and more of it should push my output up”. A negative weight means the opposite. A weight near zero means “ignore this input”. Below, the weight on the first input sweeps back and forth; watch how the sum, and then the output, follow it. You can grab the slider and set it yourself.
That is genuinely all a single neuron does. It is a weighted vote. On its own it can only draw one straight dividing line through its inputs, which is exactly the limitation Minsky and Papert pointed out. The trick is in what happens when you put thousands of them together, and in the last step, the squash.
Why the squash?
The final step is called the activation function1. Early networks used a hard step: output 1 if the sum is above zero, otherwise 0, just like McCulloch and Pitts’ switches. The trouble with a step is that it is flat almost everywhere, so a small change to a weight usually changes nothing at all, and the network gets no hint about which way to improve.
The smooth sigmoid17 fixed that: an S-shaped curve that still squeezes everything between 0 and 1 but changes gently, so every nudge to a weight has some effect. Today most networks use something even simpler, the ReLU16, which just replaces negative numbers with zero.
It looks like a detail, but without this bend the whole network collapses. Stack as many straight-line functions as you like and the result is still one straight line. The bend is what lets layers build on each other and carve out curved, intricate boundaries, like the difference between a scrawled 7 and a scrawled 1.
From pixels to numbers
A network only understands numbers, so everything it sees has to become a vector20, a list of numbers in a fixed order. For a picture that is easy. Take a small greyscale image of a handwritten digit, read the pixels row by row, and write down how dark each one is, from 0 for blank paper to 1 for solid ink.
The classic dataset of handwritten digits uses 28 by 28 pixel images, so each digit becomes 784 numbers. We are drawing 8 by 8 here so you can see every square. Nothing else about the method changes. Text, sound and video take more work to turn into numbers, and the next chapter is about exactly that, but the destination is always the same: a vector.
Layers
Now connect those 64 numbers to a row of neurons. Every neuron in this first layer sees every pixel, each through its own weight, and produces one number. Feed those numbers into a second row of neurons, and those into a final row with one neuron for each digit from 0 to 9. The network’s answer is whichever output neuron ends up with the largest value.
The rows in the middle are called hidden layers8, because nobody tells them what to do. We only ever say what the input is and what the output should have been. What the hidden neurons end up representing is decided entirely by training, and when researchers look inside trained networks they find a pleasing pattern. Early neurons respond to simple things like a short stroke at a particular angle. Later ones combine those into loops, corners and crossings. The last layer only has to notice that “a horizontal stroke on top plus a diagonal down the middle” usually means 7.
Each connection has a weight and each neuron has a bias, and all of these learnable numbers together are the network’s parameters14. Our toy network has a couple of thousand. The language models you talk to have hundreds of billions.
It is all multiplication
Writing out every neuron one at a time gets tedious, and computers agree. Put a whole layer’s weights in a grid, a matrix11, with one row per neuron and one column per input. Running the layer is then a single operation: multiply the matrix by the input vector. Each output number is one row of weights multiplied against the inputs and summed, which is exactly what each neuron was doing anyway.
This is why people say neural networks are “just matrix multiplication”. It is not quite fair, because the activation functions in between matter enormously, but it is where almost all of the work goes. Answering one question with a large language model means multiplying matrices with billions of entries, over and over, for every word it writes.
Why graphics cards?
A normal processor is built to do one complicated thing after another very quickly. A matrix multiplication is the opposite: millions of simple, independent multiply-and-add steps that could all happen at once. That happens to be exactly what drawing a 3D scene needs too, where every pixel on the screen is worked out separately, so the chip designed for it, the GPU5, turned out to be the perfect machine for neural networks.
Modern accelerators go further, with dedicated circuits that do nothing but multiply small blocks of matrices, and towers of memory stacked right next to the processor so the weights can be fed in fast enough. Often the limit is not how fast the chip can multiply but how fast it can fetch the numbers to multiply, a theme we will come back to in the hardware chapters.
Learning
A fresh network starts with random weights, so its answers are random too. To improve it we need two things: examples where we already know the right answer, the training data18, and a way to measure how wrong the network is.
That measure is called the loss10. Show the network a batch of digits, compare its outputs with the correct labels, and boil all the mistakes down to a single number. A perfect network would score zero. Training is nothing more than the search for the set of weights that makes the loss as small as possible.
Here is a useful way to picture it. Imagine every possible setting of the weights as a location on a landscape, and the loss as the height of the ground there. Random weights drop you somewhere high on a hillside. Good weights are down in a valley.
The landscape for a real network has millions or billions of dimensions, one per weight, so nobody can look at it. But you do not need to see the whole thing to walk downhill. You only need to feel which way the ground slopes where you are standing. That slope is the gradient6, and repeatedly taking a small step against it is called gradient descent7.
The size of each step is the learning rate9, and choosing it is a craft. Take tiny steps and training takes forever. Take huge ones and you leap straight over the valley and land on the opposite slope, sometimes higher than you started.
Backpropagation
Feeling the slope means answering one question for every single weight: if I nudged this weight up a tiny bit, would the loss go up or down, and by how much? Trying each weight one at a time would take forever. Backpropagation answers the question for all of them in one sweep.
It works backwards. Start at the output, where we know exactly how wrong each answer was. Each output neuron’s error can be shared out among the connections that fed into it, in proportion to how much each one contributed. Those connections came from neurons in the layer before, so now we know how much each of those neurons was to blame, and we can repeat the process one layer further back, and so on until we reach the input.
Mathematically it is the chain rule from school calculus applied very carefully, and the remarkable thing is the cost: working out the gradient for every weight takes roughly as long as running the network forwards once more. That efficiency is the whole reason training networks with billions of weights is possible at all.
The training loop
Put the pieces together and training becomes a loop. Run a batch of examples forward and get answers. Measure the loss. Run backpropagation to get the gradient. Nudge every weight a small step downhill. Then fetch the next batch and do it again.
One pass through the whole training set is called an epoch4, and a digit reader might train for a few dozen. A large language model reads a large slice of the public internet, usually only once or twice, which takes thousands of accelerators several months. Either way, nobody writes down what the network should know. It is all squeezed into the weights, one tiny nudge at a time.
Memorising is not learning
There is a trap here. A network with enough weights can drive its training loss to almost zero without learning anything useful, by simply memorising every example, smudges and mistakes included. Ask it about a digit it has not seen and it falls apart. This is called overfitting13.
The defence is to keep score honestly. Before training starts you set aside a slice of examples, the validation set19, that the network never trains on. As long as the loss on those unseen examples keeps falling, the network is learning something general. When it starts rising while the training loss keeps falling, it has started memorising, and it is time to stop.
What a trained network knows
When training ends, the weights are frozen, and that pile of numbers is the model. It is the only thing that gets shipped. Everything it learned, from the shape of a 7 to the grammar of a language, lives there, and none of it is written down anywhere in a form you could read.
That has an important consequence. A network cannot learn anything new just by being used. When you tell a chatbot your name, its weights do not change. If it seems to remember you tomorrow, something else, built around the network, wrote your name down and handed it back. How those systems work, and how they decide what to keep, is what the rest of this manual is about.
First, though, we need to deal with words. Pixels became numbers easily. Turning “bank” into numbers that know whether you meant a river or a building takes a more interesting idea, and it is the subject of the next chapter.
Glossary
1Activation function — The small function a neuron applies to its weighted sum before passing it on. It bends the straight line of the sum, which is what lets stacked layers learn curved, complicated patterns.
2Backpropagation — The method for working out how much each weight contributed to an error, by passing the error backwards through the network one layer at a time using the chain rule of calculus.
3Bias — An extra number added to a neuron’s weighted sum. It shifts the point at which the neuron switches on, independently of its inputs.
4Epoch — One full pass of training over every example in the training data. Networks are usually trained for many epochs.
5GPU — Graphics processing unit. A chip built to do thousands of simple calculations at the same time, originally for drawing pixels and now the workhorse of neural networks.
6Gradient — For every weight, the direction and steepness of the slope of the loss. Following the gradient downhill is how a network improves.
7Gradient descent — Repeatedly nudging every weight a small step in the direction that lowers the loss, like walking downhill in fog by feeling the slope under your feet.
9Learning rate — How big each step of gradient descent is. Too small and training crawls; too large and it overshoots the bottom of the valley.
10Loss — A single number measuring how wrong the network’s answers are on a batch of examples. Training is the search for weights that make it small.
11Matrix — A rectangular grid of numbers. A whole layer’s weights fit in one matrix, so running a layer is one matrix multiplication.
12Neuron — In a neural network, a tiny function: multiply each input by a weight, add them up with a bias, and pass the result through an activation function.
13Overfitting — When a network memorises its training examples, noise included, instead of learning the pattern behind them, so it does worse on anything new.
14Parameter — Any number the network learns during training: its weights and biases. Model sizes are quoted in parameters.
15Perceptron — Frank Rosenblatt’s 1958 learning machine: a single layer of artificial neurons whose weights adjust themselves from examples.
16ReLU — Rectified linear unit. The activation function that outputs zero for negative inputs and the input itself otherwise. Simple, fast, and the default in most networks.
17Sigmoid — An S-shaped activation function that squashes any number into the range 0 to 1.
18Training data — The labelled examples a network learns from, such as thousands of images of handwritten digits paired with the digit each one shows.
19Validation set — Examples held back from training and used only to check how well the network does on data it has never seen.
20Vector — An ordered list of numbers. Images, words and sounds all have to become vectors before a network can work with them.
21Weight — A number on a connection between two neurons that says how strongly the first one’s output should count towards the second. Learning is changing weights.
Get the next chapter by email.
One email per new chapter. Nothing else.