Neural networks and CNNs, explained

Your phone unlocks when it sees your face. An app can name a plant from one photo. Nobody sat down and wrote the rules for these. They were learned from examples by a program called a neural network.

This post explains how that works, from nothing. The only maths you need is multiplying and adding, and one idea about slopes, explained when it comes up. The words “neural network” sound scary, but by the end you will see that one is built from very simple parts, and that the clever bit is the way they get trained.

Everything here is real. Two small networks were trained on 1,797 handwritten digits, and all the numbers and pictures below come from those runs.

The problem: why not write the rules?

Think about how you would write a program to recognise a handwritten 3. You might try “two curves, one above the other, open on the left”. Then someone writes their 3 with a pointy top. Someone else writes it tilted. Someone’s pen slips. Your rules keep growing, and they still miss things.

Your brain does this easily, but you cannot explain how, so you cannot write it down as rules. The alternative is to build a program that has lots of adjustable knobs, show it thousands of examples, and have it turn its own knobs until it gets them right. That is machine learning.

What a computer sees

A computer has no idea what a picture is. It gets numbers.

This image is 8 pixels wide and 8 tall, so that is 64 numbers. A photo on your phone might be 12 million pixels, each with three numbers for red, green and blue.

So the task becomes a maths problem: take 64 numbers in, and give back one answer, a digit from 0 to 9.

One neuron

A neuron is the basic part. Despite the name, it is a tiny calculator, and it is only loosely inspired by brain cells. It does three things:

1. It takes in some numbers.
2. It multiplies each by a weight and adds them up, then adds one more number called the bias.
3. It passes the result through a small function that decides how strongly to fire.

Here is an example. Say you are deciding whether to go to a party, based on two inputs:

Multiply and add: $(3 \times 2) + (2 \times -3) = 0$. With a bias of 1, the total is 1. A positive total means yes, so you go.

The weights are the neuron’s opinions. A big positive weight means “this input really matters, in favour”. A negative weight means “this counts against”. If you hate homework more, that weight gets more negative.

In maths, with inputs $x$, weights $w$ and bias $b$:

Here is the key point of the whole post: a neural network learns by changing its weights and biases. There is nothing else in there. Those numbers are the knobs.

Why the neuron needs a bend

The third step uses an activation function. The most common one is ReLU, and it could not be simpler: if the number is negative, output 0, otherwise keep it.

It may look pointless, but it is the secret ingredient. Without it, however many layers of neurons you stacked, the entire thing would only be able to draw straight lines. Multiplying and adding always gives something straight. The little bend at zero lets networks build curves, in the same way you can draw a circle from lots of tiny straight lines.

A network is neurons in layers

One neuron can only make a simple decision, so we stack them.

64 inputs  ->  64 hidden neurons  ->  10 outputs
(pixels)       (the working out)      (one per digit)

Every input connects to every hidden neuron, and every hidden neuron connects to every output. Each connection has its own weight. The layers in the middle are called hidden because you never see them directly.

The output layer has 10 neurons, one for each digit. Their scores get turned into probabilities that add up to 100%, by a function called softmax. An output might look like this: 3 at 91%, 8 at 6%, 5 at 2%, and small amounts for the rest. The network’s answer is whichever is highest.

Our plain network has 4,810 knobs in total. Before training, every one of them is random, so the network is guessing.

What the hidden layers are doing

Nobody tells the hidden neurons what to look for. They work it out. A useful way to picture this is that each layer rearranges the data to make the job easier for the next layer. The final layer only has to draw simple cuts, and all the layers before it exist to get the data into a shape where simple cuts work.

Training: how the knobs get turned

This is where the magic is, and it comes down to three steps repeated many times.

Step 1: guess

Feed in a digit, and let numbers flow forward through the network to get an answer. This is called the forward pass.

Step 2: measure how wrong it was

We need one number that says how bad the guess was. This is the loss. For this kind of problem, the loss is based on the probability the network gave to the right answer:

You do not need to love logarithms to see how it behaves:

Being right and confident costs nothing. Being wrong and confident costs a lot. Training means making this number as small as possible.

Step 3: adjust

Picture yourself standing on a hilly landscape in thick fog, and you want to reach the lowest point. You cannot see far, but you can feel the slope under your feet. So you take a step downhill, feel the slope again, and repeat.

That is exactly how training works, and it is called gradient descent. The landscape is the loss. Your position is the current setting of all the knobs. There are 4,810 knobs rather than two, so it is a landscape in 4,810 dimensions, which no one can picture, but the maths is the same.

The slope tells you which way to go. For each knob, we want to know: if I turn this a tiny bit, does the loss go up or down, and by how much? That number is called the gradient.

Then update every knob a little in the downhill direction:

The learning rate is your step size. Too small, and training takes forever. Too big, and you leap straight over the valley and get worse.

Backpropagation: finding all the slopes

There are thousands of knobs. Testing each one separately would be extremely slow. Backpropagation gets all of the slopes in one backward sweep through the network, using a rule from maths called the chain rule.

The idea is about passing on blame. Think of gears. If gear A turns gear B three times as fast, and gear B turns gear C twice as fast, then A turns C six times as fast. You multiply along the chain. In a network, every weight influences the loss through a chain of steps, so its effect is a product of how much each step affects the next.

Here is a full worked example with one neuron, one input and a sigmoid output, which squashes any score to between 0 and 1.

Say the input is $x = 2$, the weight is $w = 0.5$, and the right answer is 1.

Forward:

Backward. The chain rule gives a slope of $-(1 - 0.731) \times 2 = -0.538$.

The slope is negative, which means that raising the weight lowers the loss. With a learning rate of 0.1, the new weight is $0.5 + 0.054 = 0.554$. The score rises a little, the probability rises, and the loss falls. One tiny step in the right direction. Backpropagation does this for every weight, all at once.

Real training does not use one example at a time, or all of them at once. It uses small random groups called batches. Ours used 32 digits per step. One full pass through the training data is called an epoch.

Watching it learn

Our plain network trained on 1,400 digits, and 397 others were kept aside as a test. Keeping them separate matters. If you test someone using the exact questions they revised, you learn nothing about whether they understood. A network that just memorises is said to be overfitting.

The plain network got to about 89% after one pass, and finished at 99.0% on this run. Averaged over three runs it was 98.9%. That is remarkable for a program that started out guessing, and that nobody told what a digit is.

So, problem solved? Not quite.

The problem with plain networks

Plain networks have two big problems with images.

Too many knobs. In a plain network, every pixel connects to every neuron. Our image has 64 pixels, so that is fine. A 12 megapixel colour photo has 36 million input numbers. With just 1,000 hidden neurons, that is 36 billion weights in one layer. Training that would need gigantic amounts of memory and data.

No sense of space. A plain network treats pixels as a flat list. It has no idea that two pixels are next to each other. If you shuffled all the pixels the same way in every image, it would learn just as well. That is a sign that it is ignoring what makes pictures pictures. Worse, a pattern it learned in one corner means nothing to it in another. It has to relearn “a curve” separately for every location.

You do not see like that. When you spot your friend’s face, it does not matter where in your view it is.

Convolutional neural networks

A convolutional neural network, or CNN, builds in two facts about pictures:

1. Nearby pixels matter to each other, and far-apart ones mostly do not.
2. A pattern is the same pattern wherever it appears.

The filter

Instead of looking at the whole picture at once, a CNN uses a tiny window called a filter or kernel, often 3 by 3 pixels. It is a little grid of weights.

Slide this window across the image, one position at a time. At each position, multiply the 9 pixel values by the 9 weights and add them up. Write that one number into a new image. That operation is called a convolution, and the output is called a feature map.

It is like moving a small stencil over a page and, at each spot, writing down how well the stencil matches.

Here is a filter that detects vertical edges:

-1  0  1
-1  0  1
-1  0  1

Think about what this does. On a flat area, the left and right columns cancel out and you get about 0. Where the image goes from dark on the left to bright on the right, the right column adds a lot and the left takes away little, so you get a big number. The filter lights up on edges.

Flip the idea round by 90 degrees, and you have a horizontal edge detector. Here are both on a real handwritten 7:

Those two filters were written by hand. The huge step in CNNs is that the numbers in the filters are the knobs. Nobody writes them. Training finds them, using the same gradient descent as before.

Why this saves so many knobs

One 3 by 3 filter has 9 weights, and it gets reused at every position. That is called weight sharing. It is why a filter that learns to find a curve can find it anywhere in the image.

A big image costs no more weights than a small one, because the same little filter just slides further. This is the same reason you do not need to relearn what a wheel looks like for each position on a car.

Pooling

After the filters, a pooling step shrinks the feature maps. The common version, max pooling, splits the map into 2 by 2 squares and keeps only the biggest number in each one.

The thinking is that it matters that an edge exists around here, and less exactly which pixel it is on. Pooling keeps the strong signal, makes everything smaller and faster, and makes the network slightly less fussy about exact positions.

Other words you will meet

Channels. A colour image has three layers, for red, green and blue. Feature maps have many more, one per filter.

Stride. How far the filter moves each time. A stride of 2 skips every other position and shrinks the output.

Padding. Adding a border of zeros so that filters can sit on the edges of the image.

Receptive field. How much of the original image one value can see. Each extra layer sees a bit more. It is how a stack of tiny 3 by 3 windows ends up seeing a whole face.

Building things up, layer by layer

Here is the beautiful bit. A CNN stacks these layers, and each layer’s filters look at the previous layer’s output, so they find patterns made of patterns.

Researchers have shown this by working out which images activate a given filter most strongly. It is how a picture of a cat turns into the word “cat”: pixels become edges, edges become shapes, shapes become ears and whiskers, and those become a cat.

After the convolution layers, the feature maps are flattened and passed to a normal small network that makes the final decision.

The CNN against the plain network

Our small CNN has 8 filters of 3 by 3, one max pooling step, then one output layer. It has only 810 knobs, which is six times fewer than the plain network’s 4,810. It trained on the same digits.

Averaged over three runs each, on the 397 test digits:

The plain network was a little more accurate here, with six times as many knobs. On images this tiny, there is not much space for a CNN’s advantages to show. On a real photo, that gap in knobs would be in the billions, and plain networks are not even a serious option.

The second test is more interesting. What happens if every test digit is moved over a little, without any retraining?

Both networks fall a lot, and that is an honest result. A shift of one pixel took the plain network from 98.9% to 46.5%, and the CNN from 98.2% to 61.5%. The CNN held up better, because its filters work anywhere. It was not immune, though, since its final layer still cares about position. Two pixels is a quarter of this tiny image, and that defeated both, at 13.4% and 16.4%, only a little above the 10% you would get by guessing.

So a CNN is not magic. Real systems handle this with data augmentation, which means training on shifted, rotated and flipped copies of each image, so the network sees that a moved digit is still the same digit.

The CNN got 9 of the 397 test digits wrong on this run. Here are some of them:

What makes modern CNNs work

Our network has one layer of filters. Real ones have dozens or hundreds, and a few discoveries made that possible.

Lots of data and fast chips. In 2012, a CNN called AlexNet won a major image recognition contest by a large margin. It had been trained on over a million images using graphics cards, which are very good at the multiply-and-add work that networks are made of. That result started the modern boom in AI.

Residual connections. Very deep networks got harder to train, because the slope signal fades as it passes back through many layers, like a message in a long game of whispers. ResNet, in 2015, added shortcuts that let the signal skip layers, which made 100-layer networks work.

Dropout. While training, randomly switch off some neurons. It stops the network leaning too hard on any one of them, a bit like a sports team training with random players missing.

Batch normalisation. Keeps the numbers flowing through the network in a sensible range.

Transfer learning. Take a network trained on millions of images, keep its early layers, which already know edges and textures, and retrain only the last part for your own task. You can get good results with a few hundred images.

Where CNNs are used

Face unlock, the night mode on phone cameras, medical scans, reading postcodes, cars that spot lanes and pedestrians, and apps that identify plants all use convolutional networks or close relatives. Text and sound can use them too, by treating a sound as a picture of its frequencies.

Newer models called Vision Transformers split an image into patches and compare every patch with every other one. They do very well with huge amounts of data, and they use the same training ideas as everything here.

Where networks go wrong

It is important to know the limits.

They learn what you show them, including mistakes. A network shown only daytime photos may fail at night. If its data is biased, so are its answers.

They can pick up the wrong clue. If every photo of a boat has water in it, a network may learn “water” and not “boat”.

They can be fooled. Changing an image in tiny ways you cannot see can flip the answer completely. These are called adversarial examples.

They do not understand. A network finds patterns in numbers. It has no idea what a 3 is, and it can be very confident and very wrong.

The whole thing in one go

1. An image is a grid of numbers.
2. A neuron multiplies numbers by weights, adds them up, and applies a small bend.
3. A network is layers of neurons, and training means finding good weights.
4. Each step guesses, measures the loss, and uses backpropagation to find every slope.
5. Gradient descent moves every weight a little downhill, thousands of times.
6. A CNN uses small sliding filters, so that it needs far fewer knobs and can spot a pattern anywhere.
7. Stacked filters build up from edges to parts to whole objects.

Each piece is simple enough to do by hand. What makes it powerful is the scale: millions of tiny multiplications, repeated over millions of examples, until the knobs settle somewhere useful.

If you want to go further, Machine learning from 0 to 1000 on this site continues from here, up to language models and beyond.