Machine learning from 0 to 1000

Most machine learning is one idea repeated at larger and larger scale: write down a function with adjustable numbers in it, measure how wrong it is, and nudge the numbers so it is less wrong.

This article starts at that idea and goes all the way up, through classical models, neural networks, transformers, diffusion, reinforcement learning and running models in production. It follows the approach that 3Blue1Brown made famous: start with a picture, get the intuition, and only then write the formula, so the formula reads as a description of something already understood.

Every chart here was computed from a real run, and every code block was executed. Sections are numbered as levels from 0 to 1000, and each one depends on the ones before it.

Level 0: what machine learning is

A normal program is a set of rules written by a person:

input + rules -> output

Machine learning turns that around:

inputs + outputs -> rules

Nobody can write down the rules for recognising a cat in a photo, or for what a good reply to a sentence looks like. People can, however, collect a million examples. A learning algorithm searches for a function that reproduces those examples and also works on new ones.

That last clause carries all the difficulty. Memorising examples is trivial. A lookup table does it perfectly and learns nothing. The goal is generalisation: doing well on data the model has never seen.

Every machine learning system has four parts, and it is worth naming them, because the rest of this article only changes one or more of them:

The types of learning

The types differ in what feedback the model gets.

Inside supervised learning there are two main shapes of task. Regression predicts a number. Classification predicts a category. Generative models, which produce new images and text, cut across the table: they are usually trained with self-supervised methods.

Level 50: the maths, as pictures

Four pieces of maths carry nearly everything. None needs to be deep to begin with, but each needs to be seen correctly.

Vectors are points, and data is vectors

A vector is a list of numbers, and it is also an arrow or a point in space. A house with 3 bedrooms and 120 square metres is the point $(3, 120)$. A 28 by 28 greyscale image is a point in 784-dimensional space. A sentence, after embedding, is a point in a few thousand dimensions.

This is the first big shift in view. A dataset is a cloud of points, and a classifier is a surface that cuts the cloud. High dimensions cannot be drawn, but the two-dimensional pictures in this article carry over surprisingly well.

The dot product measures agreement

It is large when two vectors point the same way, zero when they are perpendicular, and negative when they oppose. That one operation is how a linear model weighs features, how a neuron decides to fire, how attention decides what to look at, and how a search engine compares embeddings.

A matrix is a transformation of space

A matrix is usually introduced as a grid of numbers. A better description is that it moves space. Its columns say where the basis vectors land, and everything else follows along, with grid lines staying straight, parallel and evenly spaced.

Multiplying matrices means doing one movement after another. A neural network layer is one such movement followed by a bend, and a deep network is many in a row.

Two more words follow from this picture. An eigenvector is a direction that a transformation only stretches, without turning. The determinant is how much it scales area. Both come back in PCA and in the stability of training.

A derivative is a sensitivity

The derivative answers one question: if this input moves a tiny amount, how much does the output move? For a function of many inputs, the list of all those sensitivities is the gradient, written $\nabla f$. It is a vector that points in the direction of steepest increase, so its negative points straight downhill.

The chain rule then says that sensitivities multiply along a pipeline. If $y$ depends on $u$, and $u$ on $x$:

If turning one dial moves a second dial by 3, and the second moves the output by 2, then the first moves the output by 6. Backpropagation, the algorithm behind all of deep learning, is that sentence applied a billion times.

Probability is bookkeeping for uncertainty

A model rarely knows the answer, so it outputs a distribution. A few facts recur constantly.

Bayes’ theorem says how to update a belief with evidence:

Expectation is a weighted average over outcomes, $\mathbb{E}[X] = \sum_x x\,P(x)$. Variance is the expected squared distance from that average.

Entropy measures surprise. A rare event carries more information than a common one, and information is defined as $-\log P(x)$, so that independent events add. Entropy is the average surprise of a distribution:

Cross-entropy is the average surprise when reality follows $p$ but the model believes $q$:

The gap between them, $H(p,q) - H(p)$, is the KL divergence. It is zero only when the model’s belief matches reality. Training a classifier means shrinking exactly this gap.

Level 100: the first model

The simplest possible case is a line through some points.

Here $w$ and $b$ are the two adjustable numbers, called parameters. Different values give different lines, so the problem is choosing a good pair. That needs a definition of good.

The loss

For each point, the error is the vertical gap between prediction and truth. Square each gap and average them, and the result is the mean squared error:

Squaring has real reasons behind it. It makes the loss smooth, it punishes big mistakes much more than small ones, and it follows directly from assuming the noise is Gaussian, as covered later under maximum likelihood.

The loss is a landscape

Now change the view. Stop looking at data space and look at parameter space. Every pair $(w, b)$ is a point on a flat plane, and the loss at that point is a height. The result is a landscape, and for linear regression it is a smooth bowl.

Training is finding the bottom of that bowl. Linear regression actually has an exact answer, the normal equation, which comes from setting the gradient to zero:

This has a clean geometric meaning. The predictions $X\mathbf{w}$ can only reach a flat subspace, and $\mathbf{y}$ usually sits outside it, so the best answer is the shadow of $\mathbf{y}$ on that subspace. Least squares is an orthogonal projection.

Most models have no such formula, which is why the next idea matters so much.

Gradient descent

If you stand on a hillside in fog, you can still feel which way is downhill. Take a small step that way and repeat. That is the whole algorithm:

The symbol $\theta$ stands for all the parameters, and $\eta$ is the learning rate, the size of each step. Here it is on 200 points generated from $y = 3x + 7$ plus noise:

import numpy as np

rng = np.random.default_rng(0)
x = rng.uniform(0, 10, 200)
y = 3 * x + 7 + rng.normal(0, 2, 200)

w, b = 0.0, 0.0
for step in range(5001):
    error = (w * x + b) - y
    w -= 0.01 * 2 * np.mean(error * x)
    b -= 0.01 * 2 * np.mean(error)
    if step in (0, 100, 1000, 5000):
        print(step, round(np.mean(error ** 2), 3), round(w, 3), round(b, 3))

0 612.629 3.019 0.46
100 8.167 3.53 3.012
1000 4.169 2.958 7.033
5000 4.169 2.949 7.094

It recovers roughly 3 and 7 from data alone. The loss stops at about 4.17 and goes no lower, because the noise has variance 4. That is the part no model can ever predict, called the irreducible error.

The learning rate

This one number decides whether training works at all.

Classification: logistic regression

Now predict a yes or no. A line’s output can be anything from minus to plus infinity, but a probability has to live between 0 and 1. Pass the score through the sigmoid:

Then pick a loss. The right one is binary cross-entropy:

Geometrically, logistic regression is a flat cut through space. The set of points where the probability equals 0.5 is a straight line in two dimensions and a flat hyperplane in more.

For many classes, each class gets a score, and softmax turns the scores into a distribution:

The $T$ is a temperature. It is 1 by default, and it controls how sharp the distribution is. The same mechanism later controls how adventurous a language model’s output is.

Where the losses come from

Squared error and cross-entropy look like arbitrary choices, but both come from one principle, maximum likelihood: choose the parameters under which the observed data was most probable.

Assume $y$ is the model’s prediction plus Gaussian noise. Taking the log of the likelihood turns products into sums, the Gaussian’s exponent falls out, and maximising it is the same as minimising squared error. Assume $y$ is a coin flip with probability $p$ and the same steps give cross-entropy. So a loss function is a statement about how the noise in the data is believed to behave.

If a prior belief about the parameters is added as well, the result is maximum a posteriori estimation, and that prior turns out to be regularisation, covered next.

Level 200: why models fail

A model that scores 99% on its training data can be worthless. This level is about telling the difference, and it matters more in practice than any algorithm.

Overfitting

Here are 14 noisy points from a smooth curve, fitted with polynomials of increasing degree.

In this run the best validation error was about 0.069, at degree 5. At degree 13 the training error was 0.0017 and the validation error about 165,870. The model had enough freedom to memorise the noise, and away from the training points it swung off to enormous values.

This is why data is split into three sets:

The settings chosen by a person, like degree or learning rate, are called hyperparameters. Picking them on the validation set slowly fits that set too, which is why the test set stays untouched until the very end. When data is scarce, k-fold cross-validation rotates the validation role through k slices and averages the results.

Bias and variance

Test error splits into three parts:

Bias is the error from assumptions that are too rigid. Train the same model on many datasets and its average answer is still wrong. Variance is the error from sensitivity. Small changes in the data move the model a lot. $\sigma^2$ is the irreducible noise.

Degree 1 above is high bias. Degree 13 is high variance. The classical view says that the art is finding the balance between them. Level 1000 shows where that story breaks.

Regularisation

Regularisation deliberately makes a model simpler by adding a penalty on large parameters:

With $p = 2$ this is L2, or ridge. With $p = 1$ this is L1, or lasso. They behave differently, and the reason is a pretty piece of geometry.

Adding a penalty is equivalent to forcing the parameters to stay inside a shape. The solution is where the error contours first touch that shape.

On a diamond, the solution tends to land near or on a corner, and at a corner some weights are exactly zero. So L1 performs feature selection, and L2 only shrinks. In Bayesian terms, L2 is a Gaussian prior on the weights and L1 is a Laplace prior.

Measuring properly

Accuracy misleads as soon as classes are unbalanced. With 1% fraud, a model that always answers “not fraud” is 99% accurate and useless. The confusion matrix gives the real picture.

A classifier outputs a score, and a threshold turns it into a decision. Moving the threshold trades precision against recall.

AUC, the area under that curve, has a concrete meaning: the probability that a random positive scores above a random negative. It was 0.92 here. For rare positives, the precision-recall curve is usually more informative.

For regression, the common measures are MSE, RMSE, which is in the original units, MAE, which is more robust to outliers, and R², the share of variance explained.

Data problems

Most real failures live here.

Leakage is information in the training data that will not be available at prediction time. A hospital model that learns that patients with a certain scan machine are sicker has learned nothing about disease. Random splits of time series leak the future.

Imbalance is handled by reweighting the loss, resampling, or choosing the threshold deliberately.

Feature scaling matters whenever features have very different ranges. It makes the loss bowl rounder, which makes descent faster, and it is required for distance-based methods.

Encoding turns categories into numbers, using one-hot vectors or learned embeddings.

No free lunch. Averaged over all possible problems, no model beats any other. Every model works because its assumptions match the kind of structure that real data has. Those assumptions are called inductive bias, and picking a model means picking a set of assumptions.

Level 300: the classical toolkit

These methods are not obsolete. On tables of business data they often beat deep learning, and they train in seconds.

Each model below has a different idea of what a sensible boundary looks like. This figure shows four of them on one dataset, two interleaved crescents.