Building an LLM
From a pile of text to a model that answers.
A large language model is less mysterious once each part has been built by hand. This track builds a small GPT from nothing, one piece per chapter, and keeps every piece small enough to see. A sentence becomes numbers, the numbers become vectors, attention lets each word look back at the words before it, and a stack of identical blocks turns that into a prediction of the next word. Training makes the predictions better, and the same structure, filled with the weights OpenAI published for GPT-2, writes fluent text.
Every chapter runs on one Mac, and every number shown came from a real run. The examples start tiny, a few words and a few dimensions, so the arithmetic can be followed line by line, and grow to the real model only when the small version is clear. The last chapters adapt the model to a single job and then to following instructions, and look at what today’s models changed.
The code, cut down to one script per chapter, is in 45h0kg/build-an-llm on GitHub.
Other parts of the site cover the surroundings: the essays on machine learning and neural networks cover training in general, the GPU track covers what inference costs on hardware, and the AI Engineering track builds applications on top of a finished model.
Chapters
- 1From text to numbersWhy a language model needs numbers, how characters and words fall short, byte-pair encoding built by hand on a ten-word corpus, and GPT-2's real tokenizer on a whole book.Draft
- 2Tokens become vectorsToken IDs turned into learned vectors with an embedding table, positions added so order survives, and a long text cut into input and target pairs for training.Draft
- 3Attention, slowlySelf-attention built up from three words and a 3 × 3 table: dot products, softmax, weighted sums, queries, keys and values, scaling, the causal mask, dropout and multiple heads, ending with attention maps from the real GPT-2.Draft
- 4The transformer blockAttention wrapped with layer norm, a GELU feed-forward network and residual connections, stacked twelve times under an output head into a GPT-2-shaped model whose 124,439,808 parameters are counted one by one.Draft
- 5Learning to predictCross-entropy and perplexity on a four-word example, a small GPT trained on The Time Machine on one Mac until it overfits, and greedy, temperature and top-k decoding compared on real samples.Draft
- 6Borrowing GPT-2's weightsOpenAI's published GPT-2 124M weights downloaded, mapped tensor by tensor into the chapter 4 model, checked against Hugging Face's implementation, and then measured: what it writes, what its numbers look like, and how fast it runs on a Mac.Draft
- 7Teaching it one jobGPT-2's output head swapped for a two-way one and trained to tell spam text messages from real ones: the data, which layers to train, accuracy before and after, and the messages it still gets wrong.Draft
- 8Teaching it to follow instructionsGPT-2 fine-tuned on a thousand human-written instructions and answers: the prompt format, training only on the answer, before and after on held-out requests, a crude evaluation and why it's crude, LoRA as the cheaper way, and preference tuning in brief.Draft
- 9What changed since GPT-2The changes between GPT-2 and models like Llama 3 and Qwen3, each written as a diff to the chapter 4 code and measured where a Mac can measure it: RMSNorm, rotary positions, SwiGLU, the KV cache, grouped-query attention and mixture of experts.Draft