8 — Teaching it to follow instructions

Chapter 7 made GPT-2 useful for exactly one thing. With its output head replaced by a two-way one, it tells spam from real text messages at about 97% accuracy, and it can no longer write a word. Every new job would need a new head, a new labeled dataset and a new training run.

The more useful thing is a model that keeps writing text and does whatever a plain-language request asks: answer this, list that, sort these into groups. That kind of model is trained on a different kind of example. Not a message and a label, but an instruction and the response a person would want. This is instruction tuning, also called supervised fine-tuning. It keeps GPT-2’s own output head, the one that scores all 50,257 tokens, and changes only what text the model has learned to expect after a request.

Everything ran on an Apple M3 Pro with 18 GB of memory, on its GPU through PyTorch’s MPS backend, with Python 3.12, PyTorch 2.15.0 and tiktoken 0.14.0. The model is chapter 6’s GPT-2 124M, loaded with load_gpt2. Seeds are 123 throughout.

What GPT-2 does with a request

The request format introduced below puts the instruction under a heading and ends with a ### Response: heading for the model to fill. Given that, greedily decoded for up to 100 tokens, GPT-2 does this:

### Instruction:
How many Olympic rings are there?

### Response:

### Response:

### Response:

### Response:
...

It has seen pages with headings, so it writes more headings. Asked for the capital of Germany it writes “The capital of Germany is the capital of Germany.” over and over. Asked how to fix a computer that won’t start: “I’m sorry, but I can’t find the answer.” followed by another ### Response: heading and the same sentence again. Across 100 held-out requests, it never once produced the end-of-text token to stop on its own, 46 of its outputs contain a new ### heading, and 90 of them are dominated by repetition, meaning fewer than 40% of their words are distinct. It knows a great deal; it has no idea it’s being asked anything.

The data

The examples come from databricks-dolly-15k, 15,011 instruction and response pairs written in 2023 by Databricks employees, released under the Creative Commons Attribution-ShareAlike 3.0 license, which allows any use with attribution and requires derived datasets to carry the same license. Contributors were told not to use generative AI and not to copy from the web, except for passages from Wikipedia in the categories that need a reference text. Each record has four fields: instruction, an optional context passage, response, and a category such as open question answering, closed question answering (answer from the given passage), classification, brainstorming, summarization, information extraction or creative writing.

Two things keep the run small. Records whose prompt and response together exceed 256 tokens are dropped, which keeps 11,964 of the 15,011 and keeps batches short. From those, a seeded shuffle takes 1,000 for training, 50 for validation and 100 for the test. In the training set, open questions are the largest category (287), then classification (184), general questions (156) and brainstorming (139); 199 of the 1,000 come with a context passage. No test instruction appears in the training set.

A thousand examples is tiny next to what production models are tuned on, but it’s enough to see what the format teaches, and it trains in a few minutes here.

One format for every request

A language model only ever continues text. To make “continue this text” mean “answer this request”, every training example is written in one fixed layout, so the model learns that after a certain marker comes an answer, and after the answer comes the end. The layout used here has two or three labeled sections:

### Instruction:
Name a primary color.

### Response:
Blue.<|endoftext|>

When the request comes with material to work on, a paragraph to summarize or a passage to answer from, it goes in an ### Input: section between the two. The exact words don’t matter much. What matters is that training and use share one layout: at inference time the model receives everything up to and including ### Response: and a newline, and generates from there.

The <|endoftext|> at the end is token 50,256, the same token GPT-2 saw between documents during its own training (chapter 1). Putting it after every response teaches the model to emit it when an answer is complete, and the generation loop stops when it appears. Without it, nothing tells the model where an answer ends, and it keeps going.

def prompt_of(r):
    p = f"### Instruction:\n{r['instruction'].strip()}\n\n"
    if r["context"].strip():
        p += f"### Input:\n{r['context'].strip()}\n\n"
    return p + "### Response:\n"

Training only on the answer

Training is the same next-token prediction as in chapter 5: inputs are the token IDs, targets are the same IDs shifted left by one, and the loss is cross-entropy at every position. The one change is which positions count.

The tiny example above is 17 tokens of input. Written out as (input token → target token) pairs:

'###'          -> -100
' Instruction' -> -100
':'            -> -100
'\n'           -> -100
'Name'         -> -100
' a'           -> -100
' primary'     -> -100
' color'       -> -100
'.'            -> -100
'\n'           -> -100
'\n'           -> -100
'###'          -> -100
' Response'    -> -100
':'            -> -100
'\n'           -> 'Blue'
'Blue'         -> '.'
'.'            -> '<|endoftext|>'

Fourteen of the seventeen targets are replaced by -100. PyTorch’s cross_entropy has an ignore_index argument, -100 by default, and positions whose target equals it are left out of the loss and its average. Only three positions are trained here: predict “Blue” after the response marker, ”.” after “Blue”, and the end token after ”.”.

The reasoning is that the prompt is given, not produced. Predicting the words of the user’s question teaches the model to write questions, which isn’t the job, and in a typical example the prompt is as long as the answer or longer. In the slice of data used below, 53,463 of the 101,987 training tokens are response tokens; without masking, nearly half the training signal goes to text the model will never have to write. Whether this matters in practice is measured below rather than assumed.

def encode(r, mask_prompt=True):
    """Token ids and targets. Targets are the inputs shifted left by one;
    prompt positions get -100 so the loss ignores them."""
    p = enc.encode(prompt_of(r))
    a = enc.encode(r["response"].strip()) + [EOT]
    ids = p + a
    x, y = ids[:-1], ids[1:]
    if mask_prompt:
        y = [-100] * (len(p) - 1) + y[len(p) - 1:]
    return x, y

Examples have different lengths, so a batch is padded to its longest member. Padding inputs use the end token; padding targets use -100, so padding never adds to the loss either. Because padding goes on the right and attention is causal, the real tokens never attend to it.

def collate(examples, mask_prompt=True):
    pairs = [encode(r, mask_prompt) for r in examples]
    T = max(len(x) for x, _ in pairs)
    X = torch.full((len(pairs), T), EOT)
    Y = torch.full((len(pairs), T), -100)
    for i, (x, y) in enumerate(pairs):
        X[i, :len(x)] = torch.tensor(x)
        Y[i, :len(y)] = torch.tensor(y)
    return X, Y

torch.full(shape, value) makes a tensor filled with one value; the loop then copies each example into the start of its row.

Training on this Mac

The loop is chapter 5’s, with the batches from collate and all 124,439,808 weights trainable:

model = load_gpt2("data/gpt2/model.safetensors", device="mps")
opt = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)
gen = torch.Generator().manual_seed(123)
for epoch in range(2):
    model.train()
    order = torch.randperm(len(train), generator=gen).tolist()
    for i in range(0, len(order), 4):
        X, Y = collate([train[j] for j in order[i:i + 4]])
        X, Y = X.to("mps"), Y.to("mps")
        loss = F.cross_entropy(model(X).flatten(0, 1), Y.flatten(), ignore_index=-100)
        opt.zero_grad(); loss.backward(); opt.step()

model(X) returns logits of shape (batch, positions, 50,257). flatten(0, 1) merges the first two dimensions, so every position of every example becomes one row, and Y.flatten() lines the targets up with them; cross_entropy then averages over the rows whose target isn’t -100.

Batches are 4 examples. A first attempt at 8 ran out of GPU memory partway through: the logits alone for 8 sequences of up to 255 positions are 8 × 255 × 50,257 numbers, 410 MB in 32-bit floats, and the backward pass needs several tensors that size at once, on a machine shared with other work. Two epochs at batch size 4 are 500 steps. They took 173 seconds. The MPS driver had 10.7 GB allocated at its peak, a figure that includes memory PyTorch keeps cached for reuse, not only tensors in use.

Progress is measured as the average cross-entropy over response tokens only, on examples the model never trains on:

A loss of 3.114 means the model gives the correct next token of a reference answer, on average, a probability of e^−3.114, about 1 in 22.5; after tuning it’s about 1 in 15.6. The validation loss is lower after one epoch than after two. The second pass over the same thousand examples made the model better at those examples and slightly worse at new ones: overfitting, as in chapter 5, after only 500 steps. The model tested below is the one after epoch 2, the run as planned; stopping after the first epoch would have been the better choice, and the numbers say so.

Before and after

Here are responses to held-out instructions, greedy decoding, stopping at the end-of-text token or after 100 tokens. Long outputs are cut where marked.

### Instruction:
Which of these sentences is correct: "If I was you, I would do it" or "If I were you, I would do it"?

Before:  I would do it.

         ### Response:

         I would do it. ... [repeats to the 100-token limit]
After:   If I were you, I would do it

### Instruction:
Identify which animal species is alive or extinct: Platybelodon, Lion

Before:  Identify which animal species is alive or extinct: Platybelodon, Lion

         ### Response:
         ... [repeats]
After:   Lion is alive, Platybelodon is extinct.

### Instruction:
Who is István Szalay?
### Input:
[a Wikipedia paragraph about him]

After:   István Szalay (22 March 1944 – 1 September 2022) was a Hungarian mathematician and
         politician. A member of the Hungarian Socialist Party, he served in the National
         Assembly from 1998 to 2002.

### Instruction:
Can you write down some different boroughs of New York City?

After:   There are many different boroughs in New York City. Some of the boroughs are:
         - Brooklyn, Queens, and Staten Island

Those are the good ones. They answer, in the form asked for, and stop. The bad ones answer, in the form asked for, and stop:

How many Olympic rings are there?
After:   There are three Olympic rings in the world. The first is the Olympic gold ring,
         which is the most famous ring in the world. The second is ... [cut]

What is the capital of Germany?
After:   German is the capital of Germany

Which is a species of fish? Black Mamba or Black Sea Bass
After:   Black Mamba is a species of fish

Identify which instrument is string or woodwind: Violetta, Kaval
After:   Kaval is string, Violetta is woodwind.

The last one is telling. The training set has 32 questions of the form “Identify which instrument is string or …”, and the tuned model reproduces the shape of their answers perfectly. Which instrument goes in which slot is close to a coin toss, because a 124-million-parameter model trained on web text in 2019 doesn’t know what a kaval is, and a thousand examples can’t teach it. Instruction tuning taught the model how an answer looks. It didn’t add knowledge the model never had.

The outputs before tuning aren’t shown for most of these because they’re the same failure every time: headings, echoes, loops. One exception is worth noting. Before tuning, GPT-2 answered the fish question with “The Black Sea Bass is a species of fish that is found in the Gulf of Mexico”, correct, and then repeated its next sentence until it ran out of tokens. After tuning it answers cleanly and wrongly. A single question proves nothing, but it’s a reminder that tuning for format and being right are separate things.

Measuring it, crudely

Reading outputs is the most informative check and the least scalable. A few numbers can be computed automatically over all 100 test instructions:

The first three rows measure format, and on format the change is large and real. The tuned model stops, stays out of the template’s headings, and loops far less.

The F1 rows try to measure content. Word-overlap F1 lowercases both texts, splits them into words, and counts the words they share: precision is the shared count over the output’s length, recall is the shared count over the reference’s length, and F1 combines the two:

$F_1 = \frac{2 \cdot \text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$

For “German is the capital of Germany” against “The capital of Germany is Berlin”, the output has 6 words, the reference 6, and they share 5 (the, capital, of, germany, is), so precision and recall are both 5/6 and F1 is 0.83. The answer is wrong. “Kaval is string, Violetta is woodwind” shares every word with “Violetta is string, Kaval is woodwind” and scores a perfect 1.0. The answer is wrong. Meanwhile a correct answer phrased differently from the reference would score low.

So the 0.637 on classification items means the tuned model answers in the reference’s vocabulary, not that it’s right. Graded by hand, it gets 4 of the 20 classification items fully right (Lion and Platybelodon, Sasando and Ranat ek, Cak and Rattle, and Cisco as the fish). Several of the rest are half right, which F1 rewards and a person wouldn’t.

This is the general problem with evaluating open-ended output. Overlap metrics are cheap and reproducible and measure the wrong thing. Human raters measure the right thing and are slow, expensive and inconsistent. Using a strong language model as the judge sits in between, with biases of its own: toward longer answers, and toward answers that resemble its own. The AI Engineering track covers building evaluations for applications in more depth. For this chapter the honest summary is: the format is fixed, and the content is limited by the model underneath.

Does masking the prompt matter?

Training only on response tokens is the standard recipe, and the argument for it is sound. Whether it matters here is a separate question with an easy test: run the same training with every target kept, prompt included, and compare.

Both are scored on response tokens only, so the losses are comparable. They’re within 0.02 of each other, and the generation numbers differ by a few test items each, in mixed directions. At this scale, with one seed, masking made no difference worth reporting. That doesn’t make it pointless: with long prompts, a long document to summarize, say, the unmasked run would spend most of its training on reproducing documents. It does mean the reason to mask is that argument, not a measurement on this data.

The cheap alternative: LoRA

Full fine-tuning updates all 124 million weights, and AdamW keeps two extra numbers per weight, so the optimizer state alone is twice the model. At this size that’s fine. At 8 billion parameters it’s the difference between one GPU and several, and every fine-tuned version is another full copy of the model on disk.

LoRA, from Hu et al. (2021), starts from an observation about what fine-tuning changes. The update to a weight matrix, the difference between the weights after and before, tends to be simple compared with the matrix itself. So instead of learning the update ΔW directly, LoRA freezes W and learns ΔW as a product of two thin matrices:

$W x + \Delta W x = W x + \frac{\alpha}{r} B A x, \qquad A \in \mathbb{R}^{r \times d_{\text{in}}},\ B \in \mathbb{R}^{d_{\text{out}} \times r}$

r is the rank, small: 8 here. For GPT-2’s 768 × 768 attention output projection, a full update has 589,824 numbers. A rank-8 update has 8 × 768 for A plus 768 × 8 for B, 12,288 numbers, 2% as many. B starts at zero, so at the first step B A is zero and the model is exactly the pretrained one; training moves away from it gradually. α/r is a fixed scale, and with α = 2r it is 2.

A tiny case shows what “low rank” means. With d = 4 and r = 1, A is a row of 4 numbers and B is a column of 4, and B A is a 4 × 4 matrix built from 8 numbers, every row a multiple of the same row. It can’t represent every 4 × 4 update, only those of that shape. The bet is that the useful part of a fine-tuning update mostly has this kind of structure.

class LoRALinear(nn.Module):
    def __init__(self, base: nn.Linear, rank=8, alpha=16):
        super().__init__()
        self.base = base
        for p in base.parameters():
            p.requires_grad = False
        self.A = nn.Parameter(torch.empty(rank, base.in_features))
        self.B = nn.Parameter(torch.zeros(base.out_features, rank))   # zero: starts as a no-op
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.scale = alpha / rank

    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

nn.Parameter marks a tensor as trainable, so the optimizer sees it. kaiming_uniform_ fills A with small random values, the scheme nn.Linear uses for its own weights; the paper uses Gaussian random values instead. The essential part, in both, is that B starts at zero. Wrapping is a matter of replacing each linear layer in the blocks with a LoRALinear around it:

def add_lora(model, rank=8, alpha=16):
    for p in model.parameters():
        p.requires_grad = False
    for block in model.blocks:
        a, f = block.attn, block.ffn
        a.qkv, a.proj = LoRALinear(a.qkv, rank, alpha), LoRALinear(a.proj, rank, alpha)
        f.fc, f.proj = LoRALinear(f.fc, rank, alpha), LoRALinear(f.proj, rank, alpha)
    return model

With rank 8 on all four linear layers of all twelve blocks (the fused query-key-value layer, the attention output projection, and the feed-forward network’s two layers), the trainable count is 1,179,648, 0.95% of the model. The same 1,000 examples, two epochs at batch size 4, with a learning rate of 0.0005, ten times the full run’s, because the new matrices start from nothing and are small:

LoRA did at least as well with less than 1% of the trainable weights. Its test loss is the lowest of the three runs, and its validation loss didn’t rise in the second epoch. Changing a million numbers gives the model less room to memorize a thousand examples than changing 124 million does, which on a dataset this small is an advantage. Its outputs look like the full run’s, and in places are identical: it also says “Kaval is string, Violetta is woodwind.”

The savings here are modest, 25% of the time and 18% of the memory, because at 124 million parameters the optimizer state isn’t the main cost. The frozen weights still sit in memory, the forward and backward passes still run through every layer, and the 50,257-wide logits are as large as before. At 8 billion parameters the balance changes. For Llama 3 8B’s shape, the full run’s AdamW state alone would be 64 GB in 32-bit floats. Rank-8 LoRA on each block’s seven linear layers (query, key, value, output, and SwiGLU’s three, from chapter 9) comes to about 21 million trainable numbers, so about 170 MB of optimizer state, and the trained adapter is tens of megabytes to store instead of a full copy of the model. The AI Engineering track’s chapter on fine-tuning covers when that trade is worth making, QLoRA’s 4-bit variant, and a measured run on a larger model.

Beyond imitation: preference tuning

Instruction tuning teaches by imitation: here is a request, here is a good answer, produce answers like it. It can’t express that one answer is better than another, and it can only be as good as the answers someone wrote. The next stage in building an assistant uses comparisons instead. People are shown two responses to the same prompt and pick the better one, which is much easier than writing an ideal answer. In the recipe described for InstructGPT, those comparisons train a separate reward model to score responses, and the language model is then trained with reinforcement learning (PPO) to produce responses that score well, while being held near the instruction-tuned model so it doesn’t drift into nonsense the reward model happens to like. Labelers preferred the outputs of the 1.3B-parameter InstructGPT to those of the 175B GPT-3, a model a hundred times larger. Direct Preference Optimization later showed the same objective can be optimized with a simple classification-style loss on the preference pairs directly, with no separate reward model and no sampling from the model during training, which makes it much simpler to implement and run. Neither changes the model’s architecture; they change which outputs the same weights make likely.

Where this leaves things

A thousand examples and three minutes of training changed what GPT-2 does after a request. It answers instead of echoing, uses the shape the request asks for, a list, a sentence, a label, and stops. Measured over 100 held-out instructions, it stopped on its own 93 times where it had stopped zero times, and the number of outputs that were mostly repetition fell from 90 to 19. LoRA got the same result by training less than 1% of the weights.

What didn’t change is what the model knows. It still says there are three Olympic rings, that a black mamba is a fish, and that a kaval is a string instrument. Instruction tuning gives a model a way to answer; it can’t supply facts or reasoning the pretrained model lacks, and a 124-million-parameter model trained in 2019 lacks a lot.

The fix is the pretrained model, and current open models are better at it for two reasons: they are much larger and trained on far more text, and their blocks are built somewhat differently from chapter 4’s. Chapter 9, the last, takes the second part apart: what changed inside the transformer block between GPT-2 and models like Llama and Qwen, written as changes to this track’s own code.