14 — When prompting isn’t enough

Chapter 13 ended on a step that sits in front of every answer and doesn’t work. Before retrieval runs, the assistant could decide which part of the site a question belongs to, so the search can be filtered to that track, the metadata filter from chapter 7. Asked to make that one-word decision, the 8B model costs a generation call each time and is right 13% of the time on passages from chapters it hasn’t seen. A model small enough to make the call in a fraction of the time does even worse.

That gap is where changing the model’s weights becomes worth considering. This chapter works out when that’s true, explains the method that makes it cheap enough to try on a laptop, runs it on this Mac, and measures what changed and what didn’t.

Everything below ran on one Mac (Apple M3 Pro, 18 GB). The small model is Qwen2.5-0.5B-Instruct, 494 million parameters, in bf16 and in a 4-bit version, trained and run with Apple’s MLX through mlx-lm 0.32.2. The comparison model is llama3:latest from chapter 1, 8B parameters, Q4_0, served by Ollama with its default 4,096-token context (num_ctx).

Three ways to change what comes out

There are three places to intervene, and they change different things.

The prompt changes the input. Instructions, examples, a format to follow. It costs nothing to change and takes effect on the next call. It works when the model already has the ability and only needs to be told which one to use.

Retrieval changes what the model can see. Chapters 3 to 8 were about this. It’s the right tool when the missing thing is knowledge, especially knowledge that changes: a document edited this morning is in the index this afternoon, and the answer can cite where it came from.

Fine-tuning changes the weights, so it changes what the model does by default, with no instructions in the prompt. It’s good at behavior: a format, a tone, a fixed set of labels, a decision rule that’s easier to show by example than to write down. It’s poor at knowledge. Facts trained into weights can’t be cited, can’t be updated without training again, and come back unreliably, which this chapter measures directly below.

A rough way to choose:

The track-routing step falls in the fourth row. The labels follow how this site happens to be organized, which is a house convention, and the step also has to be cheap, which is the last row’s concern, except that here no model does it well to begin with.

The task and the data

The task: given a short passage, name the track it comes from, one of blog, kernel, gpu, design, dsa or essays. Each example is a section heading plus the first sentence under it, taken from the site’s markdown. The label is the folder the file lives in, so there’s no hand labeling and no doubt about the right answer. A real router would see users’ questions, not section openings; the openings stand in for them here because they come already labeled, and that’s a limitation of this experiment worth keeping in mind.

The split is by file, not by example. If sections of the same chapter appeared in both training and test, the model could match a chapter’s vocabulary instead of learning the convention, and the test score would be inflated. With whole files held out, the test set asks about chapters the model has never seen.

# ft/make_data.py (key part)
for m in re.finditer(r"^##+ (.+)\n+(.+)", text, flags=re.M):
    head, para = m.group(1).strip(), m.group(2).strip()
    if para.startswith(("|", "-", "#", "*", "!", "<", "$")): continue
    sent = re.split(r"(?<=[.!?])\s", para)[0][:240]
    if len(sent) < 40: continue
    rows.append({"text": f"{head}. {sent}", "label": track, "file": f"{track}/{name}"})

files = list(rows_by_file); random.shuffle(files)          # seed 7
test_files  = set(files[: len(files) // 4])
valid_files = set(files[len(files) // 4 : len(files) // 4 + len(files) // 10])

Three test examples, to show what the model sees:

"The template: choose, explore, unchoose. Nearly every backtracking function has the same three-step body:"   -> dsa
"Level 300: brokers and clusters. A **broker** is a Kafka server, and several form a **cluster**."            -> essays
"4. Data model, from access patterns. The data model starts from the queries, not from the entities."        -> design

The second one is the hard kind. A passage about Kafka brokers could sit in the System Design track; it’s an essay because of where it was written, not what it says. That’s the convention a prompt struggles to state.

The classes are badly unbalanced. Three tracks provide most of the examples, and gpu has only 16 training examples, all from the few GPU chapters that landed in the training split. The test set’s 26 gpu examples come mostly from one GPU chapter about packaging AI infrastructure. That imbalance matters later.

Every example goes into the chat format mlx-lm reads, with the same system prompt the baselines use:

{"messages": [
  {"role": "system", "content": "Name the section of the website this passage comes from. Answer with one word: blog (Linux, Observed: building a Linux system from source), kernel (Kernel, Traced: reading Linux kernel code paths), gpu (Inside the GPU: CUDA, GPU hardware, LLM inference), design (System Design: distributed systems and worked designs), dsa (Data Structures and Algorithms), essays (standalone essays on various topics)."},
  {"role": "user", "content": "Post-order: a stack of iterators. Recursion does more than visit vertices in order; ..."},
  {"role": "assistant", "content": "dsa"}]}

What prompting gets

Before training anything, the cheaper options get their turn on the same 276 test examples. Decoding is greedy and the answer is parsed by checking which label the output starts with; anything else counts as invalid.

The 0.5B model given only the instruction is worse than useless. It said blog 198 times out of 276, and most of the rest weren’t labels at all: backtracking, tls, dijkstra, gcc, the topic of the passage rather than its track. It understood “name the section” as “name the subject”. Twelve worked examples in the prompt fixed the format (2 invalid answers instead of 61) but not the judgment: it leaned hard on design, getting 60 of 63 System Design passages right and none of the 66 essays, for 37.0%, the same as always answering dsa.

The 8B model is better at the format and still mostly wrong. It got 11 of 15 blog passages and 17 of 66 essays, but only 3 of 102 data-structures passages and 3 of 63 System Design ones. The raw outputs of that run weren’t kept, so the reason can’t be read off directly, but the instruction describes each track by its subject, and matching subjects is what that invites. A passage about Kafka brokers is about distributed systems, so it isn’t an essay; a section about cache behavior inside a lab exercise could be kernel or blog or design. The descriptions are accurate, and they don’t contain the rule, because the rule is “whichever folder the author put it in.”

The last three rows are the retrieval answer to the same problem, and they need no training at all. Every training example is embedded once with nomic-embed-text; a new passage is embedded and labeled by a vote among its k nearest training examples:

# ft/baselines.py (key part)
tv = emb([r["text"] for r in train])              # 813 vectors, computed once
qv = emb([r["text"] for r in test])
for q in qv:
    top = np.argsort(-(tv @ q))[:k]
    preds.append(Counter(train[i]["label"] for i in top).most_common(1)[0][0])

With 15 neighbors it reaches 65.9%. It knows the convention because it looks at examples of it, which is exactly what the prompt couldn’t convey. This is the bar fine-tuning has to beat to be worth its cost: not the prompt, but a labeled set and a vector search.

LoRA: training a small correction

Full fine-tuning updates every weight. For this model that’s 494 million parameters, plus an optimizer that keeps two extra numbers per parameter (Adam’s running mean and variance), plus the gradients. For an 8B or 70B model, that’s a cluster, not a laptop. It also produces a complete copy of the model for every task.

LoRA (Hu et al., 2021) starts from an observation: the change a task needs is small in a specific sense. Take one weight matrix $W$ in the network, of size $d \times k$. Fine-tuning learns an update $\Delta W$ of the same size. LoRA assumes that update has low rank, and writes it as the product of two thin matrices:

$W' = W + \Delta W = W + \frac{\alpha}{r} B A, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k)$

$W$ stays frozen. Only $A$ and $B$ are trained. A layer’s output for input $x$ becomes

$h = W x + \frac{\alpha}{r} B (A x)$

and computing $B(Ax)$ costs two small matrix products: $x$ is squeezed down to $r$ numbers, then expanded back out.

Two details make this work in practice. $B$ starts at zero, so at step 0 the product $BA$ is zero and the model is exactly the original; training begins from the base model’s behavior, not from noise. And the factor $\alpha / r$ sets how loud the correction is, so the rank can change without retuning the learning rate. In MLX’s implementation the code is one line, with the scale given directly instead of as $\alpha / r$:

# mlx_lm/tuner/lora.py, LoRALinear.__call__
y = self.linear(x)
z = (self.dropout(x) @ self.lora_a) @ self.lora_b      # lora_b initialized to zeros
return y + (self.scale * z).astype(x.dtype)

The saving is in the count. A full update to a $d \times k$ matrix has $dk$ parameters; the low-rank version has

$r (d + k)$

For this run, the adapters go on the query and value projections of 16 of the model’s 24 layers, at rank 8. Qwen2.5-0.5B has a hidden size of 896, and because it uses grouped-query attention (2 key/value heads of 64 dimensions), its value projection outputs only 128 numbers. So:

$\text{per layer} = 8\,(896 + 896) + 8\,(896 + 128) = 14{,}336 + 8{,}192 = 22{,}528$

$16 \times 22{,}528 = 360{,}448$

mlx-lm reported the same figure at the start of training: Trainable parameters: 0.073% (0.360M/494.033M). The saved adapter is 1.4 MB. The base model it modifies is 969 MB.

Since $BA$ has the same shape as $W$, it can be added into $W$ once training is done, and the merged model runs at exactly the original speed. Kept separate, one base model can serve many tasks, each with its own small adapter swapped in per request.

Why low rank is enough is an empirical claim, not a theorem. The LoRA paper measured that on GPT-3 175B the method cut trainable parameters by 10,000 times and GPU memory by 3 times, while matching full fine-tuning on the tasks it tested. A narrow task like routing to six labels is the friendliest case: the change needed is a small redirection of behavior the model mostly has.

QLoRA: a frozen base in 4 bits

The frozen weights still take memory, and for a large model they take most of it. QLoRA (Dettmers et al., 2023) stores the frozen base in 4 bits and trains LoRA adapters in higher precision on top. Gradients flow through the quantized weights (dequantized on the fly for each product) into the adapters, which are the only thing updated. The paper reports fine-tuning a 65B model on a single 48 GB GPU this way, and adds three techniques for memory: a 4-bit data type called NormalFloat (NF4) designed for normally distributed weights, double quantization (quantizing the quantization constants too), and paged optimizers to absorb memory spikes.

MLX’s 4-bit format isn’t NF4. It’s affine quantization in groups of 64: every weight is stored as a 4-bit integer $q$, and each group carries its own scale $s$ and offset $b$:

$\hat{w} = s \cdot q + b, \qquad q \in \{0, 1, \dots, 15\}$

Loading the 4-bit model and inspecting one layer shows the layout: the 896 × 896 query weight is stored as uint32 of shape (896, 112), eight 4-bit values per word, with float16 scales and biases of shape (896, 14), one pair per 64 weights. That’s 4 bits per weight plus 32 bits per 64 weights, so 4.5 bits on average. The 4-bit model is 283 MB on disk against 969 MB in bf16. Training LoRA on top of it is QLoRA in the general sense: frozen quantized base, trainable full-precision adapters.

The run

The configuration, in full:

# ft/lora.yaml
model: mlx-community/Qwen2.5-0.5B-Instruct-bf16
train: true
data: data                  # train.jsonl, valid.jsonl, test.jsonl
fine_tune_type: lora
mask_prompt: true           # loss only on the answer, not on the system prompt and passage
num_layers: 16
batch_size: 8
iters: 300                  # about 3 passes over 813 examples
learning_rate: 1.0e-4
steps_per_eval: 100
save_every: 100
adapter_path: adapters/lora_bf16
seed: 0
lora_parameters:
  rank: 8
  scale: 20.0
  dropout: 0.0
  keys: ["self_attn.q_proj", "self_attn.v_proj"]

python -m mlx_lm lora -c lora.yaml

mask_prompt matters for a task like this. Without it, the loss would include predicting the system prompt and the passage, hundreds of tokens, and the one-word answer would be a rounding error in the objective. With it, the model is trained only on producing the label and the end-of-turn token.

The loss, as mlx-lm reported it:

  iter   train_loss
     1    val 9.898
    25    1.112
    50    0.299
    75    0.265
   100    val 0.497
   100    0.216
   150    0.133
   200    val 0.292
   200    0.148
   250    0.065
   300    val 0.361
   300    0.070

Training loss keeps falling to 0.07. Validation loss is lowest at step 200 and rises by step 300, the usual first sign of overfitting. Validation loss isn’t the number that matters, though; the label is. So each saved checkpoint was scored on the 88 validation examples, and the test set was left alone until one was picked:

Step 300 has the best accuracy despite the higher loss. Loss measures how much probability the model puts on the right label; accuracy only asks whether the right label is the most likely one. A model can grow more confident on its wrong answers (raising loss) while also getting more answers right. Step 300 it is.

The run took 285 seconds of wall-clock time for 300 steps, including three passes over the validation set, with a peak memory footprint of 4.4 GB for the training process (macOS’s figure, which counts GPU allocations, since the GPU shares the same memory). The machine was shared with other jobs, which affects both numbers, and one earlier attempt at the same run failed with a Metal out-of-memory error while several other models were loaded.

The QLoRA run used the same configuration on the 4-bit model and took 314 seconds, with a peak footprint of 3.7 GB. At this size 4-bit storage saved about 0.6 GB and no time: the bf16 weights are under 1 GB, so most of the memory goes to activations, optimizer state for the adapters, and MLX’s buffers, and dequantizing the weights on every step costs some compute. The trade looks different for a larger model. An 8B model’s weights are about 16 GB in bf16 and about 4.5 GB at 4.5 bits per weight, and on an 18 GB machine that difference decides whether training fits at all.

Before and after

On the 276 held-out examples:

Three hundred steps on 813 examples took the small model from 5.4% to 68.8%, and from 61 malformed answers to one. The 4-bit base did as well as the bf16 one (67.8%), even though the 4-bit model on its own was the worst of all: 204 of its 276 answers weren’t a label. The adapter fixed the format first, as the loss curve’s drop in the first 50 steps suggests, and then the judgment.

The sample size limits what these numbers can say. With 276 test examples, a 95% confidence interval around 68.8% is about ±5.5 points, so LoRA, QLoRA and the nearest-neighbor vote (65.9%) can’t be told apart here. What the data does support is the gap between all three and every prompt-only approach, including the 8B model.

The comparison that decides whether this was worth doing is speed. Thirty test passages were classified both ways under the timing lock, each with its model already loaded:

The compute times are close, because generating one word is mostly prefill, and the 8B model on this GPU prefills a short prompt quickly. The difference that matters is the last row. The 0.5B model runs inside the assistant’s own process, so it never waits in the model server’s queue, which on a busy server is where nearly all the time goes, as chapter 13 found. The nearest-neighbor vote needs one embedding call (about 30 ms of compute in chapter 13) and a matrix product, which makes it the cheapest option of all, if the embedding model is at hand.

What it changed and what it didn’t

The confusion matrix shows where the gain came from. Of the adapter’s 276 answers, the biggest groups:

Data structures and algorithms is now almost perfect, and System Design mostly right. The confusion that remains is between essays and design, which is the convention itself: an essay about Kafka and a design chapter about queues read alike, and nothing in a single sentence says which folder it’s in.

gpu went from 0 of 26 to 0 of 26. The adapter labeled 24 of them essays. Two causes, both visible in the data. Training had 16 gpu examples against 307 dsa, so the adapter barely learned the label exists. And most of the test gpu examples come from one GPU chapter about packaging AI infrastructure (OCI Model Packaging, Lazy-Loading OCI Container Snapshotters, Containerized RDMA, RoCEv2, and GPUDirect), which reads like an essay. Fine-tuning learns the distribution it’s shown. A class with little data stays weak no matter how good the method is, and a held-out chapter that differs from the training ones exposes it.

Then the costs that don’t show in accuracy. The same five prompts went to the base model and to the adapted one:

# ft/probe.py (key part)
for adapter in [None, "adapters/ck300"]:
    model, tok = load("mlx-community/Qwen2.5-0.5B-Instruct-bf16", adapter_path=adapter)
    for name, m in cases:
        p = tok.apply_chat_template(m, add_generation_prompt=True, tokenize=False)
        print(name, generate(model, tok, prompt=p, max_tokens=60))

Each row shows something different.

The first two rows are the adapter leaking into prompts it was never trained on. Three hundred steps of “answer with one short word” taught the model to answer with one short word, even without the system prompt: asked for a sentence about Paris, it gave the country’s name. The sea sentence degraded into repetition. Low-rank and narrow doesn’t mean contained. An adapter for a router should only be loaded for routing calls, which is what per-request adapters are for, and a merged model would carry this damage everywhere.

The third row is a forced choice. The recipe isn’t from any track, and the label set has no “none of these”. The base model answered with a topic word; the adapter confidently picked essays. Fine-tuning made the output always well-formed, which also means it never signals that the input is out of scope. The router needs an explicit other label with examples, or a check on the retrieval score like the drift signal from chapter 13.

The fourth row is the improvement in a single line. A clearly GPU sentence, which the base model called blog, is now gpu. The class does work when the input is unambiguous; it’s the test chapter that’s the problem.

The last row is the one this chapter promised to measure. The site’s caching chapter says leases cut peak database load from 17,000 to 1,300 queries a second. Neither model knows. The base model made up a plausible figure; the adapted one didn’t even produce a sentence. Training on 813 section openings didn’t teach facts about the site, and it wasn’t trying to. That’s what retrieval is for, and it’s why the assistant built over the previous thirteen chapters still retrieves before it answers.

Preference tuning and distillation

Supervised fine-tuning, as above, needs the right answer for each input. Two related methods cover the cases where that’s not the shape of the data.

Preference tuning uses comparisons instead: for one prompt, a response people preferred and one they didn’t. That’s often easier to collect than a perfect answer, especially for open-ended output where many answers are acceptable and some are better. The classic pipeline, RLHF as described in the InstructGPT paper (Ouyang et al., 2022), fits a reward model to the comparisons and then optimizes the language model against it with reinforcement learning. That paper reports that people preferred outputs of a 1.3B tuned model over the 175B base model. DPO (Rafailov et al., 2023) removes the separate reward model and the reinforcement learning loop, and trains directly on the pairs with a classification-style loss. For a prompt $x$ with preferred response $y_w$ and rejected response $y_l$:

$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right)$

$\pi_\theta$ is the model being trained, $\pi_{\text{ref}}$ is a frozen copy of where it started, $\sigma$ is the logistic function, and $\beta$ controls how far the model may drift from the reference. The loss rewards raising the probability of the preferred response relative to the reference more than that of the rejected one. The reference term is what stops the model from simply collapsing onto whatever pattern separates the pairs, which the adapter above did to a lesser degree with its one-word answers. For this assistant, the pairs would come from the logs of chapter 13: answers that users marked helpful against ones they didn’t, or a faithful answer against one the judge from chapter 8 flagged.

Distillation (Hinton et al., 2015) trains a small model to imitate a large one. The original form trains the student on the teacher’s full probability distribution, softened by a temperature $T$:

$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

At $T > 1$ the distribution flattens, and the student learns not only the teacher’s top answer but which wrong answers the teacher found nearly right. In practice with language models it often takes a simpler form: run the big model on many inputs, keep its outputs, and fine-tune the small model on them as if they were labels.

That’s the realistic version of this chapter’s run. The labels here came free from the folder structure, but the router’s real inputs are users’ questions, which have no folder. The 8B model, or a hosted one with the prices from chapter 13, can label a few thousand logged questions, a person can check a sample, and the 0.5B model learns from the result. The big model’s cost is paid once, offline, and the small model carries the skill at a fraction of the latency on every request.

Where this leaves the track

For the router, the measurements point to a plain conclusion. The fine-tuned 0.5B model, a 1.4 MB adapter, gets about 69% of held-out passages right at a median 151 ms in-process. A nearest-neighbor vote over the same labeled examples does about as well with no training. Either beats every prompt tried, including the 8B model. And the adapted model picked up a habit of one-word answers outside its task, can’t say “none of these”, and still doesn’t know a single fact about the site. The labeled examples did most of the work; fine-tuning was one way to use them, not the only one.

The track started with a model that gave a different answer each time it was asked the same question, and knew nothing about the documents it was meant to answer from. Each chapter took the failure the last one left behind. Structured output made the answers usable by a program. Embeddings, indexes and a retrieval pipeline gave the model the passages it needed, and failure analysis and measurement showed which fixes helped. Tools let it act, memory and step limits kept long tasks on track, and MCP gave the tools a standard interface. Then the documents themselves turned out to be an attack surface, real use brought latency, cost, caching and failures into view, and this chapter moved one narrow skill into the weights of a model small enough to run on every request.

A few things held throughout. The model is a component with measurable failure rates, and most of the work is in the system around it: what goes into the prompt, what comes back from retrieval, which tools exist, what checks the output, and what happens when any part is slow or wrong. Each change was worth exactly what a measurement on held-out data said it was worth, and several of the measurements here were small enough that the next run could disagree. Knowledge belongs in retrieval, where it can be cited and updated; behavior can go in the prompt or, when that runs out, in the weights. And the parts that matter most for safety, which tools exist, who approves a side effect, what an answer may contain, live in ordinary code that doesn’t depend on the model behaving.