8 — Knowing whether it helped

Chapter 7 ended with a pipeline that found the answer for 19 of the 20 questions it was tuned on, up from 16. That number can’t be trusted. Every fix was chosen because it repaired something in those twenty, so the twenty are the one set of questions guaranteed to look good. The real question is what happens on questions nobody looked at while choosing the fixes.

That needs a second set of questions, written separately, with the answers marked in advance, and a way to score the pipeline against it. This chapter builds one, runs the baseline and each fix against it, and then asks how much the resulting numbers can be believed.

The setup is the same as before: one Mac (Apple M3 Pro, 18 GB), Ollama 0.40.11, llama3:latest (8B, Q4_0) for answers and nomic-embed-text (137M, F16) for embeddings. Generation settings are as in chapter 7: temperature 0, a fixed seed, num_ctx 4,096 and num_predict set on every call (512 for answers, 20 for a judge’s one-letter verdict). Two more models appear later: qwen2.5:7b (Q4_K_M) as a second judge, and the MiniLM cross-encoder from chapter 7.

Sixty questions nobody tuned on

The set has 60 questions, 10 from each of the six parts of the site the assistant searches: the Linux build series, the kernel track, the GPU track, the system design track, the data structures and algorithms track, and the essays. They were made like this:

1. Cut every chapter into passages, and pick passages at random (with a fixed seed), ten per track.
2. Skip any passage that is mostly code, a table, markup or under 400 characters, and take the next one. The point is a question a reader would ask, and most readers don’t ask about a line of a code listing.
3. For each passage, write one question in my own words that the passage answers, avoiding the passage’s phrasing where possible.
4. Copy the exact sentence or phrase that answers it from the chapter, and write a one-line reference answer.

{"id": 16, "doc": "/kernel/12-signals-timers-rcu",
 "question": "What is the default timer slack for a task?",
 "span": "50,000 ns by default",
 "answer": "50,000 ns."}

The 60 come from 37 different chapters. Random sampling doesn’t spread evenly: six of the ten GPU questions landed in one long chapter, on packaging the AI stack. None of these questions was used in chapter 7, and none of chapter 7’s twenty is in this set.

The copied span is what makes the set reusable. A passage counts as relevant to a question if it contains the span, or any 30-character window of it, which allows for a span that a chunk boundary cut in two. Because relevance is defined by text rather than by passage number, the same labels work for any way of chunking the site: 400-character passages, 1,600-character passages, passages with headings in front. A label tied to “passage 2,417” would have to be redone every time the chunker changed.

Why not let the model write the questions

Writing sixty questions by hand is slow, and the obvious shortcut is to show the model each passage and ask it for a question. I tried that on the same 60 passages, with llama3 at temperature 0:

Here is a passage from a technical website.

{passage}

Write one question that a reader could ask and that this passage answers. Output only the question.

Reading the 60 results, 37 were questions the passage clearly answers, 10 were partly answered, and 13 weren’t answered by the passage at all. Some were too vague to have an answer (“What is the most important component in a system and why?”). Some asked about the passage’s opening question, which the passage sets up and leaves for the next one (“How does a packet get from a pod on one node to a pod on another?”, from a passage ending with that sentence). Some invented a premise (“How many passwords can an attacker’s GPU check per second?”, from a passage about how many a server can check). Over a fifth of a generated set would have measured nothing, or something wrong.

The two sets also differ in a way that affects which search looks best. The share of a question’s content words that also appear in its passage was 0.66 for my questions and 0.56 for llama3’s. Questions that reuse the passage’s words favor word matching. On my questions BM25 found the source passage in the top five for 87%, against 80% for embeddings; on llama3’s questions, BM25 found 77% and embeddings 83%. Same passages, opposite verdict. (This comparison counts a passage as relevant if it shares at least 30% of the source passage’s text, since the generated questions have no hand-copied span.) The question set is part of the measurement, and a set written by someone who has just read the passage leans toward lexical search. I kept the hand-written set and keep that bias in mind below.

What to measure

For retrieval, the two standard measures are simple.

Recall@k is the fraction of questions for which at least one relevant passage is in the top $k$. With one answer per question, as here, it’s also called the hit rate. Recall@5 matters most for this pipeline, because the model sees exactly five passages.

$\text{Recall@}k = \frac{1}{|Q|}\sum_{q \in Q} \mathbf{1}\left[\text{a relevant passage is ranked} \le k \text{ for } q\right]$

MRR (mean reciprocal rank) rewards putting the answer first. For each question, take one over the rank of the first relevant passage, or zero if there is none in the top 10, and average:

$\text{MRR@10} = \frac{1}{|Q|}\sum_{q \in Q} \frac{1}{\text{rank}_q}$

A relevant passage at rank 1 scores 1, at rank 2 scores 0.5, at rank 5 scores 0.2. Recall@5 can’t tell rank 1 from rank 5; MRR can. Both come from the same loop:

def first_relevant(item, ranked):
    for pos, i in enumerate(ranked):
        if relevant(item, chunks[i]):
            return pos + 1
    return None

firsts = [first_relevant(it, search(it["question"], k=10)) for it in items]
recall5 = np.mean([f is not None and f <= 5 for f in firsts])
mrr10 = np.mean([1 / f if f else 0 for f in firsts])

Retrieval runs are deterministic here: the same question, index and code give the same ranking every time. So unlike the model’s answers, the retrieval numbers don’t vary between runs. They vary between question sets, which is a different problem, and the one that matters below.

The baseline and each fix

Every configuration from chapter 7, run over the 60 questions:

The track filter isn’t in the table because none of the 60 questions names a track, so it changed nothing. That’s a finding in itself: a fix can be right for one kind of question and invisible to a test set that doesn’t contain that kind. Whether that matters depends on how often real users name a track, which this set can’t say.

Most of chapter 7 holds up. Hybrid search and reranking help here too, and together they take recall@5 from 0.82 to 0.95. The heading line hurt embedding search alone (recall@1 fell from 0.60 to 0.55) but helped once BM25 was in the mix, because the heading words are exactly the kind BM25 can match. Query rewriting is as bad here as on the twenty, and HyDE, which tied the baseline on the twenty, is worse on the sixty: 0.77 against 0.82.

Passage size is where the two sets disagree most. On chapter 7’s twenty, 400-character passages did slightly worse than 800 with embeddings alone, and 1,600-character passages with reranking did best. On the sixty, 400-character passages did better with embeddings alone (0.87 against 0.82), and with hybrid search and reranking the order was 800, then 1,600, then 400 (0.95, 0.92 and 0.90). The long passages were the worst for embeddings alone on both sets, so that result is consistent. The rest flipped between two sets of questions, which says they’re within the noise of both. The next sections put a number on that noise.

Checking the labels before believing the numbers

A miss in this table means “no passage in the top five contains the span I copied”. It doesn’t necessarily mean the top five lacks the answer. The same fact is often stated in more than one place, and I copied one of them.

So before comparing configurations, I read the top five passages for every question the baseline or the best configuration missed, and checked whether any of them answered the question in other words. Three did:

• “When does fork actually copy memory?” The span was from the kernel page fault chapter’s summary. The baseline found the same chapter’s copy-on-write section (“This single decision is why fork() is cheap”), and the Redis essay’s section on snapshots (“When the parent writes to a page, the CPU faults, and the kernel copies that one page”).
• “After about how many random seven-character codes does a collision become likely?” The span was in the URL shortener’s summary; the baseline found the birthday paradox section, which says “a first collision is likely after only about 2.2 million links”.
• “How do SOCI and eStargz cut container pull times?” The baseline found the sections describing each snapshotter, which explain the lazy loading in more detail than the sentence I’d copied.

Adding those passages as acceptable answers and re-scoring every stored ranking (nothing needs to be searched again, since the rankings are saved) changed the picture:

All three corrections raised the baseline, and none of them changed the best configuration, which had already found the passages I’d labeled. The gain from chapter 7’s retrieval fixes shrank from 15 points to 10. Label errors aren’t random noise. They come from the labeler’s own idea of where the answer is, and a better retriever tends to agree with that idea more, because it’s better at finding the exact passage the labeler was reading. Uncorrected, they flatter whatever finds the labeled passage, at the expense of whatever finds an equally good one.

The audit only looked at misses. A hit can be wrong too, if a passage contains the span by coincidence, though with spans this specific that’s unlikely. All numbers from here on use the audited labels.

How much can sixty questions tell you?

At 60 questions, one question is 1.7 percentage points of recall. The baseline’s 0.87 is 52 hits; one more or one fewer moves it to 0.88 or 0.85. That granularity alone says the third digit in any of these numbers means nothing.

The more useful question is how much the number would change with a different 60 questions of the same kind. The bootstrap answers it without writing new questions: resample the 60 with replacement many times, recompute recall each time, and look at the spread.

def bootstrap_ci(hits, n=10_000, seed=0):
    rng = np.random.default_rng(seed)
    hits = np.asarray(hits, dtype=float)
    means = rng.choice(hits, size=(n, len(hits)), replace=True).mean(axis=1)
    return np.percentile(means, [2.5, 97.5])

The intervals overlap almost everywhere. On these numbers alone, the baseline and the final pipeline could both be about 0.92.

But the intervals answer the wrong question. The configurations weren’t tested on different questions; they were tested on the same 60. What matters is the difference on each question, and that’s much less noisy than either number alone. For each question, the change is +1 if the fix found it and the baseline didn’t, −1 for the reverse, and 0 otherwise:

Only two of these are clearly away from zero: the final retrieval helped, and query rewriting hurt. The rest are plausible improvements or plausible harms that 60 questions can’t separate from luck. The counts make the same point more directly. The final pipeline fixed six questions and broke none. If a change truly made no difference, and each changed question were equally likely to go either way, six out of six in one direction would happen with probability $2 \times 0.5^6 \approx 0.03$. For hybrid search alone, three fixed and one broken happens about 62% of the time by chance.

Chapter 7’s twenty questions show how misleading a small set can be. Drawing 20 questions at random from these 60, ten thousand times:

• The baseline’s recall@5 ranged between 0.75 and 0.95 in 90% of draws.
• Hybrid search alone looked no better than the baseline in 44% of draws.
• Even the final pipeline, which never did worse than the baseline on any question, looked no better in 7% of draws.

Twenty questions would have rejected hybrid search almost half the time, and the full set of fixes one time in fourteen. Sixty questions is enough to see a ten-point gain and not enough to see a three-point one. To resolve differences of a few points, the set needs to be several times larger, and it needs to contain the kinds of question each fix was aimed at.

Answers, not passages

Recall says whether the model was shown the answer. It doesn’t say whether the model used it. For that, the answers have to be graded.

I generated answers to all 60 questions with two pipelines: chapter 6’s retrieval with its first prompt, the one with the escape hatch, and chapter 7’s full pipeline (track filter, hybrid search, reranking, table headers, rows rewritten as records, and the revised prompt). Then I read every answer next to the reference answer and marked it correct if it gave the essential information. A refusal counts as wrong. An answer with the key fact and some noise around it counts as right. I knew which pipeline wrote each answer while grading, which is a bias of its own; a stricter setup would hide that.

Compared question by question, the full pipeline fixed 12 answers and broke 1. The change is +0.18, with a bootstrap interval of +0.08 to +0.30, and a 12-to-1 split happens by chance with probability about 0.003. That’s the clearest result in this chapter.

The baseline’s 20 wrong answers fall into three kinds, and they were fixed at different rates:

Half of the baseline’s failures weren’t retrieval failures. The model had the passage and said “The passages don’t say.” That’s the escape hatch chapter 6 found and replaced, measured here on questions written without it in mind: ten of sixty. I didn’t rerun the sixty with chapter 6’s second prompt, so this doesn’t say how much of the gain that prompt alone would have captured. This comparison changes retrieval and the prompt together, so it can’t say exactly how much each contributed. But retrieval recall@5 rose by ten points, and answer accuracy by eighteen, and the largest group of fixed answers is the one where retrieval was already fine.

The one answer that got worse is worth a look. “Which function in mm/memory.c handles copy-on-write faults?” got do_wp_page from the baseline, which is right. The full pipeline had the right passage at rank 2, but the model answered from passage 5, about fork, and named copy_page_range, the function that sets up copy-on-write. A prompt that lets the model answer from partial information also lets it answer from the wrong information.

The other risk of a looser prompt is answering questions the passages can’t. The ten questions from chapter 7 that the searched chapters don’t cover went through the baseline, through hybrid search with reranking, and through the full pipeline. All three declined all ten. Ten clean results rule out a common failure, not a rare one: a pipeline that made something up on one such question in ten would still pass all ten about a third of the time ($0.9^{10} \approx 0.35$).

A model as the grader

Reading every answer next to its reference is the reliable way to grade, and it doesn’t scale. The usual shortcut is to have a model do it, often called LLM-as-judge. The judge gets the question, the reference answer and the answer to grade:

You are grading an answer to a question about a technical website.

Question: {q}
Reference answer: {ref}
Answer to grade: {a}

Does the answer to grade give the same essential information as the reference answer?
An answer that says it doesn't know is incorrect. Reply with one word: CORRECT or INCORRECT.

I ran it with llama3 as the judge on the baseline’s 60 answers and compared its verdicts with mine:

The judge agreed with me on 48 of 60, and every disagreement went the same way: it never failed a right answer, and it passed 12 wrong ones. Its estimate of the baseline’s accuracy was 0.87, against 0.67 by hand. Nine of the twelve were plain refusals. The answer was “The passages don’t say.”, the prompt said in so many words that such an answer is incorrect, and the judge replied CORRECT, nine times out of sixteen. The other three were answers that circled the right topic without the fact: asked after how many random codes a collision becomes likely, the baseline gave the formula, $\sqrt{2N \ln 2}$, and not the number, about 2.2 million, and the judge accepted it.

A judge that lenient doesn’t just get the level wrong. It compresses differences. When almost everything passes, a pipeline that answers 40 questions and one that answers 51 look much closer than they are, and the most common failure in this whole chapter, a refusal with the answer in hand, is the one this judge was worst at seeing. I stopped the judge after the baseline’s 60, because the pattern was already clear and the machine was shared.

None of this makes a model judge useless. It makes it something to calibrate before trusting: grade a sample by hand, run the judge on the same sample, and look at where they disagree. Here that took one table, and it changed what the judge’s numbers could be used for.

What the judge prefers

Grading against a reference is the easier job for a judge. The other common use is comparing two answers and picking the better one, for example to decide between two prompts or two models when there is no reference. Zheng et al. describe three biases in that setup: preferring whichever answer comes first (position), preferring longer answers (verbosity), and preferring answers written by the judge’s own model (self-enhancement). Each one is cheap to test directly.

Length

For 20 of the questions, spread across the six tracks, I took the reference answer and asked llama3 to make it about three times longer “but do not add any new facts or claims”. Then llama3, as judge, compared the short and the long version:

Two assistants answered the same question about a technical website.

Question: {q}

Answer A:
{a}

Answer B:
{b}

Which answer is better: more correct, more helpful and better grounded? Reply with one letter: A or B.

The padding didn’t do what it was told. The median reference was 40 characters and the median padded version about 1,780, far more than three times, and the padding added facts anyway, some of them false. The reference “A cpio archive in newc format” came back with the claim that newc “uses a combination of Huffman coding and arithmetic coding to compress the archive”, which isn’t true: newc has no compression of its own, which is why the site’s build commands pipe it through gzip. So in every pair, the long answer has the same core as the short one plus filler, and sometimes errors. There’s no honest reason to prefer it.

Each pair was judged twice, with the short answer first and then second. Over the 40 verdicts, the long answer won 26. Taking only the 16 questions where the judge picked the same answer in both orders, it chose the long one 11 times and the short one 5 times. A second judge, qwen2.5:7b, compared the same pairs for 10 of those questions (the shared machine was too slow for all 20). It went the other way: the short answer won 15 of its 20 verdicts, and on the 5 questions where its two verdicts agreed, it chose the short one every time. On the same 10 questions, llama3 chose the long answer in 13 of 20 verdicts. Length bias isn’t a fixed property of “LLM judges”. It differs between models, and it has to be measured for the one in use.

Position

The same 40 verdicts measure position bias, since every pair was shown both ways round. A judge without position bias picks the same answer regardless of order. llama3 picked answer A 16 times and answer B 24 times, and on 4 of the 20 questions its choice flipped when the order flipped. All four times it picked B both ways round. For those questions the order decided the verdict, not the answers, and it always favored the answer it read last. That’s one question in five.

qwen2.5 was far worse. Across its 40 pairwise verdicts in this chapter, the length pairs and the comparison below, it picked answer B 32 times. In the length test, its verdict flipped with the order on 5 of 10 questions, and in the comparison below on 7 of 10, every time in favor of B. For most questions, the order decided its verdict. The standard defense is the one used here: judge every pair in both orders, and count a win only when the two verdicts agree. For qwen2.5 that leaves very few wins to count.

Its own model’s answers

The last bias needs answers from two models. For 10 questions, qwen2.5:7b answered from the same five passages and the same prompt that llama3 had in the full pipeline. Then each model judged each pair, in both orders.

Neither judge favored its own model here. llama3 picked qwen2.5’s answers slightly more often than its own, and qwen2.5’s verdicts were mostly decided by position. That isn’t evidence that self-preference doesn’t exist. Ten questions can only detect a large effect, and the two models’ answers weren’t equally good: on the collision question, qwen2.5 gave the number (about 2.2 million) where llama3 gave a formula, and on another, qwen2.5 answered “The passages don’t.” and stopped mid-sentence. A clean test would need pairs of equally correct answers, which takes the hand grading a judge was supposed to replace.

Taken together, the three tests say the same thing as the grading test. A judge’s preferences are measurable, they differ from model to model, and none of them can be assumed away. Before a judge’s verdicts are used to choose between two pipelines, they have to be checked against a person’s on the same kind of question, and every pairwise comparison has to be run both ways round.

Keeping it from getting worse

Measurements taken once are a snapshot. The pipeline will keep changing: a new chunker, a different tokenizer, a model update, a refactor that looks harmless. The retrieval half can be checked automatically on every change, because it is deterministic and cheap. The check below runs the 60 questions through the current pipeline and fails if recall@5 falls below a floor, or if more than one question that passed in the last accepted run now fails:

CONFIG = sys.argv[1] if len(sys.argv) > 1 else "hybrid-rerank"
FLOOR = 0.90          # recall@5 must stay at or above this
MAX_NEW_MISSES = 1    # one question may flip; two is a pattern

items = json.load(open("qset_v2.json"))
baseline = json.load(open("baseline_hits.json"))   # {question id: true/false} from the last accepted run

hits = {}
for it in items:
    ranked = retrieve(CONFIG, it["question"], 5)
    hits[str(it["id"])] = any(relevant(it, chunks[i]) for i in ranked)

recall = sum(hits.values()) / len(hits)
new_misses = [q for q, ok in hits.items() if baseline.get(q) and not ok]
new_hits = [q for q, ok in hits.items() if ok and not baseline.get(q)]
print(f"{CONFIG}: recall@5 = {recall:.3f} ({sum(hits.values())}/{len(hits)}), "
      f"newly failing {new_misses}, newly passing {new_hits}")
if recall < FLOOR or len(new_misses) > MAX_NEW_MISSES:
    print("FAIL"); sys.exit(1)
print("PASS")

The per-question comparison is the important part. An average can stay flat while different questions fail, and the list of question ids says exactly which passages to look at. Run against the stored hybrid-plus-rerank results, and then against the old embedding-only pipeline as if someone had reverted it by accident:

$ python check_retrieval.py hybrid-rerank
hybrid-rerank: recall@5 = 0.950 (57/60), newly failing [], newly passing []
PASS

$ python check_retrieval.py dense
dense: recall@5 = 0.867 (52/60), newly failing ['6', '8', '19', '34', '41', '57'], newly passing ['46']
FAIL

The full run took 15 seconds, most of it the cross-encoder on the CPU, with the question embeddings cached. That’s cheap enough to run on every commit.

A second test shows what such a check misses. A plausible simplification is to tokenize for BM25 with str.split() instead of the regular expression from chapter 7. On the 60 questions, that change passed, and even found one more question. On chapter 4’s forty identifiers, it dropped the pipeline from 40 found in the top five to 32: with whitespace splitting, the question’s last word becomes the token pfcount? and the passage’s becomes `pfcount` with its backticks, so they no longer match. RefCell, task_destroy and five others went the same way. The 60 questions contain almost no identifiers, so they can’t see a regression that only affects identifiers. A regression suite needs a slice for each kind of question that a fix was made for: the forty identifiers, the twenty questions from chapter 7, the ten questions the chapters don’t cover. Each slice protects one fix from being undone without anyone noticing.

The answer half is harder to put in CI. Generation at temperature 0 with a fixed seed repeats on the same machine and the same Ollama version, but not across versions or hardware, and checking the answers needs either a person or a judge model, which brings its own problems, as the previous section showed. A practical split is to run the retrieval check on every change, and the slower answer evaluation, with a judge and a sample of hand checks, before a release or when the model changes.

Where this leaves things

Sixty hand-written questions, each with the answering text copied from the site, were enough to check chapter 7’s claims on questions that played no part in making them. Most held. Hybrid search and reranking raised recall@5 from 0.87 to 0.97 with audited labels, and the full pipeline raised the share of correct answers from 0.67 to 0.85. Some didn’t: HyDE did worse than the baseline here, and the passage sizes ranked differently on the two sets.

The numbers come with their limits attached. One question is 1.7 points. Only two retrieval changes were clearly outside the noise, and twenty questions would have rejected hybrid search almost half the time. Labels have errors, and those errors favor whichever system agrees with the labeler. A model grading answers passed 12 wrong answers out of 20 and failed none of the right ones, and when comparing two answers, it leaned toward the longer one and, on one question in five, toward whichever came second. A regression check on the retrieval half is cheap enough to run on every change, as long as it has a slice for each kind of question a fix was made for.

One kind of question failed in every configuration, and it was never in the set, because no passage answers it. Chapter 6 asked “How many chapters does the system design track have?” and got a cited answer that implied eight. With every fix from chapter 7, the track filter sends the search to the right track, and the five passages that come back are from chapters 1, 6, 23 and 31. None of them says how many chapters there are. The answer, 31, isn’t written anywhere on the site. It’s a fact about the collection: the number of files in a directory. “Which chapters on this site cover consistent hashing?” fares better on retrieval: with the fixes, the five passages come from all three chapters that mention it. But nothing in them says the list is complete, and chapter 6’s model, given good passages for the same question, never named the chapters at all.

No amount of retrieval tuning reaches those answers, and measuring it more carefully won’t either. They need the system to do something rather than read something: list the files, count them, look the answer up. Chapter 9 gives the model a way to ask for that.