7 — Why it still gets things wrong

The pipeline from chapter 6 cuts the site into passages of about 800 characters, embeds them with nomic-embed-text, takes the five nearest to the question by cosine similarity, and hands them to Llama 3 with an instruction to answer only from them and cite them. When it fails, the answer usually looks just as confident as a good one, so the failures have to be found deliberately.

Everything here ran on one Mac, an Apple M3 Pro with 18 GB of memory, with Ollama 0.40.11 serving llama3:latest (8 billion parameters, Q4_0 quantization) and nomic-embed-text (137 million parameters, F16). Every generation used temperature 0, a fixed seed, a context window of 4,096 tokens (num_ctx) and a cap of 512 output tokens (num_predict). Timings are from that machine, which was shared with other work at the time, so they’re rough.

Twenty questions, and where they failed

To find failures, I wrote twenty questions about the site. Half were ordinary reader questions; the rest probed other shapes: a number from a table, a question that names a track, a one-word query, a symptom instead of a question. Each comes with a short piece of text copied from the chapter that answers it, and a retrieved passage counts as a hit if it contains that text.

The first useful split is between two kinds of failure. Either the passage with the answer never reached the model, or it did and the model still got the answer wrong. The fixes are completely different, and the first kind can be checked without calling the model at all:

def first_hit(question, span, k=10):
    ranked = dense_search(question, k)          # chapter 6's search, top 10
    for pos, i in enumerate(ranked):
        if span in chunks[i]["text"]:
            return pos + 1                       # 1-based rank of the first passage with the answer
    return None                                  # not in the top 10

Before trusting the count, I read the passages behind every miss. Five of my labels were wrong. “How much shared memory can one SM have?” found a passage saying “up to 228 KB of it can be configured as shared memory”, a different sentence from the one I’d copied but the same answer. The Bloom filter and database-model questions found equally good answers in other chapters. Two more spans pointed at the wrong sentence. After adding the passages that really answer each question, the baseline had the answer in the top five for 16 of the 20. These were the misses and near-misses:

Some of these were predictable. Chapter 4 measured that a question naming an exact identifier finds its passage in the top five only 25 times out of 40, because an identifier like PFCOUNT is split into word pieces that carry no meaning. The readahead, Kafka and XDP questions are that problem showing up in real questions.

The passage itself: chunking

The table question shows the simplest cause. The comparison of data stores in the system design track is a markdown table, and the chunker, which splits on blank lines and then on lines when a block is too long, cut it into pieces. The piece with the answer starts like this:

varies | Joins across documents, many-to-many relationships | MongoDB, or PostgreSQL `jsonb` |

| **Wide-column** | Very high write rates, reads by a known partition key and a range inside it | Queries not planned into the table design, ad hoc filtering | Cassandra |
| **Graph** | Multi-hop traversals run constantly: networks, dependencies, recommendations | Bulk scans and aggregates | Neo4j |

The header row, | Model | Fits | Fits poorly | Examples |, went into the previous passage. Without it, “Bulk scans and aggregates” is just a cell.

The fix is mechanical. When a passage starts in the middle of a table, find that table’s header row in the source and repeat it at the top:

SEP = re.compile(r"^\|\s*:?-{3,}")          # the |---|---| line under a header

def add_table_header(chunk, source_text):
    lines = [l for l in chunk.split("\n") if l.strip()]
    first_row = next((l for l in lines[:3] if l.startswith("|")), None)
    starts_in_table = lines and (lines[0].startswith("|") or lines[0].rstrip().endswith("|"))
    if not starts_in_table or not first_row or any(SEP.match(l) for l in lines[:3]):
        return chunk                                   # not mid-table, or already has its header
    pos = source_text.find(first_row)
    if pos <= 0:
        return chunk
    above = source_text[:pos].rstrip("\n").split("\n")
    for k in range(len(above) - 1, 0, -1):            # walk back up through the table
        if not above[k].startswith("|"):
            break
        if SEP.match(above[k]):
            return above[k - 1] + "\n" + above[k] + "\n" + chunk
    return chunk

Chapter 6 counted 318 passages with table rows but no header row. This function found and restored a header for 289 passages, and only those needed new embeddings. On the twenty questions, the table question went from rank 3 to rank 1 and nothing else changed.

Retrieval wasn’t the whole failure, though. With the header in place and the passage first, llama3 answered “The passages don’t say.” With the revised prompt described later in this chapter, it answered, but wrongly: “the Relational model fits poorly for bulk scans and aggregates”. The right row was there, under a header that said “Fits poorly”, and it read the wrong one. A 4-bit 8B model reading pipe-separated text seems to lose track of which cell belongs to which column. What worked was rewriting the rows as plain records before they go into the prompt, so each value carries its column name:

Model: **Graph**; Fits: Multi-hop traversals run constantly: networks, dependencies, recommendations; Fits poorly: Bulk scans and aggregates; Examples: Neo4j

With the same five passages in that form, the answer was “the Graph model”. One question isn’t evidence that this works generally, but it shows that one failure can need both a retrieval fix and a generation fix.

Size

The passage size of 800 characters was a guess in chapter 6. Smaller passages are more focused but carry less context; larger ones carry more and dilute it. I rebuilt the index at 400 and 1,600 characters and reran the twenty, counting the questions with the answer in the top five. (Hybrid search and reranking are described below.)

For embeddings alone, the long passages were clearly worse: 11 of 20. A 1,600-character passage often covers two or three things, and its one vector sits somewhere between them. With reranking on top, the long passages did best.

Twenty questions can’t tell 17 from 18 from 19, and five long passages are twice as much text for the model to read. I kept 800 characters.

A heading on every passage

A passage cut from the middle of a chapter doesn’t say which chapter or section it came from. The chunks already carry that from chapter 6, so it costs nothing to put it in front of the text before embedding:

text = f"{c['title']} > {c['section']}\n\n{c['text']}"   # e.g. "7 — Caching > Stampedes\n\n..."
E = embed([text for c in chunks], prefix="search_document: ")

The count in the top five didn’t change (16 of 20 again), but individual questions moved a lot. The Kafka follower question went from not found to rank 1, because its passage sits under “In-sync replicas” in a chapter about Kafka, and now its vector says so. “caching” went from rank 3 to rank 1. Going the other way, the XDP question dropped from rank 4 to out of the top ten: its passage now begins with the chapter title and “The path”, which matches “possible return values” worse. A heading pulls a passage toward what its section is about, which is usually what a question is about, but not always.

The kernel-track question didn’t move. Its chapter is titled “The page cache and readahead”, and nothing in a title says “kernel”. The track is in the URL, not in the text.

Words that have to match exactly: hybrid search

Three of the failures depend on exact strings: PG_readahead, replica.lag.time.max.ms, and XDP_DROP and its siblings. Embeddings are bad at these. Plain word matching is good at them, and that’s the oldest kind of search there is.

BM25 is the standard word-matching score. For each query word that appears in a passage, it adds a weight that grows with how often the word appears there, with diminishing returns, and that is larger for rare words than for common ones. It also discounts long passages, which contain more words by chance.

$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d)\,(k_1 + 1)}{f(t, d) + k_1 \left(1 - b + b \frac{|d|}{\text{avgdl}}\right)}$

Here $f(t, d)$ is how many times term $t$ appears in passage $d$, $|d|$ is the passage length in words, avgdl is the average length, and $k_1$ and $b$ are tuning constants (the rank-bm25 package defaults are 1.5 and 0.75).

import re
from rank_bm25 import BM25Okapi

def tokenize(s):
    return re.findall(r"[a-z0-9_]+", s.lower())

bm25 = BM25Okapi([tokenize(c["text"]) for c in chunks])

def sparse_search(q, k=10):
    scores = bm25.get_scores(tokenize(q))
    return list(np.argsort(-scores)[:k])

The tokenizer matters more than it looks. Keeping underscores inside tokens means pg_readahead and tcp_v4_connect stay single tokens. A whitespace split would leave punctuation attached.

On its own, BM25 found the readahead and Kafka passages at rank 1, both of which the embedding search missed entirely. But it lost questions the embeddings had found: XDP (the constants appear in several passages, so they’re less rare than they look), Redis, and the cold-start question, which says “database”, “overloaded” and “restarted” where the passage says “empty cache” and “cache restart”. Over the twenty, BM25 scored 15 in the top five against the embeddings’ 16. They fail on different questions.

Combining two rankings

The two searches score on unrelated scales: cosine similarities between about 0.4 and 0.8, BM25 scores from 0 to 30 or more. Adding them would let one dominate. Reciprocal rank fusion uses only ranks. Each passage gets, from each ranking it appears in, one over a constant plus its rank:

$\text{RRF}(d) = \sum_{r \in R} \frac{1}{k + r(d)}$

Cormack, Clarke and Büttcher, who proposed it, fixed $k = 60$ in a pilot and didn’t tune it after. The constant flattens the curve: rank 1 is worth 1/61 and rank 10 is worth 1/70.

def hybrid_search(q, k=10, depth=50, c=60):
    score = {}
    for ranking in (dense_search(q, depth), sparse_search(q, depth)):
        for r, i in enumerate(ranking):
            score[i] = score.get(i, 0) + 1 / (c + r + 1)
    return sorted(score, key=lambda i: -score[i])[:k]

Hybrid search also found 15 in the top five. That looks like no progress, but it fixed and broke different questions:

(A dash means not in the top 10.) Fusion helps when both lists have the answer somewhere near the top, as with XDP. It can’t rescue an answer that only one list has: rank 1 in one list is worth 1/61, while a passage at rank 5 in both gets 2/65, which is more. The readahead passage ended up behind passages that both searches thought were moderately relevant.

Back to the identifiers from chapter 4

The forty identifiers from chapter 4 make a cleaner test, because each appears in exactly one passage. The question is “What is X?”, and the measurement is where that passage ranks:

BM25 gets all forty at rank 1, which is what a word index should do when the query contains a string that occurs in one passage. Fusion loses some of that sharpness, but all forty are in the top five, against 25 for the embeddings alone. PFCOUNT went from outside the top 20 to rank 2, and RefCell to rank 3. Those were two of the failures chapter 6 ended on, where the passage containing PFCOUNT ranked 1,946th and both answers said the passages didn’t mention it. With hybrid search, the passage that answers each is in front of the model.

Better ordering: reranking

Fusion fixed the order for some questions and scrambled it for others. Both searches score the question and the passage separately: each passage’s vector, or its word counts, is computed once, without ever seeing the question.

A cross-encoder reads the question and the passage together, as one input, and outputs a single relevance score. It can notice that “Reaching that folio triggers page_cache_async_ra, which starts the next window” is exactly what was asked. The cost is one model run per passage per question, so it can’t search 3,951 passages. It can reorder 30.

The one used here is cross-encoder/ms-marco-MiniLM-L6-v2, a small model trained on the MS MARCO passage ranking data, run on the Mac’s CPU through the sentence-transformers package, since Ollama doesn’t serve this kind of model.

from sentence_transformers import CrossEncoder
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def rerank(q, ids, k=5):
    scores = ce.predict([(q, chunks[i]["text"]) for i in ids])
    return [ids[j] for j in np.argsort(-scores)[:k]]

top5 = rerank(q, hybrid_search(q, k=30))

Reranking the top 30 from hybrid search put the readahead passage back at rank 1 and the Kafka passage at rank 2. Over the twenty, hybrid search followed by reranking found 18 in the top five, against 16 for the baseline. Reranking the embedding search’s top 30 instead reached only 16, because a reranker can only reorder what it’s given, and the embedding search never retrieved the readahead or Kafka passages. The first stage decides what can be found; the reranker decides what comes first.

It has side effects. “caching” went from rank 3 to 4 and the cold-start question from 2 to 4. The cross-encoder was trained on web search questions with one clear answer, and a one-word query or a question about a symptom doesn’t look like those.

Changing the question: rewriting and HyDE

“How does Redis store small hashes?” fails for a different reason. The answer is in the Redis essay under “Small collections: listpack”, but that passage talks about pointers, cache lines and a flat block of memory, while the question’s words point at hash tables, and there are dozens of passages about hash tables on this site. Two techniques try to fix this by changing what gets searched for.

Query rewriting asks the model to turn the question into a better search query. My prompt described the site and asked for the terms an article would use:

Rewrite this question as a short, specific search query for a set of technical articles
about Linux, the kernel, GPUs, system design, algorithms and data structures. Expand
abbreviations and add the technical terms an article would use. Output only the query.

The description of the site leaked into the queries. A Redis question became a Linux kernel question with GPU acceleration, and a Kafka follower became a consumer, which is a different thing.

HyDE (Gao et al., 2022) asks the model to write a hypothetical answer instead, and searches with the embedding of that answer. A fake answer, even a wrong one, uses the vocabulary of a real answer, so it should land near real answers. The fake answer itself is thrown away.

HYDE = """Write a short paragraph, in the style of a technical article, that answers this question.
Write the paragraph only.

Question: {q}"""

fake = generate(HYDE.format(q=q))
v = embed([fake], prefix="search_document: ")[0]   # embedded as a passage, not as a query
ranked = np.argsort(-(E @ v))

For the Redis question, llama3 wrote that Redis stores small hashes “in a specialized data structure called a “hash table” or “dictionary"" using “a technique called “bit-packing"". None of that is right, but it is about compact memory layouts.

Rewriting made things much worse: 11 of 20 in the top five with embedding search, against 16 for the original questions, and 13 instead of 18 in front of hybrid search and reranking. HyDE found 16 of 20, the same as the baseline, and moved the Redis passage from rank 209 to rank 26. That’s a real improvement in where the answer is, and still nowhere near the top five.

The Redis question is the one failure in the twenty that nothing in this chapter fixed. A reader who knows the word “listpack” finds the passage immediately. One who doesn’t needs the system to know it.

Both techniques also add a full generation before the search can start. On ten questions, llama3 wrote a median of 20 tokens for a rewrite and 136 for a HyDE paragraph: 0.7 s and 5.2 s of decoding, by Ollama’s own counters.

Saying where to look: metadata filters

“In the kernel track, what happens on a page cache miss?” names where the answer is, but the search doesn’t know what a track is. The embedding search returned the kernel’s page fault chapter, the URL shortener’s section on caching redirects, the Redis essay’s snapshot section and the GPU track’s KV cache section: all about some kind of cache and some kind of miss.

Every chunk records the URL it came from, and the first part of the URL is the track. A filter that recognizes a track name in the question and restricts the search to that track is a few lines:

TRACK_WORDS = {
    "kernel": r"\bkernel track\b|\bkernel chapters?\b",
    "gpu":    r"\bgpu track\b|\bgpu chapters?\b",
    "design": r"\bsystem design\b|\bdesign track\b",
    "dsa":    r"\bdsa\b|\balgorithms track\b|\bdata structures track\b",
    "blog":   r"\blinux from scratch\b|\bbuild(ing)? linux\b|\blinux track\b",
    "essays": r"\bessays?\b",
}
tracks = np.array([c["url"].split("/")[1] for c in chunks])

def track_mask(q):
    hits = [t for t, pat in TRACK_WORDS.items() if re.search(pat, q, re.I)]
    return np.isin(tracks, hits) if hits else None

def dense_search(q, k=10, mask=None):
    s = E @ embed_query(q)
    if mask is not None:
        s = np.where(mask, s, -np.inf)     # passages outside the track can't be chosen
    return list(np.argsort(-s)[:k])

With the filter, the embedding search still put the page fault chapter first, because it’s in the kernel track too and talks about the page cache. The page cache chapter’s passage appeared at rank 2 once hybrid search and reranking ran on top of the filter: the filter removed the distractions from other tracks, and the reranker picked among what was left. Over the twenty, filter plus hybrid search plus reranking found 19 in the top five. Only one question names a track, so only that one changed.

In PostgreSQL with pgvector, the filter is a WHERE clause next to the distance ordering, with the track stored as its own column:

SELECT id, url, section
FROM chunks
WHERE track = 'kernel'
ORDER BY embedding <=> $1
LIMIT 10;

I ran it on PostgreSQL 16.2 with pgvector 0.6.2 and no vector index, so the plan was a sequential scan with the filter followed by a sort on distance: exact, and fine at a few thousand rows. With an HNSW index, the filter applies to the candidates the index returns, so a selective filter can leave fewer than ten rows, or none. Chapter 5 measured that trap.

A regular expression is a crude way to detect a track, and it only helps questions that name one. When a question says where to look, that belongs in a constraint, not in the hope that the embedding picks it up.

When the answer was already there: the prompt

Chapter 6 found the biggest single cause of wrong answers in the prompt, not in retrieval. Its first prompt ended with “If the passages do not contain the answer, say “The passages don’t say.” and stop.”, and the model took that exit whenever the passages didn’t answer head-on. Chapter 6 replaced it with a second version that asks for a partial answer before a refusal, and counted how many refusals went away.

All the answers in this chapter and the next were generated with chapter 6’s first prompt as the “before”, because that’s the pipeline I started from. That turns out to be useful: it lets the prompt change be measured separately from the retrieval changes, and in both directions. Under the first prompt, “Why does Kafka use a consumer group?” got “The passages don’t say.” with the section on consumer groups among the five passages. So did “When is a partial index useful in PostgreSQL?”, “What does the verifier track in bpf_reg_state?” and “Is the cache invalidated on writes in the distributed cache design?”. In each case the passages held at least part of the answer.

My revised prompt makes the same move as chapter 6’s second one, keeping the refusal sentence but narrowing when it applies:

Answer the question using the passages below. Use only information from the passages,
and cite the passages you use with their numbers in square brackets, like [2]. If the
passages answer only part of the question, answer that part. If none of the passages is
relevant to the question, say "The passages don't say." and stop.

Loosening a refusal instruction has an obvious risk: the model may start answering from its own training. Counting refusals that went away doesn’t show that, so the test needs both kinds of question. Alongside the four refusals, I wrote ten questions the searched chapters don’t cover, checked with grep that their key terms appear nowhere in the passages, and ran both prompts over the same retrieved passages:

The ten were things like “What port does MongoDB listen on by default?”, “How does ZFS detect silent data corruption?” and “What is the default block size in HDFS?”. Llama 3 knows something about all of them, so declining is a real choice.

The four answers weren’t equally good. Three quoted the passages and cited them. The partial index answer had no citation, for a reason: the retrieved passage held only the heading “Partial indexes” and a CREATE INDEX ... WHERE status = 'pending' example, and the explanation was in the next passage, which wasn’t retrieved. The model filled the gap with “useful when you have a large table and only a small portion of the data is frequently accessed”, which is true, but came from its training. A missing citation turned out to be a useful signal of that.

Four and ten are small numbers. They show the change does what it was meant to on the cases that prompted it, not how often it misfires in general.

All of it together

The fixes stack: the track filter narrows the candidates, hybrid search finds both exact strings and paraphrases, the cross-encoder orders the top 30, table headers go back on passages that lost them, and the revised prompt stops the model declining when it has part of the answer. On the twenty questions:

MRR@10, the mean reciprocal rank, averages one over the rank of the first correct passage (1 for rank 1, 0.5 for rank 2, 0 if it’s not in the top 10). Chapter 8 defines it properly. Rewriting and HyDE stay out of the combined pipeline: one made things worse, and the other didn’t improve the count enough to pay for a generation in front of every search.

The retrieval numbers are a proxy. What matters is what the assistant says. These are five of the original failures, before and after, with the same model at temperature 0:

The Kafka answer before the fixes is the dangerous kind: confident, cited, and about something else. The Redis answer is still empty, but now honestly so.

What the fixes cost

Retrieval used to be one embedding call and one matrix multiplication. Each stage was timed on the twenty questions on the M3 Pro, with the cross-encoder on the CPU (the query embedding over ten fresh questions):

The fixes that went into the pipeline add about 120 ms per question, almost all of it the reranker. Against an answer that takes several seconds to generate, that’s small. HyDE alone would have added more than forty times as much.

On the index side, building BM25 over 3,951 passages (15,073 distinct words) took about 0.11 s, and restoring table headers needed 289 new embeddings.

Where this leaves things

Each failure of the first pipeline had a specific cause, and most had a specific fix. Exact strings need word matching next to the embeddings, combined by rank. A reranker that reads question and passage together puts the right passage first. A table needs its header, and an 8B model reads it better as records. A question that names a track should be searched within that track. A prompt that offers an easy way out gets taken up on it. Rewriting the question and searching with a hypothetical answer didn’t help here, and rewriting did real harm. The Redis listpack question still fails, because the reader’s words and the site’s words don’t overlap.

On the twenty questions, the answer is now in the top five for 19, up from 16. That number is the problem. Every fix was chosen by looking at a failure in those twenty, and every measurement was taken on the same twenty. Some fixes helped one question and hurt another in the same table: the heading line rescued Kafka and lost XDP; the reranker fixed readahead and pushed “caching” down. Nothing here says what they did to the questions nobody looked at. A pipeline tuned on its own test cases will always look like it improved. It may only have learned those cases.

Knowing whether a fix helped needs questions that played no part in choosing it, with the right answers decided before the run, and enough of them that a gain of a few points can be told apart from luck. Chapter 8 builds that set and runs every fix in this chapter against it.