3 — It doesn’t know your documents

The first two chapters kept running into the same gap. In chapter 1, the model explained mutexes without ever mentioning ownership, the idea this site’s essay on them is built around. In chapter 2, the routing questions it got wrong were the ones whose answer depended on what this site happens to contain. The assistant’s real job is to answer questions about this site’s chapters. So the obvious next step is to ask the model about them directly and count how often it’s right.

Everything here ran on the same setup as before: llama3:latest (Llama 3 8B, 4-bit Q4_0) on Ollama 0.40.11, on an Apple M3 Pro with 18 GB of memory. The timings come from that one machine, which was running other work at the same time, so they show the shape of the costs more reliably than their exact size.

Asking about the site

The test is fifteen questions with short, checkable answers, each taken from a chapter on this site. Some ask for a fact stated in a chapter’s description, some for a number in an essay, some for a chapter’s title:

Each question names the site and the track, ends with “Answer in one or two sentences”, and went to the model twice at the default settings. Then the same thirty requests went out again with a system message added: “Answer questions about the website ashokg.ai. If you do not know the answer, say “I don’t know”.” I graded all sixty answers by hand into three groups: correct, an abstention (the model said it didn’t know, or asked for more context), and wrong.

The model got almost nothing right, and that’s expected. These pages are recent and specific, and an 8-billion-parameter model trained on a snapshot of the web has no reason to know them. The correct answers were BusyBox for /bin, which it got in all four attempts, and PostgreSQL for the URL shortener, which it got once. Both are what a small Linux build or a typical web design would use anyway. They’re good guesses, not knowledge of this site.

The problem is the 22 wrong answers, and how they were phrased. Here are some, verbatim:

According to the metadata, the "Kernel, Traced" track on ashokg.ai builds kernel version 5.4.131.

The 'Linux, Observed' track on ashokg.ai uses the TensorFlow library, which is a popular
open-source software library for machine learning and artificial intelligence.

According to the rate limiter chapter on ashokg.ai, the implemented and tested rate limiting
algorithm is the Leaky Bucket algorithm, which is a well-known and widely used algorithm for
rate limiting.

SynapseOS is an open-source operating system being developed by Ashok Gajos and his team ...

According to the benchmark essay on ashokg.ai, the SynapseOS detonation chamber is compared
against the Apache Spark cluster in the performance benchmark.

None of these hedge. 35 of the 60 answers contain the phrase “according to”, attributing an invented fact to a page the model has never seen. The kernel answer cites “the metadata”. The SynapseOS answer invents an author’s surname. The leaky bucket answer is a particularly good fake: it’s the other well-known rate limiting algorithm, so it sounds right to anyone who knows the field and doesn’t know this site. The answers to the same question also disagreed between runs (kernel 5.4.131, 5.4 and 5.8; Python 3.8 and 3.9; ten, twelve and six chapters), which is chapter 1’s sampling at work: there’s no stored fact behind any of them, so each run draws a different plausible one.

The mutex question deserves its own mention. All four answers described a mutex as a lock that lets one thread in at a time, which is true, and none mentioned ownership. Graded strictly against the essay, they’re wrong. Read on their own, they’re fine. That’s the hardest kind of error to catch, because nothing in the answer looks invented.

Telling the model it may say “I don’t know” didn’t help. It abstained four times instead of six, and was wrong 23 times instead of 22. On samples of thirty, that’s no difference at all. A sentence in the prompt doesn’t give the model a way to tell what it knows from what it doesn’t. As chapter 1 showed, it produces the most likely continuation, and for a question phrased like a factual lookup, a confident factual answer is the most likely continuation.

This failure has a name, hallucination, but the word makes it sound like a malfunction. It’s the model doing exactly what it does, predicting plausible text, in a situation where plausible and true have come apart. For an assistant whose whole purpose is answering questions about these pages, it’s disqualifying. The model has to be given the pages.

Giving it the document

The direct fix is to put the text in the prompt. If the question is about the mutex essay, send the essay with it. The essay is 16,762 characters, which comes to a little over 4,000 tokens, and that’s the first problem: as chapter 1 found, Ollama loads llama3 on this Mac with a 4,096-token window.

essay = open("src/content/essays/semaphore-mutex-lock.md").read()
q = "According to the essay above, what is the key idea of a mutex? Answer in one or two sentences."
d = chat([{"role": "user", "content": essay + "\n\n" + q}],
         options={"num_ctx": 4096, "temperature": 0})

With the whole essay, the prompt came back as 4,047 tokens, trimmed to fit, and the answer was:

The key idea of a mutex is that at most one thread holds it, and the thread that locks a mutex
is meant to be the one that unlocks it, with the mutex protecting a piece of data and the
holder having exclusive use of it.

With only the essay’s first 12,000 characters (2,909 tokens, comfortably inside the window), it was sharper:

The key idea of a mutex is **ownership**: the thread that locks a mutex is meant to be the one
that unlocks it, and the mutex protects a piece of data, with the holder having exclusive use of it.

That’s the answer chapter 1 was missing, in the essay’s own words. The model didn’t learn anything between the two chapters. It was given the passage that contains the answer, and reading a passage and restating it is something it does well. The trimming in the first request happened to cut text the answer didn’t need. It won’t always.

The next step up shows what happens when it does. The Linux, Observed track is twelve chapters and a principles page, about 65,000 characters. Sent whole, with a question about chapter 4, it was cut to 2,060 tokens, the same truncation chapter 1 measured, and that’s under a sixth of the text. What survived was the end of the last numbered chapter and the principles page that follows it. Two questions against that same truncated prompt:

The first answer is a real program from the site, quoted from the part of chapter 11 that survived the cut, and confidently presented as the answer about chapter 4, which the model never saw. The second answer is right only because musl comes up all over the track, so the surviving fragment mentioned it too. From the outside, the two answers look equally sure of themselves.

The whole site doesn’t fit

Those were one essay and one track. The assistant is meant to answer questions about all of the site: the Linux, Observed, Kernel, Traced, GPU, System Design and DSA tracks, and the essays. To know how far that is from fitting, the text has to be counted in tokens. Counting every file through the model would take hundreds of requests, so two files from each group were counted exactly, in chunks of about 6,000 characters using the prompt_eval_count trick from chapter 1, and the rest estimated from each group’s ratio of characters to tokens:

The sampled files alone came to 59,964 tokens. Technical writing runs denser than the four characters per token rule of thumb: the kernel and Linux tracks, with their shell sessions and C, came in at under 3.5.

So the site is about 860,000 tokens. The window Ollama gives llama3 here is 4,096, and the most Llama 3 was trained to handle is 8,192. The site is more than 200 times the first and over 100 times the second. Pasting everything in isn’t a matter of tuning a setting.

Models with much larger windows exist, from 128,000 tokens up to a million or more, and the obvious question is whether a larger window solves this. For this site, a million-token window would technically hold it. But a window isn’t free. Every token in the prompt has to be processed before the first token of the answer appears, and on this machine, prompt processing has a measurable cost:

Each row is three runs on different stretches of text, so the prompt cache couldn’t help (each run reused only the begin-of-text token). The times are Ollama’s own prompt_eval_duration, measured while no other timed measurement was running, though the machine was otherwise busy. Prompt processing ran at about 280 tokens per second and grew in step with the prompt. Writing the 32 answer tokens took 1.2 to 1.3 seconds every time, whatever the prompt length. That’s about 26 tokens per second, faster than the 17 measured once in chapter 1, a reminder of how much the load on a shared machine moves single timings.

So a full 4,096-token window costs about 14 seconds before the first word of the answer. At the same rate, the whole site would take around 50 minutes per question, and that’s a lower bound, since attention’s cost per token rises with length. Even a model that could hold it would make every question wait for the entire site to be read again.

Even setting cost aside, a big window raises a different question: does the model actually use what’s in it? The next experiment measures that.

Does it use what it’s given?

A known weakness of long prompts is that models use information unevenly across them. Liu et al., 2023, “Lost in the Middle”, found that models answered questions best when the relevant passage was at the start or end of a long input, and noticeably worse when it was in the middle. Whether that matters here depends on the model and the length, so it’s worth measuring.

The test plants a fact in real text from this site. The filler is consecutive paragraphs from five System Design chapters (caching, resilience, partitioning, replication, and queueing), cut to a target length. Into it goes one invented sentence that the site doesn’t contain, such as “The orders table was split into 48 partitions by customer ID.”, at one of five positions: the very start, a quarter of the way in, halfway, three quarters, or the very end. Then the model is asked about it at temperature 0.

NEEDLES = [
  ("In the staging load test, the cache tier's p99 read latency was 41 milliseconds.",
   "In the staging load test, the queue tier's p99 read latency was 63 milliseconds.",
   "In the staging load test, what was the cache tier's p99 read latency?", "41"),
  ("The team set the circuit breaker on the payments client to open after 7 consecutive failures.",
   "The team set the circuit breaker on the search client to open after 12 consecutive failures.",
   "After how many consecutive failures does the circuit breaker on the payments client open?", "7"),
  ("The orders table was split into 48 partitions by customer ID.",
   "The events table was split into 96 partitions by device ID.",
   "Into how many partitions was the orders table split?", "48"),
  ("Replica lag on the reporting follower peaked at 2.4 seconds during the backfill.",
   "Replica lag on the analytics follower peaked at 9.1 seconds during the backfill.",
   "How high did replica lag on the reporting follower peak during the backfill?", "2.4"),
]
msg = f"Here are some notes:\n\n{doc}\n\nUsing the notes, answer the question in one short sentence.\nQuestion: {q}"

The second sentence in each tuple is a distractor: a near-copy of the fact with a different subject and number, the way real documents repeat a pattern with different values (“the p99 for the cache was … the p99 for the queue was …”). In the distractor runs it goes in half a document away from the fact, so a fact at the start has its distractor in the middle, and the other way round. An answer counts as correct if it contains the right number.

Four facts and five positions give 20 runs per setup. The short and medium setups used Ollama’s default 4,096-token window. The long ones raised it to 8,192 with "num_ctx": 8192, close to the most Llama 3 was trained on, with num_predict capped at 80 tokens:

Without a distractor, the model found almost every fact at every length. The one miss at 6,730 tokens was odd rather than wrong. With the replica-lag fact a quarter of the way in, the model ignored the question and answered “The takeaway is that partitioning, or sharding, splits data across machines…”, repeating a summary line from the surrounding chapter. In the same position, another answer began with “The takeaway is that…” before giving the right number. Long filler text pulls the model toward continuing the document instead of answering about it.

The distractors are where it really slipped, and where position started to matter. At about 2,950 tokens there was one wrong answer. With the payments fact at the very start and the search-client distractor in the middle, the model answered “The circuit breaker on the payments client opens after 12 consecutive failures”, taking the number from the distractor and the subject from the question. Another answer passed the check but shouldn’t really count. With the orders fact in the middle and the events distractor at the start, it said “The orders table was not mentioned in the notes, but the events table was split into 96 partitions by device ID, and the orders table was split into 48 partitions by customer ID.” The right number is in there, next to a claim that it isn’t.

At about 6,750 tokens, the pattern was sharper. All three wrong answers came from the same arrangement: the fact in the middle of the notes and its distractor at the start. In that arrangement, three of the four facts lost to their distractors:

The cache tier's p99 read latency was 41 milliseconds.        -> answered 63 (the queue tier's)
The orders table was split into 48 partitions by customer ID.  -> answered 96 partitions by device ID
Replica lag on the reporting follower peaked at 2.4 seconds.   -> answered 9.1 (the analytics follower's)

Each wrong answer kept the subject from the question and took the number from the distractor. With the fact at the start, at either quarter mark or at the end, every distractor run was answered correctly. That matches the “lost in the middle” finding in a small way: in the middle of a long input, a fact can lose to a lookalike nearer the start, and the answer still sounds like it comes from the notes.

Twenty runs per setup is a small sample, and three misses in one position isn’t a measured curve. But the direction is clear, and it’s the opposite of what a bigger window promises. The more text in the window, the more likely it contains something that looks like the answer and isn’t, and the further the real answer can sit from where the model attends best.

Where this leaves things

Three measurements, three answers to the same question.

Asked about this site with nothing to go on, the model answered 60 questions and got 5 right, all of them guessable. It made up the rest with full confidence, often citing the site as its source, and an instruction to admit ignorance changed nothing measurable.

Given the right passage, it answered well. With the first part of the mutex essay in the prompt, it finally named ownership as the key idea, in the essay’s own words. It found planted facts at any position in notes of up to about 6,700 tokens. It slipped when a near-identical sentence sat elsewhere in the text, and at the longer length, a fact in the middle lost to a lookalike at the start three times out of four.

Given everything, it can’t cope. The site is about 863,000 tokens, more than 200 times the window this model runs with here. Overfilling the window doesn’t produce an error. It silently drops text, and the model answers from whatever survived, sometimes correctly, sometimes with a confident answer about a chapter it never saw. Even with a far bigger window, reading the whole site would cost minutes per question and fill the prompt with near-misses like the distractors above.

The way forward is between the two extremes. For each question, the assistant has to pick out the few passages of the site that are actually relevant, perhaps a handful of paragraphs out of thousands, and give the model only those. That’s a search problem, and not an easy one. The question about mutex ownership never uses the word “ownership”. A question about “the program that runs as PID 1” should find a chapter titled “PID 1”, but a question about “the first process the kernel starts” should find it too. Matching words isn’t enough. The search has to match meaning.

Chapter 4 starts there, by turning text into numbers that capture what it means, so that “which part of the site talks about this?” becomes a question of which numbers are closest.