6 — The first pipeline

The pieces are all on the table now. Chapter 3 showed that llama3 invents confident answers about this site, and that the site doesn’t fit in its context window. Chapter 4 turned text into vectors that can be compared by meaning, and showed the blind spots of that comparison. Chapter 5 found the nearest vectors quickly, in numpy and in PostgreSQL.

This chapter connects them into the first working version of the assistant. It reads the site’s markdown files, cuts them into passages, embeds the passages and stores them, and then, for each question: embeds the question, retrieves the five closest passages, puts them in a prompt and asks llama3 to answer from them with citations. This pattern is usually called retrieval-augmented generation, or RAG. Every step is deliberately the simplest reasonable choice, because the point of a baseline is to show which of those choices break first.

The setup is the same as before. Generation uses llama3:latest (Llama 3 8B, Q4_0), embeddings use nomic-embed-text (137M, F16), both on Ollama 0.40, and storage is PostgreSQL 16.2 with pgvector 0.6.2 from the pgserver package. Everything ran on one Mac (Apple M3 Pro, 18 GB), shared with other work at the time.

Loading the site

The corpus is every markdown file under six folders of the site’s source: blog, kernel, gpu, design, dsa and essays. Each folder’s index.md (a landing page) is skipped, and so is this track. Each file has YAML front matter with a title, and some contain inline SVG charts.

SITE = "website/src/content"
TRACKS = ["blog", "kernel", "gpu", "design", "dsa", "essays"]
TAGS = re.compile(r"</?(figure|figcaption|sup|a|span|i|div|br|p|b|em|strong)\b[^>]*>")

def load_docs():
    docs = []
    for track in TRACKS:
        for path in sorted(glob.glob(f"{SITE}/{track}/*.md")):
            name = os.path.basename(path)[:-3]
            if name == "index":
                continue
            text = open(path, encoding="utf-8").read()
            title = name
            m = re.match(r"^---\n(.*?)\n---\n", text, re.S)
            if m:                                    # front matter: keep the title
                t = re.search(r'^title:\s*"?(.*?)"?\s*$', m.group(1), re.M)
                title = t.group(1) if t else name
                text = text[m.end():]
            text = re.sub(r"<svg\b.*?</svg>", "", text, flags=re.S)
            text = TAGS.sub("", text)
            docs.append({"url": f"/{track}/{name}", "title": title, "text": text})
    return docs

Removing the SVG matters more than it looks. The machine learning essay is 1,102,072 characters on disk, and 55,339 after the charts are removed. Without that step, about half the corpus would be <rect> and <circle> elements. Those would fill hundreds of passages with coordinates that mean nothing to a reader, and they’d turn up in searches for anything with numbers in it. The HTML stripping is narrower on purpose: only a list of known formatting tags is removed, because the kernel chapters contain things like <linux/bpf.h> and <stdio.h> that look like tags and are content.

The result is 98 documents and 2,179,166 characters: 31 from system design, 18 from data structures and algorithms, 15 essays, 14 kernel chapters, 13 blog chapters and 7 GPU chapters.

Cutting it into passages

Chapter 4 showed why whole documents can’t be embedded: one sentence barely moves the vector of a long text, and anything past 2,048 tokens is silently dropped. So each document is cut into chunks. The baseline uses about 800 characters per chunk with 100 characters of overlap, splitting on paragraph boundaries where it can:

def chunk_text(text, size=800, overlap=100):
    pars = []
    for p in re.split(r"\n\s*\n", text):            # paragraphs = blank-line separated
        p = p.strip()
        if p:
            pars.extend(split_long(p, size - overlap) if len(p) > size - overlap else [p])
    chunks, cur = [], ""
    for p in pars:
        if cur and len(cur) + 2 + len(p) > size:     # this paragraph would overflow
            chunks.append(cur)
            tail = cur[-overlap:]                    # carry the last ~100 characters
            sp = tail.find(" ")
            tail = tail[sp + 1:] if 0 <= sp < 40 else tail   # start on a word
            cur = tail + "\n\n" + p
        else:
            cur = f"{cur}\n\n{p}" if cur else p
    if cur:
        chunks.append(cur)
    return chunks

Paragraphs are packed into a chunk until the next one would push it past 800 characters. A paragraph that’s too long on its own (a big code block or table) is split on line breaks, then on sentence ends, and as a last resort at a fixed length. Each new chunk starts with the last 100 characters or so of the previous one, so a sentence that straddles the boundary appears whole in at least one of them.

Each chunk also records the nearest heading above where it starts. That gives the citation something a reader can use: not “passage 2,113” but “/design/22-rate-limiter — The token bucket”.

The corpus came out as 3,951 chunks. The median length is 678 characters, 90% are under 781, and the shortest are the ends of documents. By track: 1,851 from data structures and algorithms (the largest track by far, with long code listings), 971 from system design, 709 essays, 165 kernel, 147 GPU and 108 blog.

Here is one chunk from the observability chapter of the blog. Its first line is the overlap carried from the previous chunk, which ended mid-sentence:

programs are kernel objects with IDs, like everything else.

### Step 7: measure what looking costs

1. Write `your-syscall-loop`, a program that loops on `syscall(SYS_getppid)`.
2. Measure it with no tracer, then with a tracepoint attached:

```bash
   perf stat -r 5 -- /lo/out/your-syscall-loop
   ```

3. Publish both figures, with your kernel, CPU and VM type. **Don't quote a generic percentage.** The cost depends on the path you measure.

## If it fails

Already some rough edges are visible. The overlap starts mid-sentence. The chunk ends on a heading whose content is in the next chunk. And the code block has been damaged: in the source it is indented under the list item, but the splitter strips whitespace from the start of each paragraph, so the opening fence lost its indentation and the closing one kept it. Those are the price of a simple rule, and they come back at the end of the chapter.

Embedding and storing

Each chunk was embedded with the search_document: prefix, in batches of 32 per request to Ollama. That took 532 seconds for the 3,951 chunks, or 7.4 chunks a second. The rows went into the PostgreSQL table from chapter 5:

CREATE TABLE chunks (
    id        int PRIMARY KEY,
    track     text NOT NULL,
    url       text NOT NULL,
    section   text,
    body      text NOT NULL,
    embedding vector(768) NOT NULL
);

At 3,951 rows, chapter 5 measured an exact scan at under 5 ms. So the baseline has no vector index, and its retrieval is always exactly the five nearest chunks. An approximate index would add a second source of error, and there’s no need for one here.

Asking a question

Answering a question takes four steps: embed the question, retrieve the five nearest chunks, build a prompt, and generate.

def embed_query(q):
    r = requests.post(f"{OLLAMA}/api/embed", json={
        "model": "nomic-embed-text", "input": ["search_query: " + q]})
    return r.json()["embeddings"][0]

def retrieve(qvec, k=5):
    lit = "[" + ",".join(map(str, qvec)) + "]"
    return con.execute("""
        SELECT url, section, body, 1 - (embedding <=> %s::vector) AS sim
        FROM chunks ORDER BY embedding <=> %s::vector LIMIT %s""",
        (lit, lit, k)).fetchall()

def generate(prompt):
    r = requests.post(f"{OLLAMA}/api/generate", json={
        "model": "llama3:latest", "prompt": prompt, "stream": False,
        "options": {"temperature": 0, "num_ctx": 4096, "seed": 0,
                    "num_predict": 512}})
    return r.json()

Temperature 0 and a fixed seed make the runs repeatable: running the same question again gave the same answer, character for character, which chapter 1 explains. num_predict caps the answer at 512 tokens, and the reason it’s there is one of the failures at the end of this chapter. num_ctx is 4,096 tokens, Ollama’s default here, written out so it’s explicit. The prompts here run to about 1,000 tokens (five chunks of about 200 tokens each, plus instructions), so that leaves plenty of room for the answer.

The prompt

The prompt has three jobs. It has to hand the model the passages in a form it can refer to, tell it to answer from them and not from memory, and tell it how to cite. Each passage is numbered and labeled with its page and section:

def build_prompt(question, rows):
    passages = "\n\n".join(f"[{i}] ({url} — {section})\n{body}"
                           for i, (url, section, body, _) in enumerate(rows, 1))
    return PROMPT.format(passages=passages, question=question)

The first version of the instructions:

Answer the question using only the passages below. Cite the passages you use with
their numbers in square brackets, like [2]. If the passages do not contain the answer,
say "The passages don't say." and stop.

{passages}

Question: {question}
Answer:

The last sentence is there because of chapter 3: an assistant that doesn’t know should say so instead of inventing something. It turned out to be the most consequential line in the prompt.

On the first questions, the pipeline answered a few things well and refused almost everything else. Of sixteen hand-written questions, this prompt answered four and replied “The passages don’t say.” to twelve. Several of those refusals were wrong. For “Which chapters on this site cover consistent hashing?”, the top passage was the section titled “Consistent hashing” in the partitioning chapter. For “Why does Kafka use a consumer group?”, the fourth passage, from the queues chapter, says what a consumer group is for. The model had the answer in front of it and declined.

The likely cause is the escape hatch. The prompt offers a fixed, safe sentence and says to stop after it, and the model reaches for it whenever the passages don’t answer the question head-on. A second version says the same things with a different balance:

Use the numbered passages below to answer the question. Base the answer only on what the
passages say, and cite the passage number in square brackets after each claim, like [2]. If
the passages only partly answer the question, give the part they support and say what is
missing. If they contain nothing relevant, say so.

{passages}

Question: {question}
Answer:

It still restricts the model to the passages, still asks for citations (now after each claim), and still allows “nothing relevant”. But it asks for a partial answer before a refusal. Both versions ran on the same retrieved passages for two sets of questions: the sixteen hand-written ones, and 24 of the questions llama3 wrote from random passages in chapter 4. For those 24, the source passage is known, so it’s possible to tell when a refusal was wrong. The source passage was in the top five for 20 of them.

A “refusal” here means the first prompt’s exact sentence, or an answer from the second that said the passages don’t contain the answer and gave nothing in its place. The second prompt’s three refusals on the hand-written set were all correct: two questions whose passage retrieval never found, and one about a pgvector default that this site doesn’t mention. The 8 generated questions that the first prompt refused include the second prompt’s 2 refusals. On the other 6, the second prompt gave a substantive answer, including the stream–table join, merging intervals and what read() returns when a signal interrupts it.

The sample is small, and fewer refusals isn’t the same as more correct answers. I didn’t grade every answer for accuracy, and chapter 8 builds the tools for that. But the difference is too large to be noise: refusals on questions whose answer was sitting in the prompt fell from 8 in 20 to 2 in 20. The second prompt is the baseline from here on. Its median answer was 174 tokens, against 7 for a refusal. That’s the cost: answering takes longer than declining, and as the failures below show, it also gives the model more room to go wrong.

Answers that work

With the second prompt, many answers are exactly what the assistant should produce. These are verbatim apart from line breaks, with the sources the citations point to.

“What is the difference between a semaphore and a mutex?”

According to the passages, the main difference between a semaphore and a mutex is that a
mutex is a lock that allows one holder at a time, who then has to release it [1], whereas a
semaphore is a counter that controls how many threads may proceed, and any thread can raise
it [2]. A mutex is used to protect data, whereas a semaphore is used to limit concurrency
and signal between threads [2].

All five passages came from the semaphore essay, with [1] in “Semaphore vs mutex vs lock” and [2] in “The core difference”. The two definitions are word for word from the essay, and the ownership point that chapter 3’s answers missed is there: one holder, who has to release it, against any thread can raise it.

“How does the token bucket in the rate limiter chapter refill tokens?”

All five passages came from the rate limiter chapter: three from “The token bucket” and two from “The distributed race”. The answer (abridged here) cites three of them and quotes the code correctly:

According to [1], the token bucket refills tokens lazily when a request arrives. The refill
calculation is:

```text
tokens = min(capacity, tokens + elapsed * refill_rate)
```
...
In [3], the refill calculation is:

```lua
local elapsed = math.max(0, now - last_refill)
tokens = math.min(capacity, tokens + (elapsed * refill_rate))
last_refill = now
```
...
So, the token bucket in the rate limiter chapter refills tokens lazily when a request
arrives, based on the elapsed time since the last refill and the refill rate.

Passage [3] is the Lua script that runs inside Redis in the distributed version. Finding the same formula in the Python and the Lua is exactly the kind of connection a reader would want pointed out.

“What did the SynapseOS benchmark measure, and what were the results?”

According to the passages, the SynapseOS benchmark measured the performance of the
detonation chamber in detecting malicious samples. The results are as follows:

* The chamber flagged 65 of 73 malicious samples [1]
* picklescan flagged 64 of 73 malicious samples [1]
* Between them, the two tools flagged all 73 malicious samples [1]
* Neither produced a false positive on 24 clean scans [1]
* The run exposed three real bugs in SynapseOS, one of which affected a verdict in the results [1]

Every number matches the benchmark essay’s summary, and all are attributed to passage [1], which is that summary. Chapter 3 asked the model about the same benchmark without any passages and got an invented comparison against “the Apache Spark cluster”. The model is the same. What changed is what it was given to read.

“What is coordinated omission?”

This question is in the reader’s vocabulary, and retrieval only half worked: four of the five passages were about Kafka offsets, rebalancing and delivery guarantees, which have nothing to do with it. The fifth, from the queueing chapter, has the answer, and the model found it there:

The passages do not directly answer the question "What is coordinated omission?" However,
they do contain a mention of coordinated omission in passage [5]
(/design/10-queueing-and-overload — Other defences):

"Latency measurements often have a subtle flaw. A load generator that waits for each
response before sending the next one stops sending during a stall. ..."

This passage describes coordinated omission as a flaw in latency measurements, where a load
generator stops sending requests during a stall, resulting in an artificially good-looking
tail.

The quotation is exact. The opening hedge is odd, because the passage contains a heading that says “Coordinated omission”. Its label says “Other defences” because that is the heading in force where the chunk starts, which is a small labeling flaw that comes back below. Even so, a reader gets the right explanation and a pointer to where it lives.

Turning numbers into links

The model cites by number, and the numbers only mean something next to the prompt. Turning them into links is plain string handling:

def sources(answer, rows):
    cited = sorted({int(n) for n in re.findall(r"\[(\d+)\]", answer)})
    return [(n, rows[n-1][0], rows[n-1][1]) for n in cited if 1 <= n <= len(rows)]

The page and section of each cited passage become a link to the page, such as /design/22-rate-limiter, with the section name (“The token bucket”) beside it. That makes every answer checkable by a reader in one click. It’s also the cheapest defense against the failures below, because a reader can see when a citation doesn’t support the sentence it’s attached to.

Where the time goes

Seven questions were timed end to end, holding the shared timing lock. The run used the first prompt, so two of the answers are seven-token refusals; the second prompt is a few dozen tokens longer, which shouldn’t change the picture. Ollama reports its own counters with every response: how many prompt tokens it processed and how long that took, and the same for the generated tokens. Those counters measure the model’s work. The wall-clock time around each call measures what a user would wait.

The starred rows had been asked once before with the identical prompt, and Ollama reused the processed prompt from its cache instead of reading it again. The two fresh prompts show the real cost: about 280 tokens a second to read the prompt, so 3.2 to 3.5 seconds for five passages. Generation runs at a steady 19 to 20 tokens a second, so the length of the answer decides the rest. A seven-token refusal takes a third of a second, and a 150-token explanation takes seven and a half.

Retrieval is a rounding error next to that. The SQL query took 12 to 27 ms after the first one, including sending 768 numbers as text and returning five passages. That’s more than the 4.6 ms exact scan in chapter 5, because this query also returns the passage text. Embedding the question took between 0.35 and 2.7 seconds of wall-clock time. That spread comes from the shared machine: Ollama queues requests from every process using it, and other work was running. Generation’s wall-clock time (4.8 to 24.5 seconds) was inflated for the same reason, well beyond the model’s own counters. On an idle machine, a typical answer here costs roughly 3.5 seconds to read the passages plus 0.05 seconds per answer token. Chapter 13 comes back to latency and what can be done about it.

Where it fails

The answers above are the good half. The sixteen hand-written questions were chosen to cover the kinds of things a reader of this site might ask, including some that seemed likely to be hard, and the hard ones failed in ways that are worth recording precisely. Each failure below is a verbatim output under the final prompt unless it says otherwise. They fall into four groups, by which step broke.

Retrieval never finds the passage

“What does PFCOUNT do?” The site mentions Redis’s PFCOUNT once, in the hashing chapter, in a passage about HyperLogLog. The five passages retrieved came from the SynapseOS essay, the observability chapter and three from the one-machine chapter of system design. Measured against all 3,951 passages, the one that contains PFCOUNT ranks 1,946th. The model did the right thing with what it got:

There is no mention of PFCOUNT in the provided passages.

“How does RefCell work in Rust?” is the same: the passages were about a URL shortener, the open() path, JWT refresh tokens, RDMA flow control and RCU, and the answer said they don’t mention RefCell. The Rust essay does.

These are chapter 4’s identifier blind spot, showing up in the product. From the user’s side, these answers are misleading in their own way: the assistant says the site doesn’t cover something that it does.

The chunks cut things in the wrong place

“When is a partial index useful in PostgreSQL?” Retrieval found the right place in the PostgreSQL indexes essay. The third passage ends with the heading ### Partial indexes and its CREATE INDEX ... WHERE status = 'pending' example. But the sentence that answers the question (“A partial index covers only rows matching a condition. If 1% of orders are pending, the index is about 1% of the size…”) is the first line of the next chunk, which wasn’t retrieved. The answer:

Based on the passages, a partial index is useful in PostgreSQL when you want to index a
specific subset of rows, as described in [4]. This is useful when you have a large table and
only a portion of the rows need to be searched frequently.

The second sentence is the model’s own general knowledge, and it’s roughly right. But the citation is wrong: passage [4] is the essay’s section on extension index types, which says nothing about partial indexes. A reader who clicks the citation finds no support for the claim.

The splitter also damages text in ways that don’t show up in any one answer. 977 of the 3,951 chunks (25%) contain an odd number of code fences, which means a code block was cut across two chunks. 318 contain markdown table rows but no header row, so their columns have no names. And because each paragraph is stripped of leading whitespace, the first line after any blank line inside a code block loses its indentation. In the rate limiter chunk above, def _get_user_lock(self, key) sits at column 0 inside a class, which in Python is a different program.

The section labels used for citations have a related flaw. Each chunk is labeled with the heading in force where its new text starts. The passage that defines false sharing begins with the last two sentences of the memory-model section and then switches to “Cache coherence and false sharing”, so it’s labeled “Memory models”. The coordinated omission passage above was labeled “Other defences” for the same reason. Both citations point to the right page but the wrong section.

Questions about the site, not its content

“How many chapters does the system design track have?” The answer is 31. No passage says so: the track’s landing page is excluded from the corpus, and the number appears nowhere in the chapters. The retrieved passages were from chapters 1, 19 and 23, plus the kernel lab setup. The answer:

According to [2], the system design track has chapters 24 to 30, which are worked designs,
and chapter 31, which collects reference tables.

Every fact in it is true and cited, and it doesn’t answer the question. A reader would conclude that the track has eight chapters. The first prompt’s “The passages don’t say.” was the better answer here.

“Which chapters on this site cover consistent hashing?” Retrieval was good. The first passage was the “Consistent hashing” section of the partitioning chapter, and the fifth was the same section of the hashing chapter. The page of each passage is printed in its label. The model never named either. It explained consistent hashing, then started describing each passage in turn (“The passage [3] does not provide direct information on consistent hashing…”, “The passage [4] does not provide any information on consistent hashing.”), and then repeated that block of five paragraphs over and over.

The first time this question ran, the request had no output limit. The log of the shared Ollama server showed it still generating at 16,153 tokens, about 21 minutes in and well past the 4,096-token context (Ollama keeps going by discarding the oldest half of the context). It only stopped when the client was killed. Every other request on the machine waited behind it. The pipeline now sets num_predict to 512, so the same question stops after 512 tokens of repetition. That’s a cap, not a fix: the answer is still useless.

Questions about the structure of the site, such as how many chapters, which ones, or what to read first, need information that lives in titles, folder names and landing pages, not in passage text. “What should I read first on this site if I’m new to operating systems?” got a confident recommendation of the boot chapter, halfway through the blog track, because that’s what retrieval happened to return.

The model goes beyond the passages

“Compare Kafka and Redis as a job queue.” Four passages were about Kafka and one was Redis’s “The short answer” on why it’s fast. The answer summarizes both, then adds:

Redis, however, is not designed as a job queue. It's primarily an in-memory data store
that's optimized for fast data retrieval and manipulation. While it can be used as a message
broker, it's not designed for the same level of scalability, durability, and exactly-once
processing as Kafka.

None of those sentences carries a citation, and none of them comes from the passages. They come from the model’s training, and the prompt’s “only what the passages say” didn’t stop them. The answer also lists Kafka’s follower fetching, which lets consumers read from a nearby replica, as a job-queue feature.

“Is the cache invalidated on writes in the distributed cache design?” All five passages were relevant, and they describe a subtle position: invalidation is driven by deletes, deletes can be lost, and a TTL bounds the damage. The answer cites four of them and ends:

Overall, the answer is that the cache is not always invalidated on writes, but it is
invalidated on writes in the sense that the database's change log drives the invalidation
process. The TTL and leases mechanisms help to ensure that the cache remains consistent with
the database.

Along the way it says the stale-set race “is fixed by using leases, which ensures that the cache is invalidated on writes”. In the caching chapter, leases refuse a late fill after the key has been deleted, so a stale value isn’t written back. They don’t do the invalidating. The answer contradicts itself, and the attribution to [4] is a misreading. It’s exactly the kind of question chapter 4 warned about: passages that all discuss invalidation, saying different things about it.

“What does the verifier track in bpf_reg_state?” The answer lists the type, the tnum and the bounds correctly, from the verifier chapter. Then it adds “Memory accesses are checked against the pointer’s type and bounds” and “Helper calls are checked against their prototypes” to the list. Both sentences are in the passage, but they describe what the verifier checks, not what bpf_reg_state holds. Every claim is cited, and the answer is still subtly wrong.

Where this leaves things

The assistant works. It reads 98 files, cuts them into 3,951 passages, stores them with their vectors in PostgreSQL, and answers a question in a few seconds with citations a reader can follow. On questions whose answer sits in one passage that retrieval finds, it’s accurate and quotes the site faithfully, including on topics where chapter 3’s bare model invented everything. One prompt sentence turned out to matter more than anything else here: an instruction to give up when unsure made it give up on questions it could answer.

The failures are specific, and each one points at a step:

Some of these can be fixed where they happen: different chunking, keyword search alongside vectors, reranking the retrieved passages, rewriting the question before searching, and filters on track or page. Chapter 7 takes the failures one at a time and tries the fix that each one suggests.