4 — Meaning as numbers

Chapter 3 ended with two facts that pull against each other. The model doesn’t know this site’s chapters, so it has to be shown them. And it can’t be shown all of them. After stripping front matter and the inline SVG charts, the six tracks the assistant will answer from (blog, kernel, GPU, system design, data structures and essays) come to 98 files and about 2.2 million characters. llama3 has a context window of 8,192 tokens. Even a generous prompt holds well under one percent of the corpus.

So the assistant has to pick. For each question it needs the handful of passages that bear on it, and nothing else. The obvious way to pick is by shared words: find passages that contain the question’s terms. That works for some questions and fails for others, and chapter 7 comes back to it. The question “how does a cache decide what to throw away when it is full?” shares almost no words with a good answer: “LRU removes the least recently used entry once capacity is reached.” A reader sees at once that they’re about the same thing. A word match doesn’t.

This chapter covers the other way to pick: turn every passage and every question into a list of numbers, arranged so that texts with similar meaning get similar numbers, and then compare numbers instead of words. It also measures, on this site’s text, where that idea holds up and where it quietly breaks.

All runs in this chapter used Ollama 0.40 on one Mac (Apple M3 Pro, 18 GB), with nomic-embed-text for embeddings (137M parameters, stored as F16) and the corpus described above.

An embedding is a fixed-length vector

An embedding model takes a piece of text and returns a vector: a fixed number of floating-point values, whatever the length of the input. nomic-embed-text returns 768. Ollama serves it on the same local server as llama3, through /api/embed:

import time
import numpy as np, requests

def embed(texts, prefix=""):
    r = requests.post("http://localhost:11434/api/embed", json={
        "model": "nomic-embed-text",
        "input": [prefix + t for t in texts]})
    r.raise_for_status()
    return np.array(r.json()["embeddings"], dtype=np.float32)

# chunks: the site's passages, cut as described in chapter 6
t = time.time()
E = embed([c["text"] for c in chunks], prefix="search_document: ")
dt = time.time() - t
n = np.linalg.norm(E, axis=1)
print(f"{len(chunks)} chunks, {E.shape}, {dt:.1f}s, {len(chunks)/dt:.1f} chunks/s, "
      f"norms {n.min():.4f}-{n.max():.4f}")

3951 chunks, (3951, 768), 532.0s, 7.4 chunks/s, norms 1.0000-1.0000

The input can be a list, so many passages can be embedded in one request. (The run above sent them 32 at a time; the batching loop is left out here.) Each of the 3,951 passages of the corpus came back as 768 numbers with a length (L2 norm) of 1: Ollama returns unit-length vectors. That detail matters in a moment.

None of the 768 numbers means anything on its own. There’s no “caching” dimension or “kernel” dimension. The model was trained so that the geometry carries meaning: texts that should be retrieved for the same questions end up pointing in similar directions. Under the hood, nomic-embed-text is a BERT-style encoder. It reads the whole input at once, produces one vector per token, and averages them into a single vector (the model card’s reference code uses mean pooling followed by a layer norm). That averaging step explains several of the failures measured later.

The prefixes are part of the input

The search_document: string in the code above isn’t decoration. The model card for nomic-embed-text-v1.5 says the text “must include a task instruction prefix”. For retrieval, passages get search_document: and questions get search_query: . Other tasks have their own prefixes (clustering: , classification: ). The model was trained with these markers, so the same text gets a slightly different vector depending on the role it’s given, and questions and passages are meant to be marked differently. Ollama’s nomic-embed-text:latest is the v1.5 build (the library page lists latest and v1.5 with the same digest as the copy installed here), so the card applies.

Does it matter in practice? To check, llama3 wrote one question for each of 100 randomly chosen passages of the corpus (for example, “How do I efficiently merge overlapping intervals?” for a passage of the arrays chapter). Each question was then searched against all 3,951 passages, and the measurement is where its source passage ranked. That’s done four ways: with and without each prefix. The whole corpus was embedded a second time without the document prefix for this.

(Counts out of 100 questions.)

Using both prefixes as the card says is best or tied on every column, and dropping both costs 7 of 100 questions in the top 5. With a sample of 100, the differences between the other three rows are within noise. These questions are also easy: they were written from the passage they’re looking for and often reuse its words. One number went the other way: the mean rank was 13.2 with both prefixes against 10.0 with none, so the few questions that missed badly missed by more. The median rank was 1 with both prefixes and 2 with none. The rest of the track uses both prefixes.

Comparing two vectors: cosine similarity

With every text turned into a vector, “how related are these two texts” becomes “how close are these two vectors”. The usual measure is the cosine of the angle between them:

It’s 1 when the vectors point the same way, 0 when they’re at right angles and −1 when they point in opposite directions. It ignores length, which is the point: a long passage and a short one about the same thing should match, even if one vector came out “bigger”.

Because Ollama already returns vectors of length 1, the denominator is 1 and cosine similarity is just the dot product. That turns out to be convenient for speed. Comparing one question against every passage becomes one matrix–vector multiplication, which is the subject of chapter 5. Cosine distance, which PostgreSQL’s pgvector uses, is one minus this number.

Here is the opening question scored against six sentences. The question used the query prefix and the sentences used the document prefix:

def cos(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

q = embed(["How does a cache decide what to throw away when it is full?"],
          "search_query: ")[0]
for d in docs:
    print(f"{cos(q, embed([d], 'search_document: ')[0]):.3f}  {d}")

This is a humbling first result, and worth reading slowly. The paraphrase that shares no technical vocabulary with the question (“the entry nobody has touched for the longest time is dropped first”) does well. That’s the thing embeddings are for. But the top score goes to a sentence that is about caches in general and doesn’t answer the question at all. And a sentence about a dog and a bowl, which shares the words “throw” and “full”, beats the textbook answer that uses the term “LRU”.

Two lessons come out of this. First, on short sentences, embeddings respond strongly to topic and to surface words. “Cache” pulls toward cache, and “throw … full” pulls toward throwing and fullness. Second, the gaps are small. The whole range from the best match to an unrelated sentence about full-text search is 0.23. The second lesson holds for full passages too, as the next section shows.

The scale is not zero to one

It’s tempting to treat cosine similarity as a confidence score, with something like “0.7 means relevant”. On this corpus that reading is wrong. Taking 20,000 random pairs of passages from the 3,951 in the corpus, most of which have nothing to do with each other:

rng = np.random.default_rng(0)
i, j = rng.integers(0, len(E), 20000), rng.integers(0, len(E), 20000)
s = np.sum(E[i] * E[j], axis=1)[i != j]

Two passages picked at random, which nearly always means two unrelated topics, typically score 0.65. Nothing in this corpus comes close to 0 or below. The vectors all sit in a narrow cone of the 768-dimensional space, which is common for embedding models. The useful information is in the ranking, not the absolute number. A passage at 0.70 can be the best match for one question and noise for another. Any rule like “drop results below 0.7” has to be tuned on real questions for this model and this corpus, and it breaks when either changes. The pipeline in chapter 6 takes the top five by rank and uses no threshold.

What it gets right on the real corpus

On real passages and real questions, it does what it promises more often than the sentence test suggests. These three questions were written to avoid the words the site itself uses. Each was embedded with the query prefix and compared against all 3,951 passages:

def search(q, k=3):
    v = embed([q], "search_query: ")[0]
    s = E @ v                       # E: all passage vectors, unit length
    top = np.argsort(-s)[:k]
    return [(float(s[i]), chunks[i]) for i in top]

“How do I stop one noisy customer from overwhelming my API?”

The question never says “rate limit” or “fairness”, and the top two are exactly the right places. The third is a weak match.

“Why does my program get slower when two threads touch neighbouring variables?”

The answer is false sharing, and the passage that explains it is second, 0.005 behind a passage about which work Redis runs on other threads. The right passage is found, but the margin is tiny.

“What happens to in-flight work when a machine loses power?”

All three are relevant, and the first is a section written about exactly this.

This is the core promise of embeddings, and here it holds up. Questions phrased in a reader’s words find passages written in the author’s words. “Noisy customer” and “API Priority and Fairness” share no words at all, so a keyword search had nothing to match on.

What it misses

An embedding squeezes a passage into 768 numbers by averaging. Whatever doesn’t move the average much doesn’t survive. Small words that flip the meaning of a sentence are the first casualty. Each pair below was embedded with the document prefix, and the number is the cosine between the two sentences of the pair.

Negation.

Opposites and reversed order.

Numbers.

Paraphrases and unrelated pairs, for scale.

Put the tables side by side and the problem is plain. A sentence and its negation (0.98) are far “closer” than a sentence and a correct paraphrase of it (0.74). A timeout of 30 seconds is closer to a timeout of 300 seconds (0.995) than to “half a minute” (0.979), which means the same thing. To this model, “1,000 requests per second” and “1,000,000 requests per second” are nearly the same sentence.

None of this is a defect of this particular model so much as a consequence of the design. Mean pooling over tokens gives “not” one vote among a dozen, and a number is a few word pieces whose value the model never learned to compare. The consequence for retrieval is specific. Embeddings are good at finding the passage about the topic. They are bad at telling apart passages on the same topic that say different things. When the corpus has one passage saying the cache is invalidated and another saying it isn’t, both come back for either question, in an order close to random. Getting the answer right then falls to the language model reading the passages, which is one reason retrieval and generation are separate steps.

Exact identifiers

This site is full of exact names: kernel functions, configuration flags, Redis commands, Rust types. Readers will ask about them by name, and a name either matches or it doesn’t. Embeddings blur names the same way they blur numbers:

To see what that means on the real corpus, I collected identifiers from the site’s inline code (strings in backticks with an underscore or internal capital, like vfs_open or RefCell) and kept those that appear in exactly one of the 3,951 passages. For a random sample of 40, the query was simply “What is X?”, and the measurement is where the one passage that contains X ranked among all 3,951:

for ident in pick:
    target = passage_containing[ident]          # the only passage with this string
    v = embed([f"What is {ident}?"], "search_query: ")[0]
    order = np.argsort(-(E @ v))
    rank = int(np.where(order == target)[0][0]) + 1

Some of the misses:

The hits are mostly long, descriptive kernel names such as page_cache_sync_ra, security_hook_heads and sock_write_iter, whose word pieces say what they are about. The misses are names whose pieces don’t. The model’s tokenizer is BERT’s uncased word-piece tokenizer (per the model card), so PFCOUNT is lowercased and split into word pieces, and nothing in those pieces points to HyperLogLog. The passage that mentions it is really about counting distinct elements, and the vector reflects that. A question that names it lands nowhere near.

So with plain embedding search, a question that names an exact identifier finds its passage in the top 5 about five times in eight on this sample (25 of 40), and sometimes doesn’t find it at all. For text search over a site like this, that’s a serious gap. A plain grep would get all 40. Combining the two kinds of search is one of the fixes in chapter 7.

Length dilutes, and the end gets cut off

The averaging inside the model has a second consequence: the more text goes into one vector, the less any single sentence counts. To measure it, I took the start of the Kafka essay at increasing lengths and added one unrelated sentence (“This section is entirely about baking sourdough bread at home: flour, water, starter, salt.”) at the end, and separately at the start. The comparison is between the original vector and the vector with the bread sentence added:

bread = embed([extra])[0]
for n in [300, 1000, 4000, 8000, len(doc)]:
    a = embed([doc[:n]])[0]
    end = embed([doc[:n] + "\n\n" + extra])[0]
    start = embed([extra + doc[:n]])[0]
    print(n, a @ end, bread @ end, a @ start, bread @ start)

(The token counts are the prompt_eval_count that Ollama reports for each request. All texts used the document prefix.)

Three things show up. In a 300-character passage, one off-topic sentence moves the vector noticeably. In a 4,000-character passage it barely registers. A passage that mentions a topic once, among a lot of other material, is unlikely to be found by a question about that topic.

Second, position matters. The same sentence moves the vector more at the start of a text than at the end. The runs are deterministic, and the effect holds at every length measured.

Third, at 8,000 characters the input reaches 2,048 tokens, which is the context length of this model as Ollama serves it. Beyond that point, appending text changes nothing: the cosine is 1.0000, and the full 14,549-character essay gives the same numbers in every column as its first 8,000 characters. Ollama’s /api/embed has a truncate option that defaults to true, which cuts the end of over-long inputs instead of returning an error (API docs). The truncation is silent. Embed whole chapters and their second halves disappear from search without a warning.

That is the case for chunking: splitting documents into passages short enough that each vector represents one idea, and well under the model’s limit. The pipeline in chapter 6 uses chunks of about 800 characters, which is roughly 200 tokens, and chapter 7 measures what happens when that size changes.

The cost of turning the site into vectors

With chunks of about 800 characters and 100 characters of overlap (chapter 6 covers the splitting), the corpus becomes 3,951 passages. Embedding all of them in batches of 32 took 532 seconds, or 7.4 passages a second. The Mac was shared with other work at the time, so treat that as a rough figure. It’s paid once for the corpus, and again for each passage that changes.

The result is a 3,951 × 768 matrix of 32-bit floats: 12.1 MB. That is small enough to hold in memory many times over, and searching it is a single matrix–vector multiplication.

Where this leaves things

There is now a way to compare a question with a passage by meaning rather than by shared words, and on real questions it finds the right part of the site more often than not, including when the question uses none of the site’s vocabulary. It also has measured blind spots. It barely distinguishes a statement from its negation, or 30 seconds from 300. It finds a passage by an exact identifier only about five times in eight. Its similarity numbers only mean something as a ranking. And it silently ignores text beyond 2,048 tokens, so documents have to be cut into passages first.

All of that can be worked on. The immediate problem is simpler. Search so far has meant comparing the question with every passage, one by one. At 3,951 passages that’s quick. A larger site, or a company’s documents, could have millions, and checking every vector for every question stops being free. Chapter 5 measures how fast that brute-force search really is, and builds the index structures that avoid it.