2 — Answers a program can use

Chapter 1 ended with a model whose answers change every time, in wording, in length and sometimes in substance. A person can read past that. A program can’t. The assistant this track builds is a program, and before it answers anything it has small jobs to do that need a definite result.

The first of those jobs is routing. This site has six groups of writing: the Linux, Observed track, the Kernel, Traced track, the GPU track, the System Design track, the Data Structures and Algorithms track, and a set of standalone essays. When a question comes in, knowing which group it belongs to is useful later, when the assistant searches the site and can narrow the search (that’s where chapter 7 puts it to work). So the task is a classification: given a question, return one of six labels.

That needs output a program can read. The obvious way to get it is to ask for JSON. This chapter measures what happens when you do, and what each fix actually buys.

The model is still llama3:latest (Llama 3 8B, 4-bit Q4_0) on Ollama 0.40.11, on an Apple M3 Pro with 18 GB. Nothing in this chapter is a timing, so the numbers are counts, and the samples are small: 30 labeled questions, five per track, written to be unambiguous.

Asking nicely

The prompt describes the six tracks in a line each, then asks for an object with two fields:

TRACKS = {
  "linux-observed": "building a small Linux system from source: toolchain, libc, init, booting, ...",
  "kernel-traced": "following Linux kernel code paths: system calls, open(), fork/exec, the scheduler, ...",
  "gpu": "GPU hardware and CUDA: the silicon, the execution model, warps, eBPF on GPUs, ...",
  "system-design": "designing distributed systems: scaling, databases, caching, queues, ...",
  "dsa": "data structures and algorithms: complexity, searching, sorting, trees, graphs, ...",
  "essays": "standalone essays: Kafka, Redis, PostgreSQL indexes, TLS, JWT and OAuth, Kubernetes, ...",
}
desc = "\n".join(f"- {k}: {v}" for k, v in TRACKS.items())
PROMPT = f"""Classify the question into one of these tracks of a website:
{desc}

Return a JSON object with two fields: "track" (one of the track names above) and "confidence" (a number from 0 to 1).

Question: {{q}}"""

The questions are things like “How does the kernel walk a path inside open()?” (kernel-traced), “When should I add a cache in front of my database?” (system-design) and “Why is Redis fast with a single thread?” (essays). Each went to the chat endpoint once, at the default settings, and the reply went straight into json.loads.

None of the 30 parsed.

The model understood the task. It just wrapped the answer the way it wraps everything, as a reply to a person:

Here is the classification result:

```
{
  "track": "kernel-traced",
  "confidence": 0.9
}
```

The confidence score is 0.9 because the question specifically asks about the Linux scheduler, which is ...

Twenty of the 30 replies put the JSON inside a Markdown code fence, and the other ten put it between a sentence of introduction and a paragraph of explanation. Every one started with “Here is”. The replies averaged 81 tokens, for an answer whose useful part is about 15. That’s the cost of chapter 1’s lesson: the model was fine-tuned to produce helpful chat replies, and a bare JSON object doesn’t look like one.

Digging the JSON out

The first instinct is to repair it on the client side: find the first { and the matching } and parse what’s between.

def lenient(raw):
    m = re.search(r"\{.*?\}", raw, re.S)
    if not m:
        return None
    try:
        return json.loads(m.group(0))
    except json.JSONDecodeError:
        return None

On these 30 replies it works: all 30 parse, and 29 of the 30 labels are correct. But it works because the objects here are flat. The non-greedy \{.*?\} stops at the first closing brace, so any nested object breaks it, and a reply that mentions braces in its explanation, or shows two candidate objects, would hand back the wrong one. Scraping JSON out of prose is a parser for a format nobody specified. It fails on the replies nobody tested.

Asking more firmly

The next instinct is to say it louder. One line added before the question:

Respond with only the JSON object. Do not add any other text, explanation or code fences.

That one sentence changed everything. In a fresh run of the 30 questions, all 30 replies were a bare JSON object that json.loads accepted, with valid fields, averaging 17 tokens. 29 of the 30 labels were correct (the miss was Redis, filed under system-design). A second run of the same prompt, done as part of the retry experiment further down, gave the same picture: 30 of 30 valid on the first try, 28 correct.

So the instruction works for this model on this task. It still isn’t a guarantee: the model follows it because following it is likely, not because anything forces it to, and a longer prompt, a different question or a different model can tip it back into chat mode. What’s needed is a way to make anything other than valid JSON impossible.

Making the wrong tokens impossible

Ollama’s format field does that. Set to "json", it constrains generation so the output is always a well-formed JSON object (Ollama API: generate).

d = requests.post(URL + "/api/chat", json={
    "model": "llama3", "stream": False, "format": "json",
    "messages": [{"role": "user", "content": PROMPT.format(q=q)}]}).json()

The mechanism follows from chapter 1. At every step the model produces a probability for each of its 128,256 tokens, and the sampler draws one. Constrained decoding adds a step in between: a small parser tracks where the output is in the JSON grammar, and every token that can’t legally come next gets its probability set to zero before the draw. At the very start, only tokens that begin an object are allowed, so “Here” can never be picked. Inside a string, almost anything is allowed. After a complete value, only a comma or a closing brace. The model still chooses among the allowed tokens by its own probabilities, so the content is still the model’s. Only the shape is enforced.

With format: "json", all 30 replies parsed with plain json.loads, and all 30 had the two expected fields with sensible values:

{"track": "linux-observed", "confidence": 0.9}

The replies averaged 18 tokens instead of 81, which matters on a machine generating 17 tokens a second. 27 of the 30 labels were correct. The three misses were all questions labeled essays: Kafka’s partitions and Redis’s single thread were sent to other tracks, and the eBPF-and-CUDA question went to kernel-traced instead of gpu. That’s within a few questions of the earlier runs, and on one run each of 30 questions, differences of one or two aren’t meaningful. The format didn’t make the model better or worse at the task. It removed the wrapping.

The Ollama docs add one warning: in JSON mode, the prompt itself should also ask for JSON, or the model may generate large amounts of whitespace. That makes sense from the mechanism. If the model’s own preference is to start with “Here is”, and every token of that is forbidden, the most likely legal tokens may be spaces and newlines, which JSON allows between values.

A schema instead of “some JSON”

Valid JSON is not the same as the right JSON. format: "json" would happily accept {"category": "kernel"} or {"track": "operating-systems"}. The program needs a track that is one of six strings and a confidence that is a number. Ollama also accepts a JSON Schema in format (Structured outputs), and then the constraint is the schema, not just the grammar of JSON.

SCHEMA = {
    "type": "object",
    "properties": {
        "track": {"type": "string", "enum": list(TRACKS)},
        "confidence": {"type": "number"},
    },
    "required": ["track", "confidence"],
}
d = requests.post(URL + "/api/chat", json={
    "model": "llama3", "stream": False, "format": SCHEMA,
    "messages": [{"role": "user", "content": PROMPT.format(q=q)}]}).json()

The enum is the important part. Once the output reaches the value of track, the only tokens allowed are ones that continue one of the six names.

On the 30 questions, the schema run looked like the JSON-mode run: 30 of 30 valid, 27 of 30 correct, and once again all three misses were essays questions (this time Kafka, Redis and the PostgreSQL index question). The difference shows up on the questions the label set doesn’t cover. Five off-topic questions went through three setups at temperature 0:

Each column shows a different failure, or a fix.

In JSON mode, the model invented a seventh label, "none", for three of the five. It’s a sensible answer and a program bug: code that switches on six track names now has a value it doesn’t handle.

With the six-label schema, the model can’t invent anything, so it was forced to pick one of the six. It put sourdough bread and the World Cup in essays with a confidence of 0.8. The output was perfectly valid and completely wrong, and nothing in it says so. A schema guarantees the shape of an answer, never its truth, and an enum with no way out turns “I don’t know” into a confident guess.

Adding "none" to the enum, and one phrase to the prompt saying when to use it, fixed all five. The general rule is that a closed label set needs an explicit way to say “none of these”, or the constraint will manufacture an answer.

The confidence number means little

Every reply so far carried a confidence, and it’s tempting to use it, for example to ask a person when the model is unsure. Across the 30 questions in JSON mode the model only ever said 0.6, 0.8, 0.9 or 1. In the plain-prompt run it said 0.8 or 0.9, every time. The values don’t track anything. The model writes a confidence the way it writes any other text: as the kind of number that usually appears there.

The model does have a real measure of how sure it is, which is the probability it gave the label’s first token. That comes from logprobs, as in chapter 1. Each track name starts with a different token (linux, kernel, gpu, system, dsa, essays), so the probability of the first token of the track value is the model’s probability for that label, given the schema.

d = requests.post(URL + "/api/chat", json={
    "model": "llama3", "stream": False, "format": SCHEMA,
    "logprobs": True, "top_logprobs": 5, "options": {"temperature": 0},
    "messages": [{"role": "user", "content": PROMPT.format(q=q)}]}).json()
# find the token where the value of "track" starts, then read its probability

Run over the 30 questions at temperature 0, with the schema, the two measures look nothing alike:

The three wrong answers were again essays questions. Kafka went to system-design with probability 0.57, and essays was the runner-up at 0.40. Redis went to dsa with 0.41, ahead of system-design at 0.18. The PostgreSQL index question went to system-design with 0.55, with essays second at 0.39. In each case the model wrote “confidence”: 0.8, the same number it wrote for questions it was certain about.

The token probability separated them cleanly. Every wrong answer was below 0.6, and every right one was above 0.85. The lowest correct one was the eBPF-and-CUDA question at 0.88, with kernel taking most of the rest, which is a fair reading of a question that sits between two tracks. A threshold of 0.8 would have flagged exactly the three mistakes. Thirty questions is far too few to choose a threshold for real use, but the difference in kind is clear: one number is generated text, and the other is a measurement of the model.

One detail is visible in the alternatives. Tokens like Linux (capitalized) and ␣linux (with a leading space) show up among the top five, though the enum doesn’t allow them at that point. So the log probabilities Ollama reports are the model’s own distribution before the constraint is applied, just as chapter 1 found they’re the distribution before temperature. The constraint then removes the illegal options and the draw happens among the rest.

Validate, and retry with the reason

Constrained decoding is a feature of the server, and not every model, server or API offers it, or offers all of JSON Schema. Even when it does, a schema can’t express everything a program needs. So the output should be validated in code, whatever produced it, and a failure should be fed back to the model rather than silently dropped.

The pattern is short. Parse, check every field, and if anything is wrong, send the model its own reply plus a message saying exactly what was wrong, and ask again:

def validate(raw):
    try:
        o = json.loads(raw)
    except json.JSONDecodeError as e:
        return None, f"not valid JSON ({e.msg} at position {e.pos})"
    if not isinstance(o, dict):
        return None, "the top level must be an object"
    if o.get("track") not in LABELS:
        return None, f'"track" must be one of {LABELS}'
    c = o.get("confidence")
    if not isinstance(c, (int, float)) or not 0 <= c <= 1:
        return None, '"confidence" must be a number from 0 to 1'
    return o, None

msgs = [{"role": "user", "content": PROMPT.format(q=q)}]
for attempt in range(3):
    d = chat(msgs)          # POST to /api/chat, as above
    o, err = validate(d["message"]["content"])
    if o:
        break
    msgs += [d["message"],
             {"role": "user", "content": f"That reply was rejected: {err}. "
                                         "Reply again with only the JSON object."}]

The error message matters. “Invalid, try again” gives the model nothing to go on, and as chapter 1 showed, a resample at the same settings may just make the same choice. “track must be one of these six” changes the input, and so the distribution.

Run on the 30 questions with the firm prompt and no format, the loop never needed a second attempt: every first reply validated, and 28 were correct. That’s the expected outcome for an easy, flat output. The retry earns its place on harder outputs, which is where extraction comes in.

Extraction

Classification picks one value from a fixed set. Extraction pulls structured fields out of free text: names, dates, numbers, entities. The assistant will need it, for example to record which measurements a chapter reports, and it’s the more demanding of the two, because the values come from the text, not from a list.

The test: 15 prose paragraphs from this site that each contain at least three numbers with units (milliseconds, percentages, gigabytes and so on), picked at random from the 28 that qualify. The model was asked for every measured quantity, with the value copied exactly as written, under a schema, at temperature 0:

SCHEMA = {"type": "object", "properties": {"measurements": {"type": "array", "items": {
    "type": "object",
    "properties": {"what": {"type": "string"}, "value": {"type": "string"}, "unit": {"type": "string"}},
    "required": ["what", "value", "unit"]}}},
  "required": ["measurements"]}
PROMPT = """Extract every measured quantity stated in the passage.
Return JSON: {"measurements": [{"what": short description of what was measured,
  "value": the number exactly as written in the passage, "unit": the unit, or "" if none}]}

Passage:
"""

The value is a string on purpose. It makes a simple check possible: every extracted value must literally appear in the passage. A check like that can’t tell whether “43 minutes” was attached to the right description, but it can catch a number the model made up or transformed.

All 15 replies were valid against the schema. Across them the model returned 57 measurements, and 49 passed the literal check. Most passages came back clean. This one, from the capacity chapter of the System Design track, is typical:

At the database, cache misses at 90% arrive at about 2,083 a second. At 5 ms per query
(an assumption), that's about 10 queries in flight on average. ...

{"measurements": [
  {"what": "cache misses per second", "value": "2,083", "unit": "a second"},
  {"what": "time per query", "value": "5 ms", "unit": "ms"},
  {"what": "queries in flight on average", "value": "10", "unit": ""},
  {"what": "reads per second at 0% hit ratio", "value": "20,833", "unit": "reads a second"}
]}

Even here there’s a judgment call: the 5 ms is labeled “an assumption” in the text, and the model reported it as a measurement.

The eight failures came from two passages, and they show two different ways for valid output to be wrong. A paragraph from the video streaming design (“With 60 chunks of 10 seconds and 9 renditions, there are 540 independent tasks…”) produced six items whose descriptions were plausible and whose every value was the two-character string ", ". That’s a perfectly valid string, so the schema allowed it, and the reply was useless. Another passage described a ratio in words (“almost seven times”, “twice that again”), and the model copied the words as values, which a program expecting a number can’t use.

There were smaller slips that the check doesn’t count. For a paragraph reporting that one approach “took 44 ms”, the model returned the value ">= 44", adding a comparison the text doesn’t make. In several passages it left out numbers that described the setup rather than a result, such as the grid size and gate counts in a BFS benchmark, which is arguably right, but shows that “every measured quantity” means something less precise to the model than to the person who wrote the prompt.

Both failing passages went back to the model with the retry pattern, naming the values that didn’t appear in the text:

These values do not appear in the passage exactly as written: [", ", ", ", ...].
Copy each value exactly from the passage, and leave out anything that is not a number
in the passage. Return the full corrected JSON.

The video passage came back with 60, 9, 90 minutes and 10 seconds, all correct. The ratio passage came back still holding "seven" and "twice that again". One of two fixed, on a sample of two, says only that a retry with a specific reason can work, not how often.

What extraction adds to the picture is that the hard part isn’t the JSON. Constrained decoding made every reply parse. Whether the values were real, complete and attached to the right description took a check written for this particular task, and some errors, like a misattributed number, would need a check that understands the text.

Where this leaves things

The assistant can now get answers a program can use. The measured path, on 30 questions:

The format settings fix the shape and nothing else. The schema’s enum keeps labels inside the set, which is exactly what makes it dangerous without a "none" option. Validation in code catches what the schema can’t express, and a retry that states the error gives the model a real chance to fix it. The self-reported confidence is decoration, while the token probability is a real signal.

The misses point somewhere else, though. Across the runs, the questions the model got wrong were almost always the ones labeled essays: Kafka, Redis, PostgreSQL indexes. Nothing about those topics says “essay” rather than “system design”. The only way to know is to know what’s on this site, and the model doesn’t. It was trained on a snapshot of the internet that may or may not include these pages, and it has never been told what’s in them.

That’s the bigger version of the gap from chapter 1, where the model explained mutexes without mentioning ownership, the idea the site’s own essay is built around. Routing questions is a side job. The main job is answering them from the site’s chapters. Chapter 3 asks the model about this site directly and measures how often it knows, how often it says it doesn’t, and how often it makes something up.