13 — Running it for real

Chapter 12 ended with an assistant that can be trusted with its own tools, up to a point: the side effects sit behind code that the model can’t talk its way past. That was the last problem visible from one person at one keyboard. Real users bring different ones. They ask the same thing many times in slightly different words, they wait for answers and give up when the wait is long, they ask about things the documents don’t cover, and they keep arriving when the model server is slow or down.

This chapter takes the baseline pipeline (800-character chunks, nomic-embed-text embeddings, cosine top-5, llama3:latest for generation, a prompt that asks for answers only from the passages with citations) and looks at it the way an operator would. Where does the time go? What does an answer cost? What can be cached, and what goes wrong when it is? What happens when the model doesn’t answer? And when an answer is wrong, how is the cause found?

All timings come from one Mac (Apple M3 Pro, 18 GB), running llama3:latest (8B parameters, Q4_0 quantization) and nomic-embed-text under Ollama 0.40 with its default context window (num_ctx) of 4,096 tokens. Every generation call sets num_predict, a cap on output tokens, so one request that never stops can’t hold the server for everyone else; no answer here came near the cap of 512. The Mac was shared with other jobs while these ran, which turned out to be instructive: it’s the same position a model server is in under real traffic.

Where the time goes

Ten questions about the site went through the pipeline one at a time, with the answer streamed so the first token could be timed. Ollama also reports its own measurements in the final message of each response: how long it spent evaluating the prompt, how long generating, how many tokens of each. Those are compute time on the server, without any waiting.

# lat13.py (key part)
with requests.post(f"{OLLAMA}/api/chat", stream=True, json={
        "model": model, "stream": True, "options": {"temperature": 0, "num_predict": 512},
        "messages": [{"role": "system", "content": rag.SYSTEM}, {"role": "user", "content": prompt}]}) as r:
    for line in r.iter_lines():
        ev = json.loads(line)
        if ev.get("message", {}).get("content") and first is None:
            first = time.perf_counter()               # time to first token, as the client sees it
        if ev.get("done"):
            final = ev                                # server-side counts and durations
...
"prompt_tokens": final["prompt_eval_count"], "prompt_ms": final["prompt_eval_duration"] / 1e6,
"out_tokens":   final["eval_count"],        "decode_ms": final["eval_duration"] / 1e6,

The server-side numbers, over the ten questions:

Retrieval is nearly free. The question’s embedding and the search together take about 30 ms, which is under 1% of the answer. The work is in the model, and it splits into two phases with different costs.

Prefill reads the whole prompt: the system message, five passages and the question, about 900 tokens. All of those tokens are known up front, so they’re processed in parallel, at around 277 tokens a second here. Prefill sets the time to first token, the pause before anything appears.

Decode writes the answer one token at a time, each depending on the last, at about 20 tokens a second. Its cost is proportional to the length of the answer. The shortest answer here (17 tokens, “I don’t know…”) took under a second to write; the longest (230 tokens, a list of kernel functions for the syscall question) took 17.6 seconds.

The decode rate is set mostly by memory bandwidth, not arithmetic: every new token needs all of the model’s weights read once. The GPU track’s “speed limit” turns that into a bound, model size divided by bandwidth, and the measured rate has to sit below it. Prefill reuses each weight across hundreds of tokens at once, which is why it runs over ten times faster per token.

For a typical answer the budget is then roughly 3.3 s of prefill plus 2.3 s of decode, about 5.7 s of model time, and a user sees the first word after 3.3 s. Two things move that most. Fewer or shorter passages cut prefill, which is one more reason the retrieval quality work of chapter 7 matters: five passages where three relevant ones would do cost a second of waiting on every question. And a shorter answer cuts decode, which is a prompt instruction away.

What the user actually waited

The client-side time to first token told a different story: a median of 8.0 s, ranging from 2.4 to 26.6 s, against 3.3 s of actual prefill. The difference is time spent queued. Ollama here serves one model at a time and other jobs were sending it requests, so each of these requests waited its turn. In the same way, the query embeddings took a median of 3.4 s from the client’s point of view against 29 ms of compute, because the server was busy with other jobs and had to switch between models to serve them.

That’s not a flaw in the measurement; it’s the first lesson of running this for real. Compute time is what one request costs. Latency is compute plus queueing, and queueing grows quickly as a server approaches full use, which the System Design track’s chapter on queueing works through. A model server that’s 90% busy will make users wait mostly for other users. The two numbers need separate tracking: compute per request for cost and capacity planning, end-to-end latency for what users experience.

What an answer costs

Locally, an answer costs about 5.7 seconds of the Mac’s GPU. Run one at a time, that’s roughly 600 answers an hour at best, before any queueing. Power draw wasn’t measured, so this chapter doesn’t put a money figure on local compute.

Hosted APIs price by token, with input and output priced separately. Anthropic’s published prices, as fetched on 10 November 2026, for a few of the current models (US dollars per million tokens):

Applying those to the median answer measured above, 914 input tokens and 47 output tokens:

$\text{cost} = \frac{914 \times p_{\text{in}} + 47 \times p_{\text{out}}}{10^6}$

The token counts come from llama3’s tokenizer. Other models split text differently, so these are estimates of the right size rather than exact bills.

A few things follow from the formula. Input tokens dominate this workload: 914 in for 47 out. Output is priced five times higher per token, but there’s twenty times less of it. So the passages are the cost, the same way they’re the latency, and trimming them saves on both. The longest answer above (230 tokens) would cost nearly twice the median on Sonnet 5.5. An agent from chapter 9 costs more again, because each step resends the whole conversation so far: a search step and an answer step pay for the system prompt twice and the passages at least once. And there’s a factor of 40 between the cheapest and most expensive model in the table, which is the strongest argument for sending each kind of request to the smallest model that handles it well.

Caching

Some of that cost can be avoided by not computing the same thing twice. There are three different caches here, with very different risks.

The prefix cache

The cheapest one is already running. When a request starts with exactly the same tokens as the previous one, Ollama can keep the attention keys and values it computed for them (the KV cache, covered in the GPU track) and skip that part of prefill. Each of the ten questions was sent twice in a row with an identical prompt:

That’s about 58 times faster for the prompt. The answer still has to be decoded token by token, so the total saving depends on the answer’s length.

Hosted APIs sell the same mechanism as prompt caching, at the cache-hit prices in the table above: a tenth of the input price or less. It only helps when a long prefix repeats exactly. For this assistant, the fixed prefix is the system prompt, a few dozen tokens; everything after it is passages that change with the question. Prompt caching would save very little here. It pays off when a long fixed block (detailed instructions, tool definitions, a whole document being discussed) leads every request, so that block should come first and the varying parts last.

The exact answer cache

The next cache stores whole answers, keyed on the question. A hit skips retrieval and generation entirely. The question is how often users really repeat themselves.

A stream of 52 questions tests it: 10 original questions, then 30 repeats of them (some identical, some differing in case or punctuation, some reworded), then 12 “traps”, questions that look like an original but mean something else:

# cache13.py (a few of the questions)
GROUPS = [
    ["How do I stop a cache stampede?", "how do i stop a cache stampede", "How can I prevent a cache stampede?",
     "What stops a thundering herd on a cache?"],
    ["Why add jitter to retries?", "why add jitter to retries", "What is the point of jitter in backoff?",
     "Why randomize retry delays?"],
    ...
]
TRAPS = [(0, "How do I start a cache stampede test?"), (4, "Why remove jitter from retries?"),
         (8, "How does TLS encryption work?"), ...]

Of the 30 repeats, an exact string match caught 1. Lowercasing and stripping punctuation first caught 9. Both made no mistakes, because they can’t: two strings that normalize to the same text are the same question.

The semantic cache

A semantic cache goes further: embed each question, and if a previous question’s embedding is close enough, return its answer. It catches rewordings, which is where most repeats are. It also catches questions that sound alike and aren’t.

The same stream, with each question compared to every earlier one by cosine similarity of nomic-embed-text query embeddings. A “correct hit” returns the answer to the same question; a “wrong hit” returns the answer to a different one:

The wrong hits are worth reading:

These are the failure modes from chapter 4 showing up in production form. “Add” and “remove” are opposites and sit at 0.948, because embeddings capture topic far better than negation. The real rewordings ranged from 0.645 to 1.0 (median 0.920), and the traps from 0.485 to 0.948 (median 0.773). The two ranges overlap, so there’s no threshold that catches most rewordings and no traps. In this sample, 0.95 avoided every trap and caught 14 of 30 repeats, only 5 more than plain normalization.

A wrong semantic hit is worse than a slow answer. The user gets a fluent, cited, confident answer to a question they didn’t ask, and nothing in the response says so. A safer design uses a high threshold, scopes the cache to questions that are known to be frequent, and treats a semantic match as a hint (a candidate answer to check, or a way to skip retrieval but not generation) rather than as the answer.

Stale answers

Every answer cache has the problem described in the System Design track’s caching chapter: the cached copy outlives the source. Here the “source” is three things at once. If a chapter is edited, an answer built from its old text is stale. If the prompt changes, or the model is swapped, the cached answers came from a system that no longer exists.

The straightforward fix is to put all of them in the key, so a change makes old entries unreachable:

key = sha256(json.dumps({
    "q": normalize(question),
    "index_version": index.version,      # e.g. a hash of the chunk ids and their text
    "prompt_version": PROMPT_VERSION,
    "model": "llama3:latest",
}).encode()).hexdigest()

That’s correct but blunt: editing one chapter invalidates every cached answer. Finer invalidation records which chunks each answer used (they’re in the citations) and drops only the answers that cited a changed chunk. A TTL on top catches anything the bookkeeping misses.

A gateway with fallbacks and timeouts

So far the model has always answered. A hosted model sometimes returns an error, sometimes takes far too long, and sometimes stops answering entirely. The patterns from the resilience chapter apply directly, and they fit in under 90 lines placed between the assistant and its models: a gateway.

# gateway.py (key part)
class Route:
    def __init__(self, name, url, model, timeout):
        self.name, self.url, self.model, self.timeout = name, url, model, timeout
        self.breaker = Breaker(threshold=3, cooldown=30.0)

    def call(self, messages):
        r = requests.post(f"{self.url}/api/chat", timeout=self.timeout, json={
            "model": self.model, "messages": messages, "stream": False, "options": {"temperature": 0, "num_predict": 512}})
        r.raise_for_status()
        return r.json()["message"]["content"]


class Gateway:
    def __init__(self, routes, degraded):
        self.routes, self.degraded = routes, degraded

    def complete(self, messages, trace):
        for route in self.routes:
            if not route.breaker.allow():
                trace.append({"route": route.name, "outcome": "skipped (breaker open)", "ms": 0})
                continue
            t = time.perf_counter()
            try:
                out = route.call(messages)
                route.breaker.record(True)
                trace.append({"route": route.name, "outcome": "ok", "ms": ...})
                return route.name, out
            except requests.RequestException as e:
                route.breaker.record(False)
                trace.append({"route": route.name, "outcome": type(e).__name__, "ms": ...})
        return "degraded", self.degraded(messages)

Each route has its own deadline. When a route fails or times out, the next one gets the request. A circuit breaker per route counts consecutive failures, and after three it stops sending traffic to that route for 30 seconds, so a dead provider costs nothing instead of costing a full timeout on every request. When every route is gone, the last step needs no model at all: it returns the sources retrieval found, which is still useful to someone looking for the right chapter.

To test it, the primary route goes through a small local proxy standing in for a remote provider, which can pass requests on to Ollama, return 503, or hang for 60 seconds. The fallback is the local model directly. Both serve llama3:latest, so any difference in timing is the gateway’s. The primary’s deadline is 20 seconds.

The table shows the trade the timeout makes. A fast failure (the 503) costs almost nothing. A hang costs the full deadline before the fallback even starts, so each of those users waited over 20 seconds. The breaker is what limits the damage to the first few users: after three failures, the rest went straight to the fallback. A shorter deadline would cut the wait but fail requests that were just slow, and the healthy runs here took 10–11 seconds end to end on a shared machine, so 20 seconds wasn’t generous. Streaming helps: a deadline on the first token can be much shorter than one on the whole response, because a server that has started answering is very likely to finish.

In a real deployment the fallback is usually a different provider or a smaller model, and that has a cost the table can’t show. The answer changes. Prompts tuned for one model behave differently on another, and the evaluation set from chapter 8 should be run against every route the gateway might use, not just the primary. Retries belong in the gateway too, with the limits the resilience chapter describes: only on errors that might clear, with jittered backoff, and within a budget, because a retry storm against an overloaded model server keeps it overloaded.

Tracing a bad answer

When a user reports a wrong answer, the question is which stage caused it. Retrieval might have missed the passage, the prompt might have been truncated, or the model might have ignored what it was given. Without a record of each step, these look identical from outside.

A trace records each stage of one request as a span, with a duration and the attributes that matter for that stage, all sharing one trace ID. This is the same model as distributed tracing in the System Design track’s operating chapter, applied inside one request:

# trace13.py (key part)
with tr.span("request", question=q, model=model, index_chunks=len(idx.texts)) as top:
    with tr.span("embed_query", embed_model="nomic-embed-text"):
        qv = rag.embed([q], prefix="search_query: ")[0]
    with tr.span("search", k=5) as a:
        ...
        a["results"] = [{"source": h["source"], "chunk": h["chunk"], "score": h["score"]} for h in hits]
    with tr.span("build_prompt") as a:
        prompt = rag.build_prompt(q, hits)
        a["chars"] = len(prompt)
    with tr.span("generate") as a:
        ...
        a.update(prompt_tokens=r["prompt_eval_count"], output_tokens=r["eval_count"], answer=out)

One of the traced questions was “What sysctl value does SynapseOS set for kernel.kptr_restrict?” The answer is in the SynapseOS essay: 2. The assistant said:

I don't know. The passages do not contain the answer.

The trace:

embed_query    3130.2 ms   embed_model=nomic-embed-text
search            3.1 ms   k=5
                           /essays/synapseos chunk 7   0.722
                           /essays/synapseos chunk 9   0.712
                           /essays/synapseos chunk 0   0.681
                           /essays/synapseos chunk 39  0.668
                           /essays/synapseos chunk 10  0.665
build_prompt      0.0 ms   chars=3045
generate       8541.5 ms   prompt_tokens=875 output_tokens=14

Retrieval did its job. All five passages are from the right essay, and chunk 9, at rank 2, contains the table row word for word: `kernel.kptr_restrict` | 2 | Kernel pointers are hidden. The prompt was 875 tokens, well inside the context window, so nothing was cut. The failure is in generation.

The trace narrows it to a stage; a few targeted reruns at temperature 0 narrowed it further:

The model has the fact. What it won’t do is connect “SynapseOS” in the question to a passage that never says “SynapseOS”: the chunk is a table cut out of the middle of the essay, and the essay’s name appears only in its source path. The instruction to answer only from the passages makes the model strict about that link, and neither a title prefix nor an extra sentence in the system prompt loosened it. The cause is found, which is what a trace is for. The fix isn’t found yet; candidates are rewriting the question before generation (as in chapter 7), or chunking that keeps a table together with the heading and paragraph that introduce it.

Two other things showed up in traces that no log of answers would have revealed. The first trace taken spent 373 seconds in embed_query and 13 seconds in generate: the embedding model was waiting behind other jobs on the shared server. And 2 of the 10 answers in the latency test were “I don’t know”, for two different reasons a trace tells apart. For the circuit breaker question, all five passages came from the resilience chapter and described breakers opening, waiting and closing, so that was the same over-strictness in generation. For “How does HNSW differ from IVF?”, the top passages came from the video streaming chapter, the SynapseOS essay and the JWT essay, with similarity scores of 0.56 to 0.57; only one passage, a table row in the PostgreSQL indexes essay, mentioned HNSW at all. That’s a retrieval failure, the weakness with short acronyms from chapter 4, and a low top score like that is a signal in its own right, as the section on drift shows.

A trace also holds the full question, passages and answer, which is often personal data. Real traces need retention limits and access control like any other log, and sampling: keep all traces of errors, slow requests and answers users flagged, and a small fraction of the rest.

Guardrails

Guardrails are checks around the model that don’t depend on the model behaving. Chapter 12 covered the most important ones, on tools. The ones for answers are simpler, deterministic, and run on every response:

# guard13.py (key part)
def check(answer, n_passages, passages):
    problems = []
    cites = [int(c) for c in re.findall(r"\[(\d+)\]", answer)]
    refused = bool(re.match(r"\s*I don't know", answer))
    if not refused and not cites:
        problems.append("no citation")
    if any(c < 1 or c > n_passages for c in cites):
        problems.append("cites a passage that does not exist")
    for host in re.findall(r"https?://([^/\s)\]]+)", answer):
        if host not in ALLOWED_HOSTS:
            problems.append(f"link to {host}, which is not on the allowlist")
    if re.search(r"(?i)(api[_-]?key|password|BEGIN [A-Z ]*PRIVATE KEY)", answer):
        problems.append("looks like a secret")
    return {"refused": refused, "citations": sorted(set(cites)), "problems": problems}

On the ten answers from the latency test, it flagged nothing: every non-refusal cited at least one passage, every citation pointed at a passage that existed, and there were no links or secrets. It counted 2 refusals.

That result shows what checks like these can and can’t do. They catch shape: missing citations, invented citation numbers, a link to a host outside the allowlist (which is how chapter 12’s planted link gets stopped), a credential pattern. The allowlist has a cost: chapters on this site cite external sources, and an answer that repeats one of those links gets flagged too, so a flag here means “look”, not “block”, except for links the assistant presents as its own recommendation. They don’t catch an answer that’s well-formed and wrong, or a refusal that should have been an answer, and at least one of the two refusals here was that. Whether an answer is faithful to its passages takes the judge from chapter 8, which is too slow and too costly to run on every answer but can run on a sample, and on every answer a user flags.

Input checks belong on the other side: a cap on question length (which also caps prefill time and cost), rate limits per user (the token bucket from the rate limiter chapter), and a cap on agent steps, as in chapter 10.

Monitoring drift

The assistant works well on questions like the ones it was evaluated on. If users start asking about things the site doesn’t cover, quality drops without any error being raised: retrieval still returns five passages, the model still writes an answer, and the latency graph looks the same.

One cheap signal is already computed on every request: the similarity score of the best retrieved passage. Thirty questions about topics the site covers were compared with twenty about things it doesn’t (sourdough, React hooks, Terraform, cricket, Solidity and so on):

The groups separate, though not cleanly. At a cutoff of 0.65, 14 of 20 off-topic questions fall below it and 2 of 30 on-topic ones do. Off-topic questions never score near zero, because the nearest of 5,669 chunks is always somewhat related to something; “What does a Spark shuffle do?” finds a passage about data movement.

As a per-question filter, that’s too noisy to refuse on. As a trend, it works better. Mixing the measured scores into batches of 40 questions, with an increasing share drawn from the off-topic set (200 random batches each):

A 10% shift is lost in the noise of one batch; 30% moves the median well outside the day-to-day spread. Plotted daily, that line shows users drifting away from the corpus before quality complaints do.

It belongs on a dashboard with the other numbers this chapter produced, each of which catches a different failure:

The operating chapter covers how to alert on numbers like these without drowning in alerts: alert on what users feel (latency, errors, the feedback rate), and keep the rest for diagnosis.

Where this leaves things

The assistant can now run for other people. Its time and cost per answer are measured and mostly in the prompt; caching saves real work where questions repeat exactly, and semantic caching can return confident wrong answers; a gateway keeps it answering when a model fails; traces locate failures by stage; and a handful of signals show when things drift.

One step doesn’t fit. Before retrieval, the assistant could route a question to the right part of the site, the metadata filter from chapter 7, so that search only looks in one track. Asking the 8B model for that one-word decision costs a median of 230 ms of GPU per question (n = 30, server-side, 90th percentile 465 ms), plus another trip through the queue, and the decision turns out to be wrong most of the time: the labels follow how this site happens to be organized, and no description of the tracks in a prompt captures that well. Chapter 14 measures how wrong, and takes up what to do when prompting runs out and the model itself has to change.