10 — When the agent loses the thread

The agent from chapter 9 handles questions that take one or two tool calls: count the chapters in a track, look up a word count, multiply two numbers. Those are most questions. The ones that remain are the ones people would most like to hand off, because they’re tedious: do the same lookup six times, keep the results, then combine them. This chapter gives the agent three such tasks and measures how it fails.

Three longer tasks

Each task has a known answer, computed directly from the files, and is graded by whether every part of it appears in the final reply:

The first two need six tool calls and some arithmetic. The third needs three searches that each find the right passage. The tools are chapter 9’s, unchanged: search_site, list_chapters, chapter_info and calculate. Every run used the agent loop from chapter 9 with a default limit of 20 steps, Ollama 0.40 at its default settings (temperature 0.8 and, on this machine, a 4,096-token context), on one Mac (Apple M3 Pro, 18 GB). The models were llama3.1:8b and qwen2.5:7b, both 4-bit (Q4_K_M) quantizations. Run counts are small, three per task per setting unless stated, so the tables below show what happened rather than rates to quote.

llama3.1 mostly never starts

llama3.1:8b got 30 runs across these tasks: the plain loop, the loop with a 16,384-token context, the loop with a system prompt telling it to work one call at a time, and a planning variant described below. None of the 30 runs was correct. The more telling number is that in 23 of them the model made no tool call at all. Its first reply was the final answer, and the final answer was a description of the calls it meant to make:

To answer this question, I will need to call the `list_chapters` function for each track, then
calculate the total number of chapters, the average per track, and the track with the most chapters.

Here is the JSON for the function calls:
```json
{
    "name": "list_chapters",
    "parameters": {
        "track": "blog"
    }
}
{
    "name": "list_chapters",
    ...

In 21 of those 23 replies, the calls were written out as JSON in the text; in the other two, they were a numbered list in prose. Ollama only turns model output into tool_calls when the output has the shape the template asks for, and a paragraph of explanation followed by several JSON objects in a code fence isn’t it. To the loop, a reply without tool_calls is an answer, so the run ended after one step.

The reason is visible in the Llama 3.1 template from chapter 9. It tells the model to “respond with a JSON for a function call”, singular, in a fixed format. One-call questions fit that. A task that obviously needs six calls doesn’t, and the model resolves the conflict by narrating. Telling it in the system prompt to make one call at a time and never write calls as text didn’t help: 7 of the 9 runs with that prompt still made no call.

The seven runs that did call tools show the next two problems. In the words task, one run sent eleven calls in its first turn: the six chapter_info lookups, which were right, and five calculate calls on the numbers 1 to 6 (1 + 2 + 3 + 4 + 5 + 6, 1 * 2 * 3 * 4 * 5 * 6, and so on), because it had to write the arithmetic before it had any word counts to put in it. It then added the six real counts in its head and reported 42,123 instead of 47,363. In the tracks task, the two runs that called list_chapters six times got design (31) and DSA (18) right and the other four tracks wrong. Those two tracks have ids numbered from 01 up to exactly the number of chapters, and the model appears to have read off the last number instead of counting. The kernel track starts at 00 and ends with a map page, the blog has a page without a number, and the essays have no numbers at all; both runs said 12 for the kernel (it has 14) and 8 for the essays (15), and they said 11 and 8 for the blog (13). Both also said 6 for the GPU track, which has 7.

qwen2.5 starts, then drifts

qwen2.5:7b gets going. On the tracks task, every run’s first turn was six list_chapters calls in parallel, one per track, with the right arguments. On the search task, every run’s first turn was three searches. The rest of this chapter uses it, because its failures happen later in a run, where the techniques that agents are known for are supposed to help.

It still didn’t finish the tasks correctly, and the ways it went wrong are worth separating, because each points to a different fix.

It miscounts. Six chapter lists come back, 98 entries in all, and the model has to count each list. In the first baseline run it reported 13, 14, 7, 27, 18 and 12 (the right numbers are 13, 14, 7, 31, 18 and 15). In the second it reported 23, 11, 1, 26, 11 and 7. Counting a list of 31 JSON objects by reading it is exactly the kind of work chapter 9 moved into a tool for a single track, and with six tracks at once the errors compound.

It does arithmetic in its head. With calculate available and the system prompt saying “do not guess numbers”, both of those runs added the counts and divided by six in the reply text, in LaTeX. The additions were correct for the numbers it had; the numbers were wrong.

It can’t stop. One baseline run on the tracks task never ended its reply. Watching the server log, it had generated more than 4,000 tokens over six and a half minutes when I stopped it. Nothing in the loop limited the length of a single reply. After that, every call set num_predict to 1,024 tokens, a cap on the reply rather than on the run.

It trusts its own filter. In the search task, 34 of qwen’s 35 searches set track to dsa, though both chapters it needed are in the System Design track. The search took the top 20 passages and kept those from the requested track, so 18 of the 35 searches returned an empty list. The model then wrote answers anyway: “the read-to-write ratio isn’t explicitly mentioned in the provided text. However, in the context of capacity estimation, it is often discussed in terms of…” That’s partly a tool-design flaw, an optional parameter that invites a guess and a silent empty result that gives no hint, and chapter 9’s measurements show the same trap with llama3.1. Here it means the search task mostly measures that trap, which is worth keeping in mind when reading the search rows below.

The context fills up

A tool result stays in the conversation for the rest of the run, so the prompt grows with every step. On the tracks task, the six chapter lists alone brought qwen’s prompt to 3,031 tokens on the second model call of one run and 3,655 on another. Ollama on this Mac uses a 4,096-token context unless the request sets num_ctx; the Ollama documentation gives 4k as the default below 24 GiB of VRAM. Every run in this chapter used that default, except the runs labeled 16k below.

What happens past the limit was measured directly. A conversation was built with a system message holding a code word, a first user message holding a favorite number (the prompt spelled it “favourite”, which the replies echo), a growing number of chapter-list tool results, and a final question asking for both:

The same twelve-result conversation sent with num_ctx set to 8,192 measured 4,722 prompt tokens and was answered correctly. So at the 4,096 default, Ollama had cut about 740 tokens before the model saw anything. It kept the system message and the final question and dropped from the oldest end, which is where the user’s original request sits. llama3.1:8b did the same: at twelve results it answered “Your favorite number is not specified, and your code word is PELICAN.” No error is returned when this happens. The only sign is a prompt_eval_count that stops growing just under the context size.

For an agent this is the literal version of losing the thread. The first thing to fall out of a long conversation is the task.

Memory

An agent’s memory, so far, is the message list. Everything the model knows about the task is whatever is still in it, which makes the list the obvious thing to manage. There are three common ways.

Trimming. Keep recent messages whole and shrink or drop older ones. It’s cheap and deterministic.

Summaries. When the conversation gets long, ask the model to summarize what has happened so far, and replace the old messages with the summary. It keeps meaning instead of bytes, at the cost of an extra model call and the risk that the summary drops the one detail that matters later.

Retrieval-based memory. Store facts or past messages outside the prompt, in the same kind of vector index as the site search from chapter 6, and retrieve the relevant ones each turn. It scales to conversations far longer than any context, and it fails the way retrieval fails, by not finding something that was said.

Only trimming was measured here. The policy keeps the two most recent tool results intact and cuts every older one to its first 300 characters:

def compact(messages, keep_last=2, limit=300):
    """Short-term memory policy: keep the newest tool results whole, shrink older ones."""
    tool_idx = [i for i, m in enumerate(messages) if m["role"] == "tool"]
    for i in tool_idx[:-keep_last]:
        c = messages[i]["content"]
        if len(c) > limit:
            messages[i] = dict(messages[i], content=c[:limit] + " ...[older result shortened]")
    return messages

It did what it was meant to do to the prompt: the largest prompt in a tracks run fell from 3,031 to 3,829 tokens without it to 1,984 to 3,123 with it. It didn’t help the answers. No trimmed run was correct, and on the tracks task trimming destroyed exactly the data the task needed, since a chapter list cut to 300 characters can’t be counted. One trimmed run reported 6, 8, 5, 9, 17 and 13 chapters; it had passed those numbers to calculate (correctly adding them to 58), and they came from lists the model could no longer see in full. Trimming suits results that have been used and won’t be needed again. Which results those are depends on the task, and a generic policy can’t know.

The opposite fix, more room, was also tried. Three runs per task used a 16,384-token context, set with num_ctx on a separate Ollama instance (on the shared machine, the default stayed at 4,096). Nothing was truncated and nothing improved. Two of the three tracks runs used the room to wander: after listing the chapters, they called chapter_info on chapter after chapter, 46 and 44 tool calls in all, 24 and 31 of them exact repeats of earlier calls, before answering with the same miscounts. A bigger context removes one failure, losing the task to truncation, and leaves the model free to spend the space on something else.

Planning

A common remedy for wandering is to plan first. Before the loop starts, the model is asked, with no tools, for a short numbered plan with one tool call per line. The plan is added to the system prompt, and after every round of tool results the loop appends a reminder of the task and the calls made so far, so that the goal sits near the end of the context where the model attends to it most:

if plan:  # keep the goal and progress near the end of the context
    done = "; ".join(f"{n}({a})" for n, a in stats["calls"])
    messages.append({"role": "user", "content":
                     f"Reminder of the task: {question}\nTool calls made so far: {done}\n"
                     "Do the next step of the plan, or give the final answer if all steps are done."})

It made the tracks task worse: 0 of 3 parts right in all three runs, where the plain loop usually got the design track right. The plans themselves were the problem. One had 22 steps, including search_site(query="tracks:design", track="design") before each listing, and len(chapters) and total_chapters = sum([steps 3, 6, 9, 12, 15, 18]) as steps, which no tool can run. Another planned calculate(expression='1+len(list_chapters(track="design"))-1'). Following it, the model pasted the entire kernel chapter list into a calculate expression, got a syntax error, and did the same with the GPU list, then with a list of chapters that don’t exist (“1.1 — Architecture”, “1.2 — Scheduling”). The plan was written by the same 7B model that can’t count a list, before it had seen any tool results, and the loop then held it to that plan.

llama3.1:8b’s one long planning run shows the same thing from the other side. Its plan included searching for “chapter count per track” six times in a row. In execution it made 28 tool calls, 15 of them exact repeats (the same search for “track with most chapters” twelve times), and its final reply was a loop of its own: “The next step in the plan is to list the chapters for each track. The previous step has already done this, so we can proceed to the next step. The next step is to search the site for the chapter count for each track. However, the previous step has already done this…”

On the search task, planning made no visible difference (one or two parts out of four, as before), because the failure there is the track filter, which the plans repeated.

Checking its own work

Reflection, or self-checking, adds a review step. When the agent produces an answer, a separate call shows the model the question, the tool results (each cut to 400 characters) and the answer, and asks for a verdict as JSON. If the verdict is negative, the problems go back into the conversation as a user message and the loop continues, once:

def self_check(question, messages, answer):
    evidence = "\n".join(f"{m['tool_name']}: {m['content'][:400]}" for m in messages if m["role"] == "tool")
    r = chat(MODEL, tools=[], options={},
             fmt={"type": "object", "properties": {"ok": {"type": "boolean"}, "problems": {"type": "string"}},
                  "required": ["ok", "problems"]},
             messages=[{"role": "user", "content":
                        f"Question:\n{question}\n\nTool results:\n{evidence}\n\nAnswer:\n{answer}\n\n"
                        "Check the answer against the question and the tool results. Does it answer every part, "
                        "and is every number supported by a tool result or correct arithmetic on them? "
                        "Reply as JSON: ok (true/false) and problems (empty if ok)."}])
    return json.loads(r["message"]["content"])

This ran together with the one-call-at-a-time system prompt, six runs in all. None was correct. All six answers were wrong, and the checker flagged four:

The checker was useful in one specific way: on the search task, it noticed every time that the answer was made up rather than drawn from the results. That’s a real gain, because an answer that says “I couldn’t find this” is better than an invented one, and the checker’s verdict could be used to say so. But it couldn’t do the checks that would have fixed the tracks task. The evidence it saw was cut to 400 characters per result, and even with full results, verifying a count means counting, the same skill the answer lacked. A checker made of the same model shares the model’s blind spots, which is why the useful checks in practice tend to be ones a program can do: does each cited source exist, does each number in the answer appear in a tool result, does the arithmetic recompute.

Limits

The loop has a step limit of 20 model calls. No single-agent run reached it; the most any qwen run used was 6. The runs that went wrong went wrong in ways a step count doesn’t see. llama3.1’s planning run spent 28 tool calls in 2 model turns, because one turn can request many calls. The runaway qwen reply was a single step. The limits that would have caught these count something else:

• Tool calls per run, not just model turns. A cap of, say, 15 calls would have stopped the 28-call and the 46-call runs.
• Repeated calls. An identical call with identical arguments returns the identical result here, since every tool is deterministic. Refusing the repeat, or returning “you already called this; the result was above”, costs nothing.
• Tokens per reply, through num_predict. That’s the only one of these that stops a single runaway generation, and it’s the one that was added after the 4,000-token run.
• Wall time for the whole run, so a slow tool or a stuck server can’t hold a request forever.

A limit doesn’t make an agent right. It turns an unbounded failure into a bounded one, and it should end the run with an explicit “stopped at the limit” rather than whatever the model said last, so the caller can tell a finished answer from an abandoned one. The chapter 9 loop already returns stopped: "step_limit" for that reason.

More agents

The other common remedy is to split the work. An orchestrator call asks the model to break the task into independent sub-questions and return them as JSON; each sub-question goes to a fresh agent with its own short conversation; then a final call combines the workers’ answers, with calculate available:

def multi_agent(question):
    """An orchestrator splits the task, fresh workers answer each piece, then it combines them."""
    r = chat(MODEL, tools=[], options={},
        fmt={"type": "object", "properties": {"subtasks": {"type": "array", "items": {"type": "string"}}},
             "required": ["subtasks"]},
        messages=[{"role": "user", "content":
                   "Split this task into independent sub-questions that a worker with tools for searching "
                   "and listing a website's chapters can each answer alone. Leave the final combining "
                   f"and arithmetic to me. Reply as JSON.\n\nTask: {question}"}])
    subtasks = json.loads(r["message"]["content"])["subtasks"]
    results = [(s, run_agent(s, model=MODEL, max_steps=8)["answer"]) for s in subtasks]
    notes = "\n".join(f"- {s}\n  worker answer: {a}" for s, a in results)
    return loop(f"{question}\n\nWorkers have already found:\n{notes}\n\n"
                "Combine their findings into the final answer; use calculate for any arithmetic.", max_steps=6)

The split itself worked. For the tracks task, the orchestrator produced six sub-questions, one per track, in all three runs, using format with a JSON schema from chapter 2. Each worker had one list to count, in a short conversation, and that helped: across the three runs, 14 of the 18 worker answers were right. In the single-agent runs whose replies gave a number for each track, 13 of 54 per-track counts were right. The final answers came close, with totals of 96 and 97 against the correct 98. None was correct, because a task built from six parts needs all six. The kernel worker said 13 every time (the track has 14, numbered 00 to 12 plus a map page), and the DSA worker once said 17.

That’s the arithmetic of splitting. If each worker is right with probability $p$, and the parts are independent, the whole task is right with probability

$P(\text{all } k \text{ right}) = p^k$

With the measured $p \approx 14/18 \approx 0.78$ and $k = 6$, that’s about 0.22. A sample of three runs can’t confirm a number like that, but it explains why none of the three succeeded despite most of the parts being right. More agents reduce each part’s difficulty and multiply the number of parts that can fail. They also multiply cost: the tracks runs took 15 or 16 model calls instead of 2 or 3.

On the search task, splitting didn’t help at all (two, one and zero parts right out of four). Each worker made the same search with the same track="dsa" filter and got the same empty or irrelevant results, and two of them then explained at length that the capacity estimation chapter was about Python list growth, citing /dsa/01-complexity. A shared flaw in a tool is not averaged out by asking more agents to use it. It’s repeated.

The fix that worked

Every technique above tries to help the model do something it does badly. The change that fixed the tracks task removed the hard part instead. list_chapters was changed to return its own count alongside the list:

def list_chapters_counted(track: str) -> dict:
    """Same listing, with the count done by the program instead of the model."""
    items = list_chapters(track)
    return {"track": track, "count": len(items), "chapters": items}

With that one change and the plain loop, qwen2.5 got the tracks task right in 3 of 3 runs, every count correct, where the 19 recorded qwen runs of every other variant had produced no correct answer. The model still did the final arithmetic in its reply rather than with calculate, but with six right numbers the sum and the average came out right. llama3.1:8b didn’t benefit on multi-track questions, because its failure happens before any count is seen: in chapter 9’s six-track average question, the counted tool fixed qwen and left llama3.1 at 0 of 3.

The search task got the same treatment. The track parameter was removed from search_site’s schema, so the model could no longer filter itself into an empty result. The searches then found the right material. In three runs of the plain loop, all nine searches returned System Design passages; the read-to-write and rate limiter searches had their chapters at the top, and the seconds-in-a-day searches found the 86,400 figure in the track’s chapter on measuring.

And the agent lost the thread in the most literal way this chapter has measured. Three searches returning five passages of up to 800 characters each pushed two of the runs to a 4,048-token prompt, just under the 4,096 limit, so the oldest messages had been cut, and the oldest message after the system prompt is the task. The replies show it. One began “Since your question wasn’t explicitly stated, I’ll compile some key points from the responses”, another “It looks like you provided multiple chapters’ contents”. The third run, at 2,868 tokens, still had the question in its prompt and also replied with a general summary of the passages. The grader, which only checks that the expected numbers appear, passed two of the three, because the summaries happened to include them. Nobody reading those replies would call them answers.

The fix for that is the reminder from the planning variant without the plan: after each round of tool results, append a short user message that restates the task. It’s the newest message, so truncation removes it last, and it’s at the end of the context, where it competes least with the passages for attention. With the reminder, all three runs answered the three questions in order. All three got the rate limiter numbers (200 and 100) and the 86,400 seconds right, and all three got the read-to-write ratio wrong, deriving one from a different example in a retrieved passage (“1% writes and 99% reads”, and in one run “100:1”). Three out of four parts in every run, against zero or near-zero for every earlier variant on this task. That last part is a retrieval problem of the kind chapter 7 works on, not an agent problem.

The “parts right” in the upper rows include parts the model guessed or got from general knowledge; 86,400 seconds in a day doesn’t need the site.

Frameworks

Everything above is a few dozen lines of Python around a list of messages. Frameworks such as LangGraph, LlamaIndex’s agents and the agent SDKs from the model vendors package the same pieces: a state object instead of a list, nodes and edges instead of if statements, built-in summarizing memory, checkpoints so that a long run can resume, and tracing. They’re worth using when the graph of steps gets large or runs need to survive a restart. They don’t change the measurements in this chapter, because the failures happen inside the model’s replies, and no orchestration layer can count a list the model miscounts.

Where this leaves things

With small local models, the long tasks failed in specific, repeatable ways. llama3.1:8b mostly didn’t start, writing its tool calls out as text. qwen2.5:7b started and then miscounted, did arithmetic in its head, filtered its own searches into empty results, generated without stopping once, and, when the searches finally worked, lost the task to the context limit. The standard remedies each addressed a real mechanism and each fell short on its own: trimming freed space by throwing away data the task needed, a bigger context gave room to wander, a plan written before any results locked in the planner’s mistakes, a self-check by the same model missed the same errors, and workers multiplied the chances for one part to be wrong.

What worked was moving work out of the model. A count returned by the program, a search parameter taken away, a reminder of the task placed where truncation can’t reach it, and caps on reply length and tool calls. Each is a few lines in the host program, and each made the model’s job smaller instead of asking it to be more careful.

That puts most of the weight on the tools and on the code around them, and that code is all hand-written. The search, the chapter listing, their schemas, the error handling and the conversion from tool results to messages exist once, inside this one script. Another application that wanted the same search would have to copy all of it, and every change to a tool means changing every program that uses it. Chapter 11 moves the tools out of the agent and behind a standard protocol.