9 — Letting it act

Chapter 8 gave the retrieval pipeline a labeled question set and numbers to go with it. Some of the questions people actually ask about this site never had a chance in that set, because no passage answers them. “How many chapters does the DSA track have?” is a question about the collection, not about any text in it. “How long is the heaps chapter?” is a property of a file. “With the capacity chapter’s numbers, what is the peak egress?” needs a multiplication on figures that sit in a passage, and a language model sampling digits one token at a time is not a calculator.

The baseline pipeline (top-5 passages by cosine similarity, llama3:latest, 8B at Q4_0, told to answer only from the passages) shows the gap honestly. Asked “How many chapters does the DSA track have?”, it retrieved passages from a System Design chapter, the kernel lab setup, the JWT essay and the last DSA chapter twice, and answered “The passages do not contain the answer to this question.” Asked for the word count of DSA chapter 9, it retrieved nothing from that chapter and said the same. Both refusals are correct behavior, and both are dead ends. Asked for the peak egress, it found the passage and quoted “20,833 × 8 KB = 167 MB/s”, but only because the chapter happens to have done that multiplication already. A question with any other numbers would leave the model doing arithmetic in its head.

The fix is not better retrieval. The model needs a way to say “run this” and get the result back: to list the chapters, read a file’s word count or evaluate an expression, with a program doing the part that has a right answer. That is tool calling, and a loop around it is what most people mean by an agent.

The model doesn’t run anything

A tool call is text. The application describes some functions to the model, with a name, a description and a JSON schema for the arguments. When the model decides one would help, it produces a structured message naming the function and the arguments instead of an answer. The application runs the function, appends the result to the conversation, and asks the model again. The model never executes code and never reaches the network. It only writes requests, and the application decides whether to honor them.

That makes tool calling a special case of chapter 2: output a program can use. The difference is that the model has been trained to choose when to produce it and which of several shapes to use, and the runtime knows how to find it in the output.

llama3 can’t, llama3.1 can

Ollama’s chat endpoint takes a tools list. Sending one to the llama3 model used in the earlier chapters fails immediately:

curl -s localhost:11434/api/chat -d '{"model":"llama3","messages":[{"role":"user","content":"hi"}],"stream":false,
  "tools":[{"type":"function","function":{"name":"x","description":"x","parameters":{"type":"object","properties":{}}}}]}'

{"error":"registry.ollama.ai/library/llama3:latest does not support tools"}

Ollama decides this from the model’s chat template, which /api/show returns along with a list of capabilities. For llama3:latest the capabilities are ["completion"]. Its template has a system slot, a user slot and an assistant slot, and nothing else:

{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>

Meta’s original Llama 3 release in April 2024 wasn’t trained for function calling. Llama 3.1, released that July, was, and its template in Ollama has places for the tool list, for the model’s calls and for the results. Trimmed to the relevant lines:

{{- if .Tools }}
...
You are a helpful assistant with tool calling capabilities.
...
Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt.

Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. Do not use variables.

{{ range $.Tools }}
{{- . }}
{{ end }}
Question: {{ .Content }}<|eot_id|>
...
{{- else if eq .Role "tool" }}<|start_header_id|>ipython<|end_header_id|>

{{ .Content }}<|eot_id|>

So “tool support” is less mysterious than it sounds. The tool definitions are pasted into the prompt as JSON, the model was fine-tuned to reply with a JSON object in that format when a function fits, Ollama parses that reply into a tool_calls field, and tool results go back under a role the model saw in training (ipython, for Llama 3.1). The template also shows a detail worth knowing: the tool list is inserted only when the last message is from the user. Once a tool result is the last message, the definitions are no longer in the prompt, so the model’s next decision is made without them.

This chapter uses two tool-capable models of about the same size as llama3, both pulled with ollama pull. Both report ["completion", "tools"] from /api/show:

llama3.1:8b is the closest replacement for llama3, same family and size. qwen2.5:7b is a second, independently trained model, to see whether results belong to one model or to small models in general. Everything ran on one Mac (Apple M3 Pro, 18 GB) with Ollama 0.40, which serves both with a default context of 4,096 tokens.

One round trip

The request is an ordinary chat request with a tools array; the schemas are in the next section. Asked “How many words is chapter 9 of the DSA track?” with temperature 0, llama3.1:8b replied with no text and one call:

{"role": "assistant", "content": "",
 "tool_calls": [{"id": "call_g83bc2nf",
                 "function": {"index": 0, "name": "chapter_info", "arguments": {"track": "dsa", "chapter": "9"}}}]}

arguments arrives as a parsed JSON object, not a string. The program runs chapter_info(track="dsa", chapter="9"), and appends two messages to the conversation: the assistant message above, unchanged, and the result:

{"role": "tool", "tool_name": "chapter_info",
 "content": "{\"id\": \"09-heaps\", \"title\": \"9 — Heaps and priority queues\", ..., \"words\": 7584, \"code_blocks\": 32}"}

The second request sends the whole conversation again, and this time the model answers in text: “Chapter 9 of the DSA track has 7584 words.” That’s the full protocol, documented in Ollama’s API reference: tools on the request, tool_calls on the reply, role: "tool" with tool_name for each result.

The same exchange for “What is 17% of 2340?” went differently, and it’s a useful second example:

{"role": "assistant", "content": "",
 "tool_calls": [{"function": {"name": "calculate", "arguments": {"expression": "17% * 2340"}}}]}

17% * 2340 is not an arithmetic expression; % is the modulo operator, and Python’s parser rejects the line. The tool returned {"error": "SyntaxError: invalid syntax (<unknown>, line 1)"}. The model read that and replied: “I’ll try that again. To calculate 17% of 2340, I’ll use the formula: (17/100) * 2340. … 0.17 * 2340 = 397.8”. The answer is right, but it didn’t try the tool again. It did the multiplication itself, in text, which is what the tool was there to prevent. Here it happened to work.

The tools

The assistant gets four tools. One searches the site, and three are small deterministic functions with right answers:

def search_site(query: str, track: str | None = None) -> list:
    hits = s.search(query, k=20 if track else 5)
    if track:
        hits = [h for h in hits if h["track"] == track][:5]
    return [{"source": f"/{h['track']}/{h['id']}", "title": h["title"],
             "text": h["text"]} for h in hits]


def list_chapters(track: str) -> list:
    if track not in TRACKS:
        raise ValueError(f"unknown track {track!r}; expected one of {TRACKS}")
    return [{"id": d["id"], "title": d["title"]} for d in DOCS if d["track"] == track]


def chapter_info(track: str, chapter: str) -> dict:
    """chapter may be an id ('09-heaps') or a number ('9')."""
    ...
    return {"id": d["id"], "title": d["title"], "description": d["description"],
            "words": len(re.findall(r"\S+", d["body"])),
            "code_blocks": d["body"].count("```") // 2}


def calculate(expression: str) -> float:
    # walks the parsed expression and allows only numbers, + - * / ** % //, and parentheses
    ...

search_site is a minimal version of the baseline from chapter 6, rebuilt here so this chapter stands alone: the markdown of the six tracks (blog, kernel, gpu, design, dsa, essays), split on paragraph boundaries into chunks of at most about 800 characters with 100 characters of overlap, embedded with nomic-embed-text (with its search_document: and search_query: prefixes), and ranked by cosine similarity. Built independently to those settings, it produces 5,653 chunks rather than chapter 6’s 3,951, so its passage boundaries differ from chapter 6’s. The tool results in this chapter and the next two come from this index. The improvements from chapter 7 would help here too, but they aren’t the point of this chapter.

calculate deliberately doesn’t call eval. A model’s output is untrusted input to the program, and eval on it would let a model, or anything that can influence the model, run arbitrary Python. Parsing with ast and allowing only arithmetic nodes keeps the tool to exactly what its name says.

Each tool is described to the model with a JSON schema. These descriptions are the whole of what the model knows about the tools, so they’re written for it:

TRACK_ENUM = {"type": "string", "enum": TRACKS,
              "description": "Track slug: blog, kernel (Linux kernel), gpu, design (System Design), dsa (Data Structures and Algorithms), essays"}

SCHEMAS = [
    {"type": "function", "function": {
        "name": "search_site",
        "description": "Search the text of this site's chapters. Returns the 5 most relevant passages with their source. Use for questions about what the chapters say.",
        "parameters": {"type": "object", "properties": {
            "query": {"type": "string", "description": "What to search for, in plain words"},
            "track": dict(TRACK_ENUM, description="Optional: limit the search to one track")},
            "required": ["query"]}}},
    {"type": "function", "function": {
        "name": "list_chapters",
        "description": "List every chapter in one track, in order, with its id and title. Use to count chapters or find a chapter's id.",
        "parameters": {"type": "object", "properties": {"track": TRACK_ENUM}, "required": ["track"]}}},
    {"type": "function", "function": {
        "name": "chapter_info",
        "description": "Get one chapter's title, description, word count and number of code blocks.",
        "parameters": {"type": "object", "properties": {
            "track": TRACK_ENUM,
            "chapter": {"type": "string", "description": "Chapter number like '9' or id like '09-heaps'"}},
            "required": ["track", "chapter"]}}},
    {"type": "function", "function": {
        "name": "calculate",
        "description": "Evaluate an arithmetic expression exactly, e.g. '6944 * 3' or '(18 + 7) / 2'. Use for any arithmetic.",
        "parameters": {"type": "object", "properties": {
            "expression": {"type": "string", "description": "Numbers and + - * / ** % ( ) only"}},
            "required": ["expression"]}}},
]


def run_tool(name, args):
    """Run a tool call and always return a string, errors included, so the model can recover."""
    if name not in FUNCS:
        return json.dumps({"error": f"unknown tool {name!r}; available: {list(FUNCS)}"})
    try:
        return json.dumps(FUNCS[name](**args), ensure_ascii=False)
    except Exception as e:
        return json.dumps({"error": f"{type(e).__name__}: {e}"})

The track slugs are an enum because the model has no other way to know that the System Design track is called design and not system-design. run_tool never raises. An unknown tool, a missing argument or a bad expression comes back to the model as an error message it can read, instead of ending the program.

The loop

The agent loop has one idea: keep calling the model until it stops asking for tools.

import json, time
import requests
from tools import SCHEMAS, run_tool

OLLAMA = "http://localhost:11434/api/chat"
SYSTEM = (
    "You answer questions about the chapters of a technical website. "
    "Use the tools to look things up and to do arithmetic; do not guess numbers. "
    "When you have the answer, reply in plain text and cite chapter sources like /dsa/09-heaps."
)


def chat(model, messages, tools, options):
    r = requests.post(OLLAMA, json={"model": model, "messages": messages, "tools": tools,
                                    "stream": False, "options": options}, timeout=3600)
    r.raise_for_status()
    return r.json()


def run_agent(question, model="llama3.1:8b", tools=SCHEMAS, max_steps=8, options=None, system=SYSTEM):
    messages = [{"role": "system", "content": system}, {"role": "user", "content": question}]
    trace = []
    for step in range(max_steps):
        resp = chat(model, messages, tools, options or {})
        msg = resp["message"]
        messages.append(msg)
        calls = msg.get("tool_calls") or []
        trace.append({"step": step, "prompt_tokens": resp.get("prompt_eval_count"),
                      "calls": [(c["function"]["name"], c["function"]["arguments"]) for c in calls],
                      "content": msg.get("content", "")})
        if not calls:  # no tool requested: this is the answer
            return {"answer": msg.get("content", ""), "trace": trace, "messages": messages,
                    "stopped": "answered"}
        for c in calls:
            name, args = c["function"]["name"], c["function"]["arguments"]
            messages.append({"role": "tool", "tool_name": name, "content": run_tool(name, args)})
    return {"answer": None, "trace": trace, "messages": messages, "stopped": "step_limit"}

That’s the whole agent, about 40 lines. There’s no framework because there’s nothing for one to do yet. The conversation is a list, the model’s tool calls are appended to it, each result is appended as a tool message with the tool’s name, and the loop ends when the model replies with text. max_steps is there because a model can keep calling tools forever; chapter 10 is about what happens near that limit.

A first run:

$ python agent.py llama3.1:8b "How many chapters does the DSA track have?"
{"step": 0, "prompt_tokens": 583, "calls": [["list_chapters", {"track": "dsa"}]], "content": ""}
{"step": 1, "prompt_tokens": 492, "calls": [], "content": "The DSA track has 18 chapters."}
ANSWER: The DSA track has 18 chapters.

Two model calls: one that chose the tool and its argument, one that read 18 entries and answered.

Measuring tool-call accuracy

One example says little. The measurement uses fifteen questions about the site, each with a known answer computed from the files, grouped by what they need:

Each run is scored three ways. Tool choice is right if the run called the tool the question needs (or, for the last question, no tool). Arguments are valid if every call matches its schema: required fields present, no unknown fields, enum values from the list. Answer is right if the final reply contains the expected value, checked with a regular expression. For the egress question, both 167 (decimal megabytes, as the chapter computes it) and 163 (dividing by 1,024) count. The models ran at Ollama’s default temperature of 0.8, with replies capped at 1,024 tokens; llama3.1:8b ran each question three times and qwen2.5:7b twice.

By kind of question:

These are small samples from one machine, so the percentages are coarse; a difference of one run moves a row by 10 to 30 points. Three things are clear anyway.

Schema-valid is not the same as correct. Every argument in every call was valid against its schema, and 42% of llama3.1’s answers were still wrong. Across its 75 tool calls, 15 returned an error. It called chapter_info with chapter ids it made up (11-dp, dsa/10-dp, 07-dp, 04-gpu-architecture), which are valid strings. It searched for the peak factor with track set to dsa, which is a valid enum value, when the capacity estimation chapter is in design. Of the seven searches llama3.1 made that set a track, three set the wrong one. The schema catches the wrong kind of value; it can’t catch the wrong value.

The model calls several tools at once, before it has any results. Ollama returns a list of calls, and llama3.1 often filled it in one turn: two chapter_info lookups and a calculate whose expression was (word_count_4 - word_count_3) or (info4.word_count - info3.word_count), variable names for numbers it didn’t have yet. The Llama 3.1 template explicitly says “Do not use variables”, and four of its 21 calculate calls used them anyway. Each failed, and the model then did the subtraction itself.

Counting a list is not a lookup. Both models were worst at counting chapters, even though list_chapters returns exactly the list to count. llama3.1 said the kernel track has 12 chapters in all three runs of the GPU-plus-kernel question; it has 14 (numbered 00 to 12, plus a map page), and 12 is the last chapter number. qwen2.5 said the DSA track has 17 chapters in both runs, one fewer than the 18 entries in front of it. A model reading a list token by token has no counter. That is the same lesson as arithmetic, in a different form, and it has the same fix, below.

What tool design changes

To see how much of this depends on the tool descriptions, the same fifteen questions went to llama3.1 with a deliberately weaker version of the same four tools, one run each. The functions were identical; only what the model saw changed. The names were search, chapters, info and calc, the descriptions were “Search.”, “Get chapters.”, “Chapter info.” and “Calculator.”, the parameters were single letters (t, n, q, e), and there were no enums.

The errors explain the difference. Without an enum, the model had to guess what a track is called, and it guessed what a person would say: DSA track, DSA, System Design, essay, and once busiest-hour (a phrase from the question). With n as the parameter name, it passed chapters and dynamic programming where a chapter number belonged. Arithmetic, where the tool’s job is obvious from the question, was unaffected (3 of 3). The tool description is the only documentation the model reads, and an enum is the cheapest way to stop it from inventing values.

Errors as readable text mattered too. Ten runs across both models hit a tool error and continued, which they could only do because run_tool returned the error instead of raising it. qwen2.5 recovered in both its cases: asked about the dynamic programming chapter, it guessed the id 15-dp, got back “no chapter ‘15-dp’ in track ‘dsa’; call list_chapters to see ids”, called list_chapters, and answered 85, which is right. llama3.1 recovered in 1 of its 8. The error message suggesting the next step is part of the tool’s design.

Two fixes, measured

The failures above point at the tools as much as at the models, so two of them were changed and the affected questions rerun.

Counting moved into the program. list_chapters was changed to return {"track": ..., "count": 18, "chapters": [...]}. On the three single- and two-track counting questions, llama3.1 went from 6 right answers in 9 runs to 9 in 9. qwen2.5 was rerun on the questions it had missed (the DSA count, the GPU-plus-kernel total, and the average across all six tracks) and got all 6 runs right, including the six-track average it had failed both times. llama3.1 still failed the six-track average in all three runs: one reply described the calls in prose instead of making them, and two packed eighteen invented chapter_info(...) calls into a single calculate expression. A question that needs six calls is a different problem, and it’s the subject of the next chapter.

The track filter removed. search_site’s optional track parameter was an invitation to guess, and the implementation made a wrong guess silent: it took the top 20 passages and kept those from the requested track, which often left none, and an empty list gives the model nothing to go on. With the parameter removed from the schema, the three search-and-arithmetic questions about the capacity estimation chapter were rerun twice per model. llama3.1 went from 1 right answer in 9 runs to 2 in 6, and qwen2.5 stayed at 4 in 6. That is no clear improvement at this sample size. llama3.1 still often skipped the search entirely and answered from nothing, which no change to a parameter can fix. Removing the trap is still worth doing, and chapter 10 shows a multi-step task where it mattered a great deal.

Where this leaves things

The assistant can now act. A loop of about forty lines lets a tool-capable model call functions, read their results and answer from them, and on questions that take one or two calls it mostly works: qwen2.5 answered 77% of the fifteen questions correctly and llama3.1 58%, where the retrieval pipeline alone had nothing to retrieve. The measurements also showed where accuracy comes from. Clear names, descriptions and enums kept arguments sensible; errors returned as text let a model correct itself; and the biggest gains came from moving work the model does badly, counting and arithmetic, into the program, and from removing a parameter the model could only guess at.

Every question here needed at most a few calls. The six-track average, the one question that needed six lookups and a calculation, was the one both models failed most consistently before the counted tool, and llama3.1 failed it even after. That’s a small version of a general problem: as a task gets longer, the conversation fills with tool results, the original question moves further back, and each extra step is another chance to go wrong. Chapter 10 gives the agent longer tasks and measures how it wanders.