1 — What the model sees

The project starts with a question this site already answers. The essay on semaphores, mutexes and locks spends several thousand words on the difference between the two, so it makes a good first test: ask a language model the same thing and see what comes back.

The model is Meta’s Llama 3 with 8 billion parameters, served by Ollama on a laptop. Ollama’s copy (llama3:latest) is quantized to 4 bits per weight (Q4_0), which brings it to 4.7 GB on disk and about 5 GB in memory once loaded. Every run in this chapter happened on one Mac, an Apple M3 Pro with 18 GB of memory, running Ollama 0.40.11. Timings are from that machine, and from single runs unless a sample size is given.

The first request

Ollama listens on localhost:11434 and speaks JSON over HTTP. The chat endpoint takes a list of messages and returns the next one.

import requests

r = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3",
    "messages": [{"role": "user",
                  "content": "What is the difference between a semaphore and a mutex?"}],
    "stream": False,
})
d = r.json()
print(d["message"]["content"])

About 27 seconds later, the answer came back. It was long, organized and confident. Here is the start and the end of it:

Semaphores and mutexes are both synchronization primitives used to coordinate
access to shared resources in concurrent programming. While they share some
similarities, they serve different purposes and have distinct characteristics.

**Semaphore:**

A semaphore is a variable that controls the access to a common resource by
multiple processes or threads. It's a counting mechanism that allows a limited
number of processes to access the resource simultaneously.
...
In summary, semaphores are used to control the number of concurrent accesses to
a shared resource, while mutexes are used to ensure exclusive access to a shared
resource. Both are essential synchronization primitives in concurrent programming.

None of it is wrong, exactly. But the essay on this site says the key idea of a mutex is ownership: the thread that locks it is the one that has to unlock it, while a semaphore has no owner at all and any thread may signal it. The model’s answer never mentions ownership. That gap matters for the project, and chapter 3 comes back to it. First, it’s worth knowing what actually happened inside that one request.

The rest of the response is bookkeeping, and it’s more useful than it looks:

{
 "model": "llama3",
 "done": true,
 "done_reason": "stop",
 "total_duration": 26817102500,
 "load_duration": 4306916,
 "prompt_eval_count": 21,
 "prompt_eval_duration": 272567000,
 "eval_count": 456,
 "eval_duration": 26537350000
}

Durations are in nanoseconds. The question was read as 21 tokens (prompt_eval_count) in 0.27 seconds. The answer was 456 tokens (eval_count) and took 26.5 seconds to write, which is about 17 tokens per second, or 58 milliseconds per token. Reading the prompt cost about 13 milliseconds per token, and for a prompt this short most of that is fixed overhead. done_reason: "stop" means the model ended the answer itself, rather than being cut off by a length limit.

Those numbers already hint at how the model works: it reads the whole input in one go, then writes the answer one token at a time, and each written token costs about the same. The sections below take each part in turn.

What actually reaches the model

The question was twelve words, but the model saw 21 tokens. The difference is the chat template. A model like Llama 3 doesn’t receive a list of messages. It receives one flat sequence of tokens, and Ollama builds that sequence from the messages using a template stored with the model. ollama show llama3 --template prints it:

{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

{{ .Response }}<|eot_id|>

So the text the model actually continued was this, with a begin-of-text token in front:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>

What is the difference between a semaphore and a mutex?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The strings in angle brackets are special tokens, each a single entry in the vocabulary that ordinary text can never produce. The template ends with the header for an assistant turn and nothing after it, so the most likely continuation is an assistant’s reply. When the model emits <|eot_id|> (end of turn), Ollama stops and returns what came before it. That’s the "stop" in done_reason. The model was fine-tuned on millions of conversations laid out exactly like this, which is why it behaves like an assistant at all. Underneath, it’s still doing one thing: predicting what comes next in a sequence.

This is also why “chat” is an illusion the client maintains. The model has no memory between requests. A second test made that concrete: a first message introducing the user as Ash, then a follow-up asking “What’s my name?”.

The second turn “remembered” only because the client sent the whole conversation again, the model’s own previous reply included. Every turn of a conversation re-sends everything before it, and the prompt grows each time.

Tokens

A token is the unit the model reads and writes. It’s usually a common word, a piece of a word, or a punctuation mark. Llama 3’s tokenizer has a vocabulary of 128,256 entries (Meta’s model card rounds it to 128K, and ollama show llama3 -v gives the exact number). Text is split into tokens by byte-pair encoding: starting from bytes, the tokenizer repeatedly merges the pairs that were most frequent in its training text, so frequent strings end up as single tokens and rare ones are built from pieces.

Ollama doesn’t expose the tokenizer directly, but two of its features are enough to see it. Sending a string in raw mode (no template) with num_predict: 1 returns prompt_eval_count, which is the token count plus one for the begin-of-text token. And asking the model to repeat a string with logprobs: true returns each generated token as a separate entry, which shows where the boundaries fall.

def count_tokens(s):
    d = requests.post(URL + "/api/generate", json={
        "model": "llama3", "prompt": s, "raw": True, "stream": False,
        "options": {"num_predict": 1}}).json()
    return d["prompt_eval_count"] - 1          # minus <|begin_of_text|>

def split_tokens(s):
    d = requests.post(URL + "/api/generate", json={
        "model": "llama3", "stream": False, "logprobs": True,
        "prompt": "Repeat the following text exactly, with nothing before or after it:\n" + s,
        "options": {"temperature": 0, "num_predict": 60}}).json()
    return [t["token"] for t in d["logprobs"]]

Here is what came out for a few strings from this site:

A few things stand out. The same word can be one token or two depending on whether a space comes before it and how it’s capitalized, because ␣semaphore and Semaphore were common enough in training text to earn their own entries while bare lowercase semaphore wasn’t. Numbers are cut into groups of up to three digits, so the model never sees 1234567 as one quantity. A name like Gopalakrishna costs five tokens. Scripts that were rare in the training data cost more: the five-character Kannada word took ten tokens, two per character, while a short English sentence of 86 characters and 19 words took 21.

That last ratio, roughly four characters or three-quarters of a word per token for English prose, is a useful rule of thumb. Everything about a model’s cost and limits is counted in tokens, not words, and code, numbers and non-English text all cost more per character.

Tokens also explain a famous failure. Asked how many times the letter r appears in “strawberry”, the model doesn’t see ten letters. It sees str, aw and berry, three opaque symbols, and has to recall their spelling from training rather than read it. More on how that went below.

One token at a time

A language model is a function from a sequence of tokens to a probability for every possible next token. One pass through the network produces 128,256 numbers, one per vocabulary entry, and turns them into probabilities that sum to one. Generating text is a loop: pick a token from that distribution, append it, run the network again.

The distribution is visible. Ollama 0.40 returns log probabilities when a request sets logprobs: true, and up to 20 alternatives per position with top_logprobs (asking for 21 returns the error top_logprobs must be between 0 and 20). Here is the distribution after the raw text A mutex is a:

d = requests.post(URL + "/api/generate", json={
    "model": "llama3", "prompt": "A mutex is a", "raw": True, "stream": False,
    "logprobs": True, "top_logprobs": 20, "options": {"num_predict": 1}}).json()
for alt in d["logprobs"][0]["top_logprobs"]:
    print(repr(alt["token"]), round(math.exp(alt["logprob"]), 4))

The top 20 together held 88.8% of the probability. The remaining 11% was spread over the other 128,000-odd tokens. No single continuation is a clear favorite here: “A mutex is a synchronization primitive”, “a type of lock” and “a lock that…” are all reasonable, and the model spreads its bets.

After each choice the distribution changes completely. Appending the top three candidates and asking again gave:

Some positions are nearly certain (after “a type”, the word “of” has 99.9%) and some are open. A long answer is hundreds of these choices in a row, each one conditioned on all the earlier ones. An early choice between “synchronization” and “lock” sends the rest of the sentence down a different path.

The loop stops in one of two ways. Either the model draws its end-of-turn token, or the request hits a limit: num_predict caps the number of tokens generated, and a response cut off that way reports done_reason: "length" instead of "stop". A program that ignores that field will happily treat half an answer as a whole one. The loop is also why responses can be streamed. Ollama streams by default, sending one small JSON object per token as it’s generated, and the "stream": false in the code above asks it to wait and send everything at once.

That loop also explains the timings from the first request. Reading the prompt is one pass over all 21 tokens in parallel, which a GPU does well. Writing is one full pass through all 8 billion parameters per token, strictly in sequence, because each token depends on the previous one. On this Mac that came to about 58 milliseconds per token. Why that step is limited by memory bandwidth rather than arithmetic is covered in Why LLMs need all of this on the GPU track.

Choosing from the distribution

Given a distribution, there are several ways to pick a token, and they’re the settings that appear in every model API.

Greedy decoding always takes the most likely token. It’s predictable, but it tends to produce flat, repetitive text, and the single most likely token at each step doesn’t add up to the most likely sentence.

Sampling draws a token at random in proportion to its probability. “Synchronization” would come up 29% of the time and “lock” 7%. Temperature reshapes the distribution before the draw. The model’s raw scores (logits) $z_i$ are divided by the temperature $T$ before being turned into probabilities:

$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$

A temperature below 1 sharpens the distribution toward the top choice, and above 1 flattens it. At $T = 0$ the division is undefined, and implementations treat it as greedy decoding. Applying the formula to the 20 tokens above (renormalized among themselves, so the numbers are approximate) shows the effect:

At 0.2 the model is effectively greedy. At 1.5 the tenth-ranked token, simple, rises from 1.7% to 3.1%, and over hundreds of tokens those small shifts add up to text that wanders.

One detail is easy to get wrong: the log probabilities Ollama returns are the model’s own distribution, before temperature. Requesting the same position at temperatures 0.2, 1.0 and 1.5 returned identical values ( synchronization at 0.2912 each time). The sampling settings change which token gets drawn, not the probabilities reported.

Two more filters cut off the long tail before the draw, so a single unlucky pick among thousands of unlikely tokens can’t derail a sentence:

• Top-k keeps only the k most likely tokens. Ollama’s default is 40.
• Top-p, or nucleus sampling, keeps the smallest set of top tokens whose probabilities add up to p, then renormalizes. The idea comes from Holtzman et al., 2019, who showed that sampling from the full distribution produces incoherent text because of that tail. Ollama’s default is 0.9. After A mutex is a, the top 20 tokens hold only 88.8% of the probability, so a 0.9 cutoff keeps more than 20 candidates, and top-k’s limit of 40 is the tighter one. After A mutex is a type, the single token of already holds 99.9%, so the draw is among one.
• Min-p keeps tokens whose probability is at least some fraction of the top token’s. Ollama’s default is 0, which turns it off.

A repeat penalty is a different kind of adjustment: it lowers the scores of tokens that already appeared recently, to stop a model from looping on the same phrase. Ollama’s documented default is 1.0, which means no penalty.

Ollama’s documented defaults are temperature 0.8, top-k 40 and top-p 0.9 (Modelfile parameters), and llama3:latest doesn’t override them. They’re set per request in options. The order they run in matters, and it differs between serving stacks. On this install, Ollama runs the model through a llama.cpp server, which prints its sampler chain to Ollama’s server log (~/.ollama/logs/server.log):

sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist

So here, top-k and top-p cut the tail of the model’s own distribution first, temperature reshapes what survives, and dist makes the random draw. With temperature 0 the last step is greedy instead of dist, and the log shows that variant too. The temperature table above applies the formula to the top 20 tokens alone, so it shows the effect, not the exact numbers this sampler would use.

The same question, ten times

Sampling has a direct consequence: ask the same thing twice and the answers differ. To see by how much, three questions went to the model ten times each with the default settings, then five times at temperature 0, then five times with a fixed seed of 42 and the default temperature.

def ask(q, n, **options):
    out = []
    for _ in range(n):
        d = requests.post(URL + "/api/chat", json={
            "model": "llama3", "stream": False, "options": options or None,
            "messages": [{"role": "user", "content": q}]}).json()
        out.append(d["message"]["content"])
    return out

For the two-sentence question, every one of the ten answers was worded differently, and they varied in length by about 20%. All ten made the same point about counting versus exclusive access, and none of the ten mentioned ownership.

The LRU cache answers were the same in substance and different in form: “Hash Map and Doubly Linked List.”, “HashMap and Doubly-Linked List.”, “Hash table and doubly linked list.”, and once “A few words: Hash Table + Doubly-Linked List.”, which repeated part of the instruction back. A person reads those as one answer. A program comparing strings sees seven.

The strawberry question is where it gets uncomfortable. The correct answer is 3. Over ten default runs the model said 2 five times, 4 three times and 3 twice. The distribution explains why. Asking for the log probabilities of the first answer token gave:

The model leans toward the right answer, but only just. With sampling on, roughly half of all requests are expected to come back wrong. Ten runs is a small sample, and getting the right answer only twice rather than about five times is within what chance allows, but the direction is clear.

The fixed settings made the output repeatable. All five runs at temperature 0 returned identical text, and so did all five with seed 42. That’s useful for tests and debugging. But repeatable isn’t the same as right: temperature 0 answered the strawberry question with 3, because 3 was the single most likely token, while seed 42 at temperature 0.8 answered 2, five times in a row. A seed just fixes the sequence of random draws. If the first draw lands on a wrong answer, it lands there every time.

And for the LRU question, seed 42 produced a different, worse answer than the others (“A combination of: Hash table, Double-ended queue, Counter”), consistently. Determinism also isn’t guaranteed across machines, versions or batch sizes, since floating-point results on a GPU can change with how the work is split up. It’s a convenience for one setup, not a property of the model.

The context window

There’s one more limit on what the model sees: how much of it fits. The context window is the number of tokens the model can attend to at once, prompt and answer together. Llama 3 was trained on sequences of 8,192 tokens, according to its model card, and that’s the most it can use well.

Ollama doesn’t necessarily give the model all of it. ollama ps shows the context each loaded model was given:

NAME             ID              SIZE      PROCESSOR    CONTEXT    RUNNER
llama3:latest    365c0bd3c000    5.0 GB    100% GPU     4096       llamacpp

On this Mac it loaded with 4,096 tokens. Ollama’s context length page says the default depends on available GPU memory, with 4k below 24 GB, while the Modelfile reference still lists a default of 2,048, so the number to trust is the one ollama ps reports. It can be raised per request with options: {"num_ctx": 8192}, at the cost of more memory for the attention cache (ollama ps showed 5.0 GB loaded at 4,096 and 5.7 GB at 8,192) (the GPU track explains why that cache grows with context).

What happens when the input doesn’t fit is worth seeing once, because nothing announces it. The test plants one sentence, “Note for the reader: the release codename is BLUEFIN-7.”, a tenth of the way into a document, then asks for the codename at temperature 0. The document is one or two chapters of this site’s System Design track: the caching chapter alone (7,600 characters), or the caching and resilience chapters together (20,600 characters). Each was sent two ways: as one long user message ending in the question, or as a conversation where the document is an earlier message and the question comes last.

msgs = [{"role": "user", "content": doc + "\n\n" + question}]            # one message
msgs = [{"role": "user", "content": "Here is a document:\n\n" + doc},      # a conversation
        {"role": "assistant", "content": "I've read it."},
        {"role": "user", "content": question}]
d = chat(msgs, options={"num_ctx": 4096, "temperature": 0})

When the document fit, the model found the note either way. When it didn’t, both requests came back with done_reason: "stop", no error and no warning, and the only sign of trouble is in prompt_eval_count. The two layouts lost information in two different ways.

The conversation lost everything but the question. Ollama’s chat handler, when the messages don’t fit, drops whole messages from the front until they do, always keeping system messages and the latest message (server/prompt.go). The document was one message too large to fit, so it went, and the model saw 38 tokens: the template and the question. Asked for a codename it had never been shown, it invented one, “M17”.

The single message couldn’t be dropped, since it was the latest, so the runner cut tokens out of it instead, down to 2,060. The part holding the note was among what went. The model answered honestly that no codename was mentioned, which is correct for the text it actually received and wrong for the text that was sent.

Neither failure is visible unless the program compares the size of what it sent with prompt_eval_count. Everything the model knows about the current request has to fit in this window, and the window is small: 4,096 tokens is about 16,000 characters of English, roughly one long chapter of this site.

Where this leaves things

One request, followed all the way down, comes to this. The model sees a single sequence of tokens that a template built from the messages, with no memory of anything not in that sequence. It turns that sequence into a probability for every possible next token, and a sampler draws one according to temperature, top-k and top-p. It repeats that until it draws an end-of-turn token or runs out of room, and all of it has to fit in a window of a few thousand tokens.

The behavior that follows from this is now measured rather than assumed. The same question gives a different answer each time: ten different texts out of ten for a short explanation, seven spellings of one answer to a simple question, and a wrong count of letters more often than a right one. Fixing the temperature or the seed makes the output repeatable, but not correct, and not in any particular shape.

That last part is the next problem. The assistant this track is building will be a program, and programs need answers they can parse: a field they can read, a label from a known set, a list they can loop over. Prose that varies with every request is fine for a person and useless for code. Chapter 2 asks the model for structured output and measures how often it actually delivers.