1 — From text to numbers

A language model is arithmetic. Inside, it multiplies and adds arrays of numbers, and at the end it produces one score for every possible next piece of text. It can’t take the letter “c” and multiply it by anything. So the first job, before any model exists, is a fixed rule that turns text into a list of whole numbers and turns the numbers back into exactly the same text.

That rule is the tokenizer. It decides what the pieces are (letters, words, or something in between), gives every piece an ID, and the list of all pieces it knows is the vocabulary. The choice matters more than it looks. The model will predict one piece at a time, so the vocabulary is the menu it picks from at every step. Pieces that are too small make every sentence long. Pieces that are too big leave words off the menu.

This chapter tries three answers in order: characters, words, and byte-pair encoding, which is what GPT-2 uses. The running text is H. G. Wells’s The Time Machine, the Project Gutenberg plain-text edition (eBook #35, public domain in the US), with Gutenberg’s header and footer stripped: 179,698 characters. Every run here happened on an Apple M3 Pro with 18 GB of memory, using Python 3.12, PyTorch 2.15.0 and tiktoken 0.14.0, all on the CPU.

Characters

The simplest rule gives every distinct character an ID. For a six-word sentence, the vocabulary is the sorted set of characters in it, and the IDs are positions in that list.

text = "the cat sat on the mat"
chars = sorted(set(text))
stoi = {ch: i for i, ch in enumerate(chars)}   # "string to integer"
print(stoi)
print([stoi[ch] for ch in text])

{' ': 0, 'a': 1, 'c': 2, 'e': 3, 'h': 4, 'm': 5, 'n': 6, 'o': 7, 's': 8, 't': 9}
[9, 4, 3, 0, 2, 1, 9, 0, 8, 1, 9, 0, 7, 6, 0, 9, 4, 3, 0, 5, 1, 9]

Ten symbols cover the sentence, and decoding is just the reverse lookup. Over the whole book the vocabulary only grows to 75 distinct characters: upper and lower case, digits, punctuation, and a few typographic dashes and quotes.

A tiny vocabulary is attractive, and nothing a text can contain is ever missing, as long as the vocabulary includes every character that will ever appear. The cost is length. The book becomes 179,698 IDs, one per character. The model has to predict each one, and “the” alone is three predictions. A model can also only look at a fixed number of tokens at once (GPT-2 sees 1,024), so with characters that window covers about one page. Each symbol on its own also carries almost no meaning. The letter “t” says very little; the word “time” says a lot.

Words

The opposite rule gives every word an ID. Splitting the book into runs of letters and digits, with each punctuation mark as its own piece:

import re
words = re.findall(r"\w+|[^\w\s]", book)
print(len(words), len(set(words)))
print(words[:12])

38307 4906
['The', 'Time', 'Machine', 'An', 'Invention', 'by', 'H', '.', 'G', '.', 'Wells', 'CONTENTS']

The \w+ part of the pattern matches a run of word characters, and [^\w\s] matches one character that is neither a word character nor whitespace, which catches punctuation. The book shrinks to 38,307 tokens, about a fifth of the character count, and each token now means something.

The trouble is the words that aren’t in the vocabulary. A vocabulary has to be fixed before the model is trained, so it’s built from some text and then used on other text. Building it from the first half of the book and checking the second half shows the size of the problem:

half = len(words) // 2
seen = set(words[:half])
unseen = [w for w in words[half:] if w not in seen]
print(len(seen), len(unseen), f"{len(unseen) / (len(words) - half):.1%}")
print(Counter(unseen).most_common(8))

3283 2242 11.7%
[('wood', 14), ('camphor', 11), ('Underworld', 10), ('arms', 10), ('iron', 10), ('slower', 10), ('Eloi', 8), ('box', 8)]

Within one short book by one author, 11.7% of the second half’s tokens never appeared in the first half. They aren’t exotic. “Wood”, “arms”, “iron” and “box” are ordinary words that the story just hadn’t needed yet. “Slower” is missing even though “slow” was seen. A word tokenizer has nothing to do with these except map them all to a single “unknown” ID, and then the model can neither read them nor ever write them.

A bigger vocabulary built from more text helps, but it never closes the gap. There are names, typos, numbers, code, new words, other languages. And a word vocabulary treats “slow”, “slower” and “slowly” as three unrelated IDs, even though they share most of their letters and most of their meaning.

Byte-pair encoding, by hand

Byte-pair encoding (BPE) sits between the two. It starts from characters, so nothing is ever missing, and then learns to glue together the pairs of symbols that occur most often. Frequent words end up as a single token. Rare words stay as several smaller pieces, and those pieces are reused across many words.

Training is one loop:

1. Write every word as a sequence of single characters.
2. Count every pair of neighboring symbols across the whole corpus.
3. Take the most frequent pair, add the joined symbol to the vocabulary, and replace that pair everywhere.
4. Repeat until the vocabulary is the size wanted.

The list of merges, in the order they were learned, is the tokenizer. Here is that loop in Python, written to be read rather than to be fast:

from collections import Counter

def pair_counts(words):
    """Count adjacent symbol pairs. `words` maps a tuple of symbols to how often it occurs."""
    counts = Counter()
    for symbols, freq in words.items():
        for a, b in zip(symbols, symbols[1:]):
            counts[(a, b)] += freq
    return counts

def merge_pair(words, pair):
    """Replace every adjacent occurrence of `pair` with one joined symbol."""
    joined = pair[0] + pair[1]
    out = {}
    for symbols, freq in words.items():
        new, i = [], 0
        while i < len(symbols):
            if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
                new.append(joined)
                i += 2
            else:
                new.append(symbols[i])
                i += 1
        out[tuple(new)] = out.get(tuple(new), 0) + freq
    return out

def train_bpe(text, num_merges, verbose=False):
    words = Counter(tuple(w) for w in text.split())
    merges = []
    for step in range(1, num_merges + 1):
        counts = pair_counts(words)
        if not counts:
            break
        best = max(counts, key=counts.get)   # ties go to the pair seen first
        merges.append(best)
        words = merge_pair(words, best)
        if verbose:
            ...  # print the merge and the corpus after it
    return merges

The corpus is stored as a count of distinct words, so “the” appearing four times is counted once with a weight of four. zip(symbols, symbols[1:]) walks the neighboring pairs: for ('t', 'h', 'e') it yields ('t', 'h') and ('h', 'e'). When two pairs tie, Python’s max returns the first one it met, which makes every run come out the same.

The corpus is ten words: “the cat sat on the mat the rat sat on the hat”. Its alphabet is the ten letters t, h, e, c, a, s, o, n, m, r. Before any merge, “the” appears 4 times, “sat” and “on” twice each, and “cat”, “mat”, “rat” and “hat” once each.

The pairs can be counted by hand. The pair a t appears in cat, sat, sat, mat, rat and hat: 6 times. t h and h e appear once in each “the”: 4 times each. Nothing else comes close. So the first merge is a + t. Running five merges prints each step:

merge 1: 'a' + 't' -> 'at'  seen 6 times
    t h e (x4)  c at  s at (x2)  o n (x2)  m at  r at  h at
merge 2: 't' + 'h' -> 'th'  seen 4 times
    th e (x4)  c at  s at (x2)  o n (x2)  m at  r at  h at
merge 3: 'th' + 'e' -> 'the'  seen 4 times
    the (x4)  c at  s at (x2)  o n (x2)  m at  r at  h at
merge 4: 's' + 'at' -> 'sat'  seen 2 times
    the (x4)  c at  sat (x2)  o n (x2)  m at  r at  h at
merge 5: 'o' + 'n' -> 'on'  seen 2 times
    the (x4)  c at  sat (x2)  on (x2)  m at  r at  h at

Each line under a merge is the corpus after it. Merge 2 is a tie: t h and h e both have 4, and t h was seen first. After merge 3, “the” is a single symbol. Merge 4 shows that merges build on earlier merges: sat is made from s and the at learned in merge 1. After merge 5, every pair left in the corpus appears exactly once, so further merges would only memorize single words.

The vocabulary grew from 10 symbols to 15:

The last column is the number of tokens the ten words take. It drops from 34 to 16 because each merge removes one symbol per occurrence of its pair.

To tokenize a new word, the merges are replayed on it in the order they were learned:

def bpe_encode_word(word, merges):
    symbols = tuple(word)
    for pair in merges:
        symbols = next(iter(merge_pair({symbols: 1}, pair)))
    return list(symbols)

for w in ["that", "cats", "rat", "bat", "dog"]:
    print(w, bpe_encode_word(w, merges))

that ['th', 'at']
cats ['c', 'at', 's']
rat ['r', 'at']
bat ['b', 'at']
dog ['d', 'o', 'g']

“That” was never in the corpus, yet it comes out as two familiar pieces. “Cats” reuses at. Order matters: on “that”, merge 1 forms at first, which uses up the a, and merge 2 then joins t h. Merge 3 would want th + e, and there is no e, so the result is th, at.

“Bat” and “dog” show the remaining flaw. The letters b, d and g aren’t in this tokenizer’s ten-letter alphabet, so those pieces have no ID at all. Starting from characters only guarantees full coverage if the starting alphabet contains every character that can ever show up. For English letters that’s easy. For all of Unicode it isn’t, and GPT-2’s answer to that comes below.

The same loop on a whole book

The same code runs unchanged on the full book. A thousand merges take about seven seconds:

merges = train_bpe(book, num_merges=1000)
print(["".join(m) for m in merges[:20]])
print(["".join(m) for m in merges[-8:]])

['th', 'in', 'the', 'an', 're', 'at', 'er', 'on', 'ed', 'en', 'and', 'st', 'of', 'ou', 'ing', 'as', 'me', 'to', 'it', 'or']
['remember', 'ly.', 'aw', '—a', 'gave', 'colour', 'pain', 'dest']

The early merges are the most common fragments of English: th, in, the, ed, and, ing. By merge 1,000 the merges are whole words (remember, gave, colour, the book’s British spelling) and odd fragments. New words now split into reasonable parts:

Traveller ['Traveller']
Morlocks ['Morlocks']
unbelievably ['un', 'belie', 'v', 'ably']
fourth-dimensional ['four', 'th', '-', 'dim', 'ension', 'al']
slowly. ['slow', 'ly.']
slowly ['slow', 'ly']

“Traveller” and “Morlocks” are frequent in this book, so they became single tokens. “Unbelievably” doesn’t occur, but it splits into un, belie, v, ably. On this book the tokenizer averages 2.71 characters per token (146,799 non-space characters became 54,200 tokens), compared with 1 for characters.

Two of the late merges point at a real problem: ly. and —a. Because this toy splits only on spaces, punctuation stays glued to words, and the trainer spends vocabulary slots on “ly” followed by a period, a dash followed by “a”, and so on. The GPT-2 paper describes the same thing at scale. Plain BPE ended up with many versions of common words like “dog”, such as “dog.” “dog!” and “dog?”. Its fix was to stop merges from crossing between letters, digits and punctuation.

GPT-2’s tokenizer

GPT-2’s tokenizer is byte-pair encoding with two changes and a fixed, published vocabulary. OpenAI’s tiktoken library ships it:

import tiktoken
enc = tiktoken.get_encoding("gpt2")
print(enc.n_vocab)
ids = enc.encode("The cat sat on the mat.")
print(ids)
print([enc.decode([i]) for i in ids])
print(enc.decode(ids))

50257
[464, 3797, 3332, 319, 262, 2603, 13]
['The', ' cat', ' sat', ' on', ' the', ' mat', '.']
The cat sat on the mat.

get_encoding("gpt2") loads the vocabulary and merge list (downloaded once and cached). encode turns a string into a list of IDs, and decode turns a list of IDs back into a string.

Bytes, not characters

The first change fixes the “bat” and “dog” problem. GPT-2 runs BPE over the bytes of the UTF-8 encoding instead of over characters. There are only 256 possible byte values, so a starting alphabet of 256 tokens covers every possible string in every language, plus emoji, plus binary junk. The GPT-2 paper gives the reason: a character-level BPE that covered all of Unicode would need a base vocabulary of over 130,000 symbols before learning a single merge.

The vocabulary size, 50,257, is exactly the sum of three parts:

The 50,000 merges are the published merges.txt file, one pair per line, and its first line is a space followed by “t”. IDs 0 to 255 are the single bytes, IDs 256 to 50,255 are the merges in the order they were learned, and 50,256 is the special token <|endoftext|>. Decoding a few IDs one at a time shows the layout:

for i in [0, 32, 198, 220, 256, 262, 50255, 50256]:
    print(i, enc.decode_single_token_bytes(i))

0 b'!'
32 b'A'
198 b'\n'
220 b' '
256 b' t'
262 b' the'
50255 b' gazed'
50256 b'<|endoftext|>'

decode_single_token_bytes returns the raw bytes behind one ID. The single bytes are numbered in an unusual order, with printable characters first, so ! is 0 and a space is 220. ID 256 is the first merge GPT-2 learned (space + “t”). ID 262 is " the", and the very last merge is " gazed".

Since everything is bytes, nothing is ever unknown. Made-up strings, accents, other scripts and emoji all encode:

“日本” is two characters, each three bytes long in UTF-8, and the tokens cut straight through the middle of each character. The emoji is one character of four bytes, split two and two. Such a token on its own isn’t valid text (decoding just ID 33768 gives the replacement character �), but the full sequence decodes back to the exact original. The cost of rare scripts is length, not failure.

Splitting before merging

The second change keeps merges from crossing letters, digits and punctuation. Before BPE runs, the text is cut into chunks by a regular expression, and merges only happen inside a chunk. OpenAI’s GPT-2 code uses this pattern:

's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+

Read left to right, a chunk is one of: a common English contraction ending ('s, 't, 're…); an optional space followed by letters (\p{L} is any Unicode letter); an optional space followed by digits (\p{N}); an optional space followed by other symbols; or a run of whitespace. So “dog.” is always two chunks, dog and ., and BPE can never learn a “dog.” token.

The pattern also explains how GPT-2 handles spaces. A space isn’t a separate token in normal text. It sticks to the front of the word that follows it:

'The cat sat on the mat.' -> 'The' ' cat' ' sat' ' on' ' the' ' mat' '.'
'dog. dog! dog?'          -> 'dog' '.' ' dog' '!' ' dog' '?'
'hello   world'           -> 'hello' ' ' ' ' ' world'
'hello\n\nworld'          -> 'hello' '\n' '\n' 'world'

That has consequences. “the” (1169) and " the" (262) are different tokens, as are “Hello” (15496), " Hello" (18435) and “hello” (31373). “HELLO” isn’t in the vocabulary at all and becomes HE, LL, O. Nothing in the IDs says these are related; the model has to learn it. Extra spaces between words become single-space tokens (220), with the last space still attached to " world". In the last line the word after a newline has no space at all, so it’s “world” (6894), not " world" (995).

Numbers, code, and the special token

Digits are chunked separately from everything else, but within a run of digits the merges are whatever was frequent in GPT-2’s training text:

'12345 + 678 = 13023' -> '123' '45' ' +' ' 6' '78' ' =' ' 130' '23'

“12345” is 123 + 45, and “678” is 6 + 78. Digits are grouped by frequency, not by place value, so the model never sees a consistent ones-tens-hundreds layout.

Indented code costs a token per leading space. A function body indented eight spaces:

'def f(x):\n        return x + 1\n'
-> 'def' ' f' '(' 'x' '):' '\n' ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' return' ' x' ' +' ' 1' '\n'

Seven tokens of single spaces, then " return" carries the eighth. OpenAI’s later cl100k_base encoding, also in tiktoken, has a single token for those seven spaces; GPT-2’s vocabulary doesn’t.

The one special token, <|endoftext|>, marks the boundary between unrelated documents when many are joined into one training stream. It is a single ID, 50256, but only when the caller explicitly allows it. By default tiktoken refuses to encode it, so a document that happens to contain the literal text can’t inject a document boundary:

enc.encode("<|endoftext|>")
# ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.

enc.encode("Hello<|endoftext|>World", allowed_special={"<|endoftext|>"})
# [15496, 50256, 10603]

enc.encode_ordinary("<|endoftext|>")
# [27, 91, 437, 1659, 5239, 91, 29]   ordinary text: '<', '|', 'end', 'of', 'text', '|', '>'

The rest of this track wraps tiktoken in two small helpers so the special token is always allowed in training text:

gpt2 = tiktoken.get_encoding("gpt2")
END_OF_TEXT = "<|endoftext|>"

def encode(text):
    return gpt2.encode(text, allowed_special={END_OF_TEXT})

def decode(ids):
    return gpt2.decode(list(ids))

Token counts for real text

A token is not a word. For English prose with this tokenizer, a common rule of thumb is about four characters per token, and the book bears it out:

The book is 45,259 tokens: a quarter of its character count, and only about 40% more than its word count, with no word ever unknown. The Python file is much less efficient because of indentation, brackets and identifiers that aren’t English words.

Language matters too. The same short sentence in three languages:

'The machine stopped.'   4 tokens: 'The' ' machine' ' stopped' '.'
'Die Maschine hielt an.' 8 tokens: 'Die' ' Mas' 'ch' 'ine' ' hi' 'elt' ' an' '.'
"La machine s'arrêta."   8 tokens: 'La' ' machine' ' s' "'" 'arr' 'ê' 'ta' '.'

GPT-2’s vocabulary was learned from mostly English web text, so English gets the long, whole-word tokens. German and French take twice as many for the same meaning. That ratio matters in practice: a model’s window and its cost are both counted in tokens, not words.

Where this leaves things

Text now goes in and a list of integers comes out, reversibly, for any string at all. The Time Machine is 45,259 IDs drawn from a vocabulary of 50,257.

But an ID is only a label, the way a locker number is. ID 3862 is " Time", ID 3863 is " maybe", and " time" is 640. Nothing about the numbers says that " Time" and " time" are related, or that " cat" is closer to " dog" than to " the". The order of IDs is just the order merges were learned in, and the single bytes aren’t even in byte order. If the model did arithmetic on the IDs themselves, it would be doing arithmetic on locker numbers.

The model needs a representation where related tokens can sit near each other and where there is room to store what a token means. That is a vector of learned numbers for each ID, and it is where chapter 2 begins.