AI Engineering
One assistant, built up from its failures.
This track follows a single project from start to finish: an assistant that answers questions about a set of documents, here the chapters of this site. It runs on one Mac, with an open model served locally, so every output, timing and failure rate shown was produced by a real run.
Each chapter starts from what the previous one couldn’t do. A model that answers fluently turns out not to know the documents, so they have to be found and passed in. Retrieval that works most of the time fails in specific ways, and each fix needs a way to measure whether it helped. An assistant that can only read gains tools, then loses track of long tasks, then gets turned against itself by a document that contains instructions. The last chapters cover what changes with real users and when changing the model itself is worth it.
The chapters on how models work inside sit elsewhere on the site: the GPU track covers attention, the KV cache and inference hardware, and the essays on machine learning and neural networks cover training. This track assumes those exist and works one layer up.
Chapters
- 1What the model seesOne question sent to a local model, followed all the way down: the template, the tokens, the probability of each next token, how one gets picked, and the window it all has to fit in.Draft
- 2Answers a program can useAsking a local model for JSON, measuring how often the reply parses, and what JSON mode, schemas, validation and retries each fix, for classification and extraction.Draft
- 3It doesn't know your documentsAsking a local model about this site's own chapters, measuring how often it invents an answer, then trying to paste the whole site into the prompt and watching the context window and long inputs fail.Draft
- 4Meaning as numbersTurning passages and questions into vectors with nomic-embed-text, comparing them with cosine similarity, and measuring what that comparison captures and what it misses.Draft
- 5Finding the nearest ones quicklyBrute-force vector search measured, small IVF and HNSW indexes built in numpy and compared on recall, work and memory, then the same search in PostgreSQL with pgvector, including the filtering trap.Draft
- 6The first pipelineLoading the site's markdown, cutting it into passages, embedding and storing them in PostgreSQL, retrieving the top five for a question and having llama3 answer with citations, then collecting the ways it fails.Draft
- 7Why it still gets things wrongEach failure of the first pipeline traced to its cause, and each fix (chunking, hybrid search, reranking, query rewriting and metadata filters) measured against the failure it was meant to repair.Draft
- 8Knowing whether it helpedA labeled question set built from the site, retrieval and answer metrics measured for the baseline and every fix, how noisy those numbers are at sixty questions, the measured biases of a model used as judge, and a regression check.Draft
- 9Letting it actQuestions that need a count or a calculation instead of a passage: tool calling with Ollama, an agent loop in plain Python, how tool design changes accuracy, and measured tool-call accuracy for two small models.Draft
- 10When the agent loses the threadThree multi-step tasks given to small local models, the specific ways they went wrong, and what a bigger context, memory trimming, planning, self-checking, step limits and extra agents changed when measured.Draft
- 11A standard plug for toolsThe Model Context Protocol from the wire up: a server that exposes the site search, a client that uses it, and the actual JSON-RPC messages that pass between them.Draft
- 12When the documents fight backA planted paragraph in one document turns the assistant against its user; how often that works, why it's hard to stop, and what each defense actually changes when measured.Draft
- 13Running it for realWhere the time and money go in one answer, what caching saves and what it breaks, a gateway that survives a failing model, and the traces, checks and signals that show what's going wrong.Draft
- 14When prompting isn't enoughWhen changing the model's weights beats prompting or retrieval, how LoRA and QLoRA make that cheap, and a real LoRA run on a 0.5B model measured before and after.Draft