Kernel, Traced

Read the path. Trace the path. Change the path.

Linux, Observed builds a system and watches it from the outside. This series goes inside. Each chapter takes one thing the kernel does — open a file, create a process, pick the next task, fault a page in, send a packet, verify a BPF program — and follows it through the actual source, file and function, from the system-call boundary to the point where the work happens. Then it does two things most kernel write-ups don’t:

• Trace it live. ftrace’s function graph tracer and a few BPF probes, pointed at the exact functions you just read, on a kernel you built. The call graph on your screen is the call graph in the source, with timings.
• Change it. Every chapter ends with a small modification on that path — a BPF program attached at a hook you can now locate precisely, or a kernel patch of a few lines that adds a tracepoint or alters a decision — built, booted in QEMU, and observed. You haven’t understood a path until you’ve altered it and watched the alteration happen.

The kernel is a 30-million-line codebase. The trick is that any one path through it is a few dozen functions, and the paths you care about are the ones your programs exercise every millisecond. Read twelve of them properly and the rest of the tree stops being a wall.

Format of every chapter

1. The question — the one-sentence thing we want to know. “When a process calls connect(), where exactly does the kernel decide where the SYN goes?”
2. The path — the annotated call chain: file.c:function() at each step, what it does in one line, and the one data structure it works on. Kernel version pinned (6.12 LTS unless stated); line numbers avoided because they rot, function names kept because they don’t (much).
3. Trace it — a step-by-step lab with the function_graph filter or bpftrace script for that path, and what to look for.
4. Change it — the patch or BPF program, with the prediction you should test.
5. The takeaway — the one design idea the path embodies, stated once.

Prerequisites

Start with chapter 0, which builds the kernel and virtual machine every experiment uses. You can read C and aren’t afraid of a macro. You have the source tree checked out at the tag you’re running, and cscope or an LSP (clangd with compile_commands.json from scripts/clang-tools/gen_compile_commands.py) so you can jump to definitions. That last one is not optional; reading the kernel without jump-to-definition is reading it blind.

The map

Keep one text file, map.txt, throughout: every function you’ve read, one line each — file, name, what it does, which chapter. By chapter 12 it will be the most useful kernel document you own, because it’s the kernel you have walked. There’s a version at /kernel/map as a starting point and a cross-check.

The chapters name the kernel version they were checked against.