Linux, Observed

Build it. Then watch it run. Then name the principle.

There are good guides to building Linux from source — Linux From Scratch has been the standard for decades — and there are good books on system design. What I couldn’t find was the thing in between: a build where, after every component, you stop and look at it running with the instruments the kernel already carries (ftrace, perf, bpftrace, /proc, /sys), and where each thing you see is tied to a principle that shows up again in every large system you will ever design.

That’s this series. Twelve chapters, each with three parts:

1. Build — what gets compiled or configured in this chapter, from upstream source, and the decision that mattered.
2. Observe — a short experiment, usually a bpftrace one-liner or an ftrace session, that shows what the new component actually does: which syscalls, which kernel paths, how long, how often.
3. Principle — one system-design idea the observation makes concrete. Not a slogan; the specific thing the trace just showed you.

The result is a working, bootable Linux of your own: musl, a static BusyBox userland, your own init, and a kernel you configured line by line — and, more usefully, the habit of asking “what is this actually doing?” and knowing how to find out.

Background

This assumes comfort at a shell and the ability to read C. No kernel experience is needed.

What’s unique about it

• Observation is not optional. Most build guides stop at “it compiles”. Here, “it compiles” is the halfway point; the chapter isn’t done until you’ve traced it.
• These are the instruments used in production. No special instrumentation builds. If you can trace it here, you can trace it on a server at 3 a.m.
• Every build is measured. Record three numbers as you go: image size, boot-to-shell time and syscalls to first prompt, with the hash of the manifest that produced them.
• The principles accumulate. There’s a running list of principles: twelve ideas, each anchored to a trace you ran yourself.

Setup

You need a disposable x86-64 Linux VM, and a weekend or more of compile time. The series is pinned to Linux 7.3, while Kernel, Traced uses 6.12 LTS. Chapter 4 boots a kernel before chapter 5 builds yours, so either use a distro kernel there or do chapter 5 first.