Ashok G

Linux, Observed · Linux 7.3 · Draft

0 — Why build what you could download

Requirements: a disposable x86-64 Linux virtual machine with sudo. Use a VM so a mistaken install can’t touch your daily machine.

You can apt install a working Linux in minutes. Building one from source takes much longer. The reason to do it anyway is not purity and not nostalgia. It is that a system you assembled is a system you can reason about, and a system you can reason about is one you can debug at 3 a.m., secure with a straight face, and change without fear. Everything in this series is in service of that one capability.

Versions: this series was checked against Linux 7.3 and the component versions in chapter 1, as of 30 October 2026. Kernel, Traced uses 6.12 LTS, so the two tracks differ in places. Versions date quickly, so check upstream before you start.

Step 1: install the tools

These are Debian and Ubuntu package names. The first line comes from the kernel’s quick-build guide, and the second adds what this series needs:

sudo apt install bc binutils bison dwarves flex gcc g++ git make openssl pahole perl-base libssl-dev libelf-dev
sudo apt install gawk texinfo rsync cpio xz-utils qemu-system-x86 strace bpftrace

Expected: gcc --version, pahole --version and qemu-system-x86_64 --version each print a version. Compare them with the kernel’s minimums.

Step 2: check the tracer

sudo bpftrace -e 'BEGIN { printf("ok\n"); exit(); }'

Expected: it prints ok. If it fails, carry on anyway, and come back after chapter 11. Until then, bpftrace always runs on this host VM, never inside the system you’re building.

Step 3: create the tree

Your build directory is /lo, for Linux, Observed. Creating it needs sudo once. After that, stay an ordinary user, because running as root hides permission mistakes:

sudo mkdir /lo && sudo chown "$USER": /lo
mkdir -p /lo/sources /lo/cross /lo/rootfs /lo/out
PathHolds
/lo/sources/Pinned tarballs, one per package
/lo/cross/The cross-toolchain (chapter 1)
/lo/rootfs/The system being built
/lo/out/Images, logs, measurements
/lo/manifest.tomlWhat you depend on

Expected: touch /lo/out/t && rm /lo/out/t works without sudo.

Step 4: start the manifest

This is the only file in the series I’ll insist on. For every package, record the upstream URL, version, SHA-256, licence and chapter.

[[package]]
name    = "musl"
version = "1.2.6"
url     = "https://musl.libc.org/releases/musl-1.2.6.tar.gz"
sha256  = "<the hash of the file you verified>"
licence = "MIT"
chapter = 1

Verify each tarball’s upstream signature, then record the hash of the file you verified. Never copy a hash from a blog, including this one.

Keep security in mind too. musl 1.2.6 has upstream patches for known issues, so apply them (musl).

By chapter 10, this file is what makes the whole build reproducible. The habit behind it, write down exactly what you depend on, separates systems you can trust from systems you merely run.

Step 5: record a baseline

Before building anything, look at what you’re about to replace:

strace -c -f ls / 2>&1 | tail -25
strace -f ls / 2>&1 | grep -c openat

The second number is how many openat calls ls makes to list a directory. Save both results to /lo/out/baseline.txt. Your number is the result. There’s no right answer to compare it with yet.

Some calls won’t appear at all. The vDSO lets libc serve frequent calls like clock_gettime without entering the kernel (vdso(7)).

Step 6: a first instrument

Run this in one terminal, and type commands in another:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s -> %s\n", comm, str(args.filename)); }'

This shows every program run, every program those run in turn, and shell helpers that are easy to miss.

If it fails

  • Permission denied on /lo: the chown was skipped.
  • strace: command not found: Step 1 didn’t finish.

Principle: you can’t reason about what you can’t see

Every system-design conversation eventually arrives at a claim: “this is fast enough”, “this is isolated”, “this can’t happen”. The claim is worth exactly as much as the observation behind it. A distro ls isn’t slow or unsafe, but if you’ve never looked, you don’t know it’s fine; you assume it. The purpose of building from source with the tracer running is to replace assumption with observation at every layer.

As you work through the series, record three numbers at each stage: image size, boot-to-shell time, and syscalls to first prompt. Keep them with your manifest hash, so that “it’s smaller and simpler” is a claim you can check.