11 — The instrument panel

Requirements: chapter 5’s kernel, booted, with /sys and /proc mounted.

The order

Each tool adds dependencies, so use them in this order:

ftrace fits this series best: it’s an instrument the kernel already carries.

The bpftrace problem. Its dependency stack (developers.md) conflicts with this series’ small static system. The honest answer is to trace from the host or a separate tooling image, not to force it into your musl system. If you want something small, ply suits embedded use (Brendan Gregg).

BTF isn’t just about portability. Since 0.25, bpftrace’s tracepoint args require it (migration guide). And since 0.24, a failed probe attach exits the program, so one bad probe stops the whole script.

Hands-on

Step 1: check the kernel

mount -t tracefs nodev /sys/kernel/tracing 2>/dev/null
mount -t debugfs nodev /sys/kernel/debug 2>/dev/null
ls -l /sys/kernel/btf/vmlinux
wc -l /sys/kernel/tracing/available_events

Expected: the BTF file exists, and the event count isn’t zero. No BTF file means DEBUG_INFO_BTF was dropped. Note the file’s size.

Step 2: use ftrace with only a shell

cd /sys/kernel/tracing
echo 0 > tracing_on
echo do_sys_openat2 > set_graph_function
echo function_graph > current_tracer
echo 1 > tracing_on
cat /etc/hostname > /dev/null
echo 0 > tracing_on
head -40 trace
echo nop > current_tracer
echo > set_graph_function

Expected: an indented call graph of the open path. If writing the function name fails, run grep -w do_sys_openat2 available_filter_functions and pick a listed symbol.

Step 3: trace an event

echo 1 > events/sched/sched_process_exec/enable
/bin/busybox true
grep sched_process_exec trace | tail -3
echo 0 > events/sched/sched_process_exec/enable

Expected: a line for your command, with a filename field.

Step 4: build perf

Start on the host:

cd /lo/build/linux-7.3/tools/perf
make LDFLAGS=-static NO_LIBTRACEEVENT=1 CC=$TGT-gcc

A libelf failure for the cross target is expected, until elfutils is in your sysroot. Set NO_LIBELF=1 for a minimal perf, or use a host build. Then, in the guest:

perf stat -e page-faults,context-switches /bin/busybox true

Expected: a counter summary prints.

Step 5: use bpftool

Build it from tools/bpf/bpftool, then:

sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c > /lo/out/vmlinux.h
grep -c 'struct task_struct {' /lo/out/vmlinux.h
sudo bpftool prog list

Expected: the count is 1, and prog list runs, even with nothing loaded.

Step 6: trace the tracer

On the host VM:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_bpf { @[comm, args.cmd] = count(); }'

In a second terminal, start another bpftrace one-liner, then stop it after a few seconds.

Look for: rows for bpftrace with numeric cmd values. Map them with enum bpf_cmd in bpf.h. While it’s running, sudo bpftool prog list shows the loaded programs. BPF programs are kernel objects with IDs, like everything else.

Step 7: measure what looking costs

1. Write your-syscall-loop, a program that loops on syscall(SYS_getppid).

2. Measure it with no tracer, then with a tracepoint attached:

   perf stat -r 5 -- /lo/out/your-syscall-loop

3. Publish both figures, with your kernel, CPU and VM type. Don’t quote a generic percentage. The cost depends on the path you measure.

If it fails

Principle: observability is a property of the system, not a tool you add

Every instrument in this series worked because the kernel was built to expose tracepoints, keep BTF, and allow safe programs to attach at well-defined points. Those were decisions made in chapter 5, before anything ran. Observability bolted on afterwards is always partial; designed in, it’s a property of the whole: every subsystem has named events, stable identities, and a cost of observation you can measure. Build that into your systems at the point where you decide what they are, not after the incident.

Prefer tracepoints where you can, because “dynamic tracing is an unstable API” (Brendan Gregg).

This closes the series. Everything from here, the next kernel, the next distro, the next production system, is the same loop: build it, watch it, name what you saw.