3 — Userland in one binary

Requirements: chapter 1, with BusyBox 1.38.0 unpacked at /lo/build/busybox-1.38.0.

BusyBox or toybox

A multicall binary is one program that acts as many. /bin/ls is a symlink to it, and it looks at the name it was run under.

toybox’s pending directory says its contents “may or may not work”, and that everything there defaults to n (README).

This series uses BusyBox for the booted system, because everything it needs is finished. toybox is the alternative if the licence matters to you. Android is the reason toybox exists, by its author’s account (FAQ).

Hands-on

Step 1: configure

Always edit the config after make defconfig, which resets options:

cd /lo/build/busybox-1.38.0
make defconfig
sed -i 's/^# CONFIG_STATIC is not set/CONFIG_STATIC=y/' .config
sed -i 's/^CONFIG_TC=y/# CONFIG_TC is not set/' .config

CONFIG_TC is off because tc.c uses TCA_CBQ_MAX, which Linux 7.3’s headers no longer define.

Expected: grep -E '^CONFIG_STATIC=|CONFIG_TC' .config shows CONFIG_STATIC=y, and TC unset.

Step 2: build and install

make -j"$(nproc)" CROSS_COMPILE=$TGT-
make CROSS_COMPILE=$TGT- CONFIG_PREFIX=/lo/rootfs install

Expected: $TGT-readelf -l busybox | grep -c INTERP prints 0, and ls -l /lo/rootfs/bin/sh shows a link to busybox.

Step 3: test it without booting

/lo/rootfs/bin/busybox --list | wc -l
/lo/rootfs/bin/busybox sh -c 'echo shell-ok'

Expected: a non-zero count, and shell-ok.

Step 4: see the gap in toybox

cd /lo/build/toybox-0.8.15
LDFLAGS=--static CROSS_COMPILE=$TGT- make defconfig toybox
./toybox | tr ' ' '\n' | grep -xE 'sh|ip|init|getty|dhcp|route'

Expected: none of the six should be listed. That shows the gap.

Step 5: profile a pipeline

The kernel sets a process’s comm from the name it ran (exec.c), so filter on cat, grep and wc, not busybox:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_pipe2,tracepoint:syscalls:sys_enter_execve,tracepoint:syscalls:sys_enter_read,tracepoint:syscalls:sys_enter_write /comm == "cat" || comm == "grep" || comm == "wc"/ { @[comm, probe] = count(); }'

In another shell, run this, using the absolute path so it’s really your binary:

/lo/rootfs/bin/busybox sh -c 'cat /proc/cpuinfo | grep MHz | wc -l'

Look for: which stages execve. A forked stage still calls execve, unless the shell runs a built-in without forking. Count it, rather than assuming.

Step 6: repeat chapter 0’s experiment

strace /lo/rootfs/bin/ls / 2>&1 | grep -c openat

Compare it with your /lo/out/baseline.txt. Can you explain every call?

If it fails

• TCA_CBQ_MAX undeclared: disable CONFIG_TC.
• CROSS_COMPILE errors: $TGT-gcc isn’t on PATH.
• The binary is dynamic: CONFIG_STATIC didn’t stick, perhaps because make defconfig ran after your edit.

Principle: interfaces outlive implementations

Your ls is not GNU’s ls, and for callers that stay inside POSIX, nothing can tell. The POSIX interface (argv, stdin and stdout, exit codes, the filesystem) is the contract; the implementation behind it is replaceable, and was replaced, by you, with a symlink. Systems that survive are the ones whose boundaries are interfaces, not implementations. GNU-only flags do break, which is what leaving the interface looks like.

toybox’s pending directory shows the other half: a list of names isn’t a finished implementation.