2 — A libc, and what it costs

Requirements: chapter 1. Work in /lo/out, with $TGT exported.

What a loader is

A dynamic program doesn’t start at its own code. Its ELF file names an interpreter, and the kernel maps that into memory during execve and runs it first. The interpreter loads libraries and connects them, and only then calls your main().

In glibc, the loader (ld-linux) and the library (libc.so.6) are separate files. In musl they’re one file. ld-musl-x86_64.so.1 is a link to libc.so (musl design).

This series’ decision: static by default for everything in /bin and /sbin. With one multicall binary (chapter 3), one copy of libc serves the whole userland. There’s no loader at startup, no LD_PRELOAD surface, and no “which libc did this pick up?” question.

How musl differs from glibc

These differences cause real “works on my distro” bugs (functional differences):

Hands-on

Step 1: compare the two files

ls -l hello-static hello-dynamic
$TGT-readelf -l hello-static | grep -c INTERP
$TGT-readelf -l hello-dynamic | grep interpreter
$TGT-readelf -d hello-dynamic | grep NEEDED

Expected: the static count is 0, the dynamic file names its interpreter, and NEEDED lists libc.so.

Step 2: prove the loader is libc

If the loader is run under a name ending in ldd, it acts as ldd (dynlink.c):

ls -l /lo/rootfs/lib/ld-musl-x86_64.so.1
ln -sf /lo/rootfs/lib/ld-musl-x86_64.so.1 /lo/out/ldd-musl
/lo/out/ldd-musl ./hello-dynamic

Expected: the link points to libc.so, and the output lists the loader’s own path.

Step 3: trace both

strace ./hello-static 2>&1 | tee static.trace | wc -l
strace /lo/rootfs/lib/ld-musl-x86_64.so.1 ./hello-dynamic 2>&1 | tee dynamic.trace | wc -l

Look for:

• writev, not write. musl’s stdio uses writev (__stdio_write.c).
• An ioctl first, when output goes to a terminal, as stdio decides on line buffering.
• set_tid_address during startup.

Running through the loader explicitly makes it open hello-dynamic itself. That’s a side effect of this method. In normal use, the kernel maps the interpreter, so you won’t see an openat for it. The true trace appears in chapter 4, inside the finished system.

Step 4: hijack a function

1. Create a program and a hook. It must call write() directly, because puts doesn’t go through that function on musl:

   cat > w.c <<'E'
   #include <unistd.h>
   int main(void){ write(1,"real\n",5); return 0; }
   E
   cat > evil.c <<'E'
   #include <unistd.h>
   #include <sys/syscall.h>
   ssize_t write(int f,const void*b,size_t n){ return syscall(SYS_write,f,"hook\n",5); }
   E

2. Build and run both versions:

   $TGT-gcc w.c -o w-dyn && $TGT-gcc -static w.c -o w-static
   $TGT-gcc -shared -fPIC evil.c -o evil.so
   LD_PRELOAD=$PWD/evil.so /lo/rootfs/lib/ld-musl-x86_64.so.1 ./w-dyn
   LD_PRELOAD=$PWD/evil.so ./w-static

3. Expected: only the dynamic one prints hook. This is why an init should be static.

Step 5: measure the cost

perf stat -e page-faults,instructions ./hello-static

Repeat through the loader. Write down your numbers, and form your own hypothesis about any difference.

If it fails

• hello-dynamic: not found: you skipped the loader path.
• Error loading shared library: a library is missing from musl’s path file.
• evil.so is ignored or fails: build it with $TGT-gcc, not the host compiler.

Principle: indirection is never free; it is sometimes worth it

Dynamic linking buys shared memory pages across processes, in-place security updates of libc, and plugins. It costs startup work, a loader to trust, and a question about which library actually ran. Neither answer is right in general; the trace tells you the price, and your system’s needs tell you whether to pay it. Note that plugins are a weaker benefit on musl, where dlclose does nothing and there’s no NSS.