Ashok G

Linux From the Metal Up

Part 3 — How the kernel is built

3.1 Source tree layout

linux/
├── arch/          per-architecture code: boot, entry, mm, KVM, DTs (arch/x86, arch/arm64, arch/riscv ...)
├── block/         block layer, blk-mq, I/O schedulers (mq-deadline, BFQ, kyber)
├── certs/         module-signing keys, system keyrings
├── crypto/        crypto API and algorithms
├── Documentation/ reStructuredText docs → docs.kernel.org
├── drivers/       ~2/3 of all code: every device driver, grouped by subsystem
├── fs/            VFS core + every filesystem (ext4/, xfs/, btrfs/, proc/, ...)
├── include/       headers: linux/ (internal), uapi/ (user ABI), asm-generic/
├── init/          main.c (start_kernel), initramfs unpacking, Kconfig for "General setup"
├── io_uring/      io_uring implementation
├── ipc/           SysV IPC, POSIX mqueues
├── kernel/        core: sched/, fork.c, exit.c, signal.c, irq/, time/, bpf/, trace/, locking/, rcu/, module/
├── lib/           generic helpers: string, rbtree, xarray, kobject, crc
├── mm/            memory management: page_alloc.c, slub.c, vmscan.c, filemap.c, mmap.c, huge_memory.c
├── net/           networking: core/, ipv4/, ipv6/, netfilter/, xdp/, unix/
├── rust/          Rust support: kernel crate, bindings, macros
├── samples/       example code (BPF, kprobes, rust, ftrace)
├── scripts/       build tooling: kconfig/, mod/ (modpost), checkpatch.pl, get_maintainer.pl, decode_stacktrace.sh
├── security/      LSMs: selinux/, apparmor/, landlock/, yama/, lockdown/
├── sound/         ALSA
├── tools/         userspace tools shipped with the kernel: perf, bpftool, testing/selftests, memory-model
├── usr/           initramfs generation (gen_init_cpio)
├── virt/          KVM common code
├── Kbuild, Kconfig, Makefile, MAINTAINERS

3.2 Kconfig: deciding what gets built

Every directory has a Kconfig file declaring symbols:

config EXT4_FS
	tristate "The Extended 4 (ext4) filesystem"
	select BUFFER_HEAD
	select JBD2
	select CRC16
	help
	  This is the next generation of the ext3 filesystem.
  • Types: bool (y/n), tristate (y/m/n), int, hex, string.
  • Dependencies: depends on hides a symbol unless its condition holds. select forces another symbol on, so use it carefully because it ignores that symbol’s dependencies. imply is a weak select. default sets the starting value.
  • The result is .config in the build directory, plus generated include/generated/autoconf.h (#define CONFIG_EXT4_FS 1, or CONFIG_EXT4_FS_MODULE) and include/config/auto.conf, which Makefiles consume.

Useful configuration targets:

TargetEffect
make defconfigArch default (arch/x86/configs/x86_64_defconfig)
make tinyconfigSmallest possible kernel; then add what you need
make localmodconfigStart from current config, disable every module not currently loaded (lsmod) — the fastest way to a lean kernel for this machine
make localyesconfigSame, but build those modules in (=y) — handy to avoid an initramfs
make olddefconfigTake an old .config, set new symbols to defaults without prompting
make menuconfig / nconfig / xconfigInteractive editors (/ searches symbols, shows dependencies)
make kvm_guest.configMerge a fragment that makes the kernel boot as a KVM/QEMU guest
scripts/config -e FOO -d BAR -m BAZScriptable edits to .config
make savedefconfigWrite a minimal defconfig containing only non-default choices

Inspect a running kernel’s configuration with zcat /proc/config.gz (if CONFIG_IKCONFIG_PROC=y) or /boot/config-$(uname -r).

3.3 Kbuild: how the Makefiles recurse

The top-level Makefile includes the arch Makefile (arch/x86/Makefile) and descends into directories listed in Kbuild files. Each directory’s Makefile or Kbuild uses a declarative vocabulary:

# fs/ext4/Makefile (abridged)
obj-$(CONFIG_EXT4_FS) += ext4.o
ext4-y := balloc.o bitmap.o dir.o file.o fsync.o ialloc.o inode.o \
          extents.o super.o ...
ext4-$(CONFIG_EXT4_FS_POSIX_ACL) += acl.o
  • obj-y: built in and linked into this directory’s built-in.a.
  • obj-m: built as a loadable module (.ko).
  • obj-$(CONFIG_FOO) expands to obj-y, obj-m or obj- (ignored) according to .config. One line handles all three cases.
  • foo-y := a.o b.o builds a composite object foo.o from several sources.
  • ccflags-y, subdir-ccflags-y, ldflags-y set per-directory flags.
  • Dependency tracking is exact. fixdep rewrites the compiler’s dependency output so that each object depends only on the specific CONFIG_* symbols it uses, not on all of .config. Change one option and only the affected files rebuild. Command lines are recorded in .*.cmd files, so a changed flag also triggers a rebuild.

3.4 From source to vmlinux to bzImage

 *.c ──gcc/clang──▶ *.o ──ar──▶ built-in.a (per dir) ──▶ vmlinux.a (all built-in code)
                                                         │
                         modpost (scripts/mod/) ◀─ vmlinux.o + module .o's → symbol checks,
                                                         │   Module.symvers, *.mod.c
                                                         ▼
     ld -T arch/x86/kernel/vmlinux.lds  ──▶ vmlinux  (ELF, uncompressed, with symbols)
        (kallsyms: link, extract symbols, relink 2–3 times so the
         embedded symbol table is stable; BTF generated by pahole if enabled)
                                                         │
     objcopy → strip → compress (gzip/xz/lz4/zstd) → arch/x86/boot/compressed/vmlinux
     + arch/x86/boot/setup.bin (real-mode/setup code) ──▶ arch/x86/boot/bzImage
  • vmlinux at the top of the tree is the debuggable ELF you load into GDB, crash or perf. It is not what you boot on x86.
  • System.map lists symbol addresses. With KASLR, the runtime addresses are offset from it.
  • bzImage (“big zImage”) is what the bootloader loads, usually installed as /boot/vmlinuz-<version>.

3.5 Building, installing, and a minimal kernel

Kernel build requirements, as of the 7.3 docs (Documentation/process/changes.rst): GNU C 8.1 or later, or Clang/LLVM 17.0.1 or later; binutils 2.30 or later; GNU make 4.0 or later; bash 4.2; flex 2.5.35; bison 2.0; bc; Python 3.9; pahole 1.26 for BTF; OpenSSL development headers for module signing; and, optionally, Rust 1.85.0 with bindgen 0.71.1.27

# Get the source (tarball or git)
git clone --depth=1 --branch v7.3 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux

# Out-of-tree build directory keeps the source clean
mkdir -p ../build-7.3
make O=../build-7.3 defconfig          # or: cp /boot/config-$(uname -r) ../build-7.3/.config && make O=../build-7.3 olddefconfig
make O=../build-7.3 menuconfig

make O=../build-7.3 -j"$(nproc)"                       # vmlinux, bzImage, modules
sudo make O=../build-7.3 modules_install               # → /lib/modules/7.3.0/
sudo make O=../build-7.3 install                       # → /boot (calls installkernel; distro hooks build initramfs)

# Other useful targets
make O=../build-7.3 -j"$(nproc)" bzImage               # just the kernel
make O=../build-7.3 M=fs/ext4                          # just one directory
make O=../build-7.3 headers_install INSTALL_HDR_PATH=/tmp/hdr   # UAPI headers for a libc
make O=../build-7.3 bindeb-pkg                         # .deb packages; also binrpm-pkg, tar-pkg
make LLVM=1 ...                                        # build with Clang/LLD instead of GCC

A minimal kernel recipe for a QEMU guest. This is the approach used in the quickstart in Part 4:

make O=../tiny tinyconfig
make O=../tiny kvm_guest.config
cd ../tiny
../linux/scripts/config \
  -e 64BIT -e PRINTK -e TTY -e SERIAL_8250 -e SERIAL_8250_CONSOLE \
  -e BINFMT_ELF -e BINFMT_SCRIPT -e BLK_DEV_INITRD -e RD_GZIP \
  -e DEVTMPFS -e DEVTMPFS_MOUNT -e PROC_FS -e SYSFS -e TMPFS \
  -e EARLY_PRINTK -e FUTEX -e EPOLL -e MULTIUSER
make olddefconfig
make -j"$(nproc)" bzImage
ls -lh arch/x86/boot/bzImage        # typically a couple of MiB

3.6 Cross-compiling

Kbuild cross-compiles natively. Set ARCH and either CROSS_COMPILE or LLVM=1:

# arm64 with a GNU cross toolchain
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- O=../build-arm64 defconfig
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- O=../build-arm64 -j"$(nproc)" Image modules dtbs

# riscv with LLVM (one Clang targets every arch)
make ARCH=riscv LLVM=1 O=../build-rv defconfig
make ARCH=riscv LLVM=1 O=../build-rv -j"$(nproc)"

# Boot it without hardware
qemu-system-aarch64 -M virt -cpu max -m 1G -nographic \
  -kernel ../build-arm64/arch/arm64/boot/Image -append "console=ttyAMA0"

3.7 Versioning and the release cycle

  • Mainline (Linus Torvalds). Each cycle runs about 9–10 weeks. A two-week merge window opens right after a release, and subsystem maintainers send pull requests for new features. Then come weekly -rc releases (usually rc1 through rc7, sometimes rc8) that only take fixes. Linux 7.3 followed this pattern: the merge window closed with rc1 at the end of August 2026, rc7 came out on 11 October, and the final release on 18 October.134 LWN called it a record-breaker on all fronts: 18,765 non-merge commits from 3,021 developers, 807 of them first-time contributors. The 7.4 merge window opened on 19 October.
  • Major numbers mean nothing. 6.19 was followed by 7.0 (April 2026) simply because Linus prefers to bump when the minor number gets large. Version numbers do not signal ABI breaks.2
  • Stable (Greg Kroah-Hartman and Sasha Levin). Fixes must land in mainline first, then get backported to linux-X.Y.y. Releases go out roughly weekly. A non-LTS stable series ends shortly after the next mainline release.3536
  • Longterm (LTS). Typically the last release of each year. Kernel.org currently lists 6.18 (projected EOL December 2028), 6.12, 6.6, 6.1, 5.15 and 5.10. A newly designated LTS starts with a 2-year projected EOL, which can be extended if industry commits to supporting it. In February 2026, Greg Kroah-Hartman’s kernel.org update extended several branches this way: 6.6 and 6.12 to four years of support and 6.18 to at least three, moving 6.12 and 6.18 to a projected EOL of December 2028.
  • Development trees. linux-next integrates every maintainer’s -next branch daily, which catches conflicts before the merge window.
make kernelversion
git describe                 # e.g. v7.3 or v7.3-rc7-123-gabcdef
cat /proc/version

3.8 Reading a kernel oops

BUG: kernel NULL pointer dereference, address: 0000000000000010
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 3 PID: 4211 Comm: insmod Tainted: G           O       7.3.0 #1
RIP: 0010:buggy_init+0x15/0x40 [buggy]
Code: ...
RSP: 0018:ffffc90001a3bd78 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ...
Call Trace:
 <TASK>
 do_one_initcall+0x58/0x300
 do_init_module+0x60/0x240
 __do_sys_finit_module+0x...
 do_syscall_64+0x...
 entry_SYSCALL_64_after_hwframe+0x...
 </TASK>

How to read it:

  1. The first line is the diagnosis: a NULL dereference at offset 0x10, meaning a struct member at offset 16 of a NULL pointer.
  2. Tainted: flags tell you how far to trust the kernel. G means only GPL modules are loaded. O means an out-of-tree module is loaded. P means proprietary, E unsigned, W that a warning happened earlier. Maintainers deprioritise reports from tainted kernels for good reason.
  3. RIP: buggy_init+0x15/0x40 [buggy] gives the function, the offset into it, the function length, and the module.
  4. The call trace reads from the bottom up. This one runs from the syscall entry through module loading into the faulting init function.
  5. Turn the offset into a source line:
./scripts/faddr2line path/to/buggy.ko buggy_init+0x15/0x40
./scripts/decode_stacktrace.sh vmlinux /lib/modules/$(uname -r) < oops.txt
gdb vmlinux -ex 'list *(some_function+0x15)'

Build debug kernels with CONFIG_DEBUG_INFO_DWARF5=y, CONFIG_KALLSYMS_ALL=y and CONFIG_FRAME_POINTER=y (or ORC unwinding, the x86 default), and for development also KASAN, lockdep and CONFIG_DEBUG_ATOMIC_SLEEP. For crashes on production systems, set up kdump (crashkernel= plus kexec) and analyse the vmcore with crash.

3.9 Sending a patch (briefly)

git checkout -b fix-foo v7.3
# ... edit, build, test (including relevant tools/testing/selftests and KUnit) ...
git commit -s          # -s adds Signed-off-by: = Developer Certificate of Origin
./scripts/checkpatch.pl --strict -g HEAD
git format-patch -1 --base=auto -o outgoing/
./scripts/get_maintainer.pl outgoing/0001-*.patch
git send-email --to=<maintainer> --cc=<list> outgoing/0001-*.patch
# or the modern workflow:
b4 prep / b4 send

Rules that matter:

  • Plain-text email to the right list. MAINTAINERS and get_maintainer.pl tell you which list.
  • One logical change per patch. The commit message should explain why, not just what.
  • Bug fixes carry a Fixes: tag and, where appropriate, Cc: stable@vger.kernel.org.
  • Respond to review with a v2 that includes a changelog below the --- line.
  • Read Documentation/process/submitting-patches.rst before you send anything.
  • Disclose tool assistance. The 7.3 cycle saw over 1,200 commits carrying Assisted-by: tags.19 AI-assisted contributions are a live policy topic, so follow the current documented rules.