Part 2 — The kernel, in full detail

2.1 Architecture: monolithic, modular, preemptible

Linux is a monolithic kernel. The scheduler, memory manager, filesystems, network stack and drivers all run in one address space at the highest privilege level (ring 0 on x86, EL1 on arm64). A call from the VFS into ext4 into the block layer into an NVMe driver is an ordinary function call. No message is passed and no context is switched. That is the performance argument. The cost is that any bug anywhere in that address space can corrupt anything else.

Three properties make the monolith manageable:

• Loadable modules. Code is linked into the running kernel at runtime (.ko files), so one generic kernel image can support thousands of devices. Modules do not change the trust model: a module is fully privileged kernel code.
• Internal layering by convention. The VFS defines operations tables (struct file_operations, struct inode_operations). The block layer has struct blk_mq_ops. The network stack has struct net_device_ops. Each is a C vtable. Subsystems talk to each other through these tables, which gives microkernel-like modularity without IPC.
• Preemption. The kernel itself can be preempted at many points. The preemption model is a build-time choice (see 2.4). Since Linux 6.12, the full real-time model, PREEMPT_RT, is in mainline.

The kernel’s top-level contract with userspace is the syscall ABI plus pseudo-filesystems (/proc, /sys, /dev, cgroupfs, tracefs, bpffs). Internally, the kernel has no stable API: in-tree code is refactored freely, and out-of-tree modules have to keep up. This asymmetry, a stable external ABI and an unstable internal API, is the most important design decision to understand. Part 5 returns to it.

2.2 The boot chain, firmware to PID 1

The following walks through x86-64. arm64 is analogous: UEFI or U-Boot, then Image, then primary_entry in arch/arm64/kernel/head.S, then start_kernel.

 Power on
   │
   ▼
 Firmware (UEFI / legacy BIOS)  ── POST, DRAM training, ACPI tables, memory map
   │
   ▼
 Bootloader ── GRUB 2 / systemd-boot / direct EFI-stub boot
   │  loads: bzImage (vmlinuz) + initramfs (cpio, often compressed) + cmdline
   ▼
 arch/x86/boot/            (real-mode / setup code; skipped by EFI stub path)
   │  header.S "setup header" → main.c: detect memory (e820), video, go_to_protected_mode()
   ▼
 arch/x86/boot/compressed/ (decompressor)
   │  head_64.S: set up identity page tables, enter long mode
   │  misc.c: extract_kernel() → decompress vmlinux (gzip/xz/lz4/zstd), apply KASLR relocations
   ▼
 arch/x86/kernel/head_64.S  startup_64 → early page tables → x86_64_start_kernel() (head64.c)
   ▼
 init/main.c  start_kernel()
   │  setup_arch()          ← parse e820/EFI memmap, ACPI, memblock, early ioremap, NUMA
   │  mm_core_init()        ← buddy allocator, SLUB, vmalloc come online
   │  sched_init()          ← runqueues, idle task
   │  init_IRQ(), timers, time_init(), console_init() ...
   │  rest_init()
   │     ├─ kernel_thread(kernel_init)   → will become PID 1
   │     └─ kernel_thread(kthreadd)      → PID 2, parent of all kernel threads
   ▼
 kernel_init() → kernel_init_freeable()
   │  do_basic_setup(): driver core, initcalls (all built-in drivers probe here)
   │  unpack initramfs into rootfs (a tmpfs/ramfs)
   │  try rdinit= (default /init) in the initramfs
   │  else mount root= device and try init=, /sbin/init, /etc/init, /bin/init, /bin/sh
   ▼
 Userspace PID 1 (systemd, or /init script in initramfs)
   │  initramfs: load modules, unlock/assemble storage, mount real root at /sysroot
   │  switch_root /sysroot /sbin/init   (deletes initramfs contents, chroots, execs)
   ▼
 Real PID 1 on the real root filesystem → services → login

Here is why each stage exists:

• Why decompress? Firmware and bootloaders load bytes from slow media into memory with few constraints. A compressed image loads faster, and the decompressor doubles as the place where position independence and KASLR are handled before the “real” kernel runs.
• The EFI stub. With CONFIG_EFI_STUB=y, bzImage is also a valid PE/COFF executable. UEFI firmware, or systemd-boot, can start it directly with no GRUB. The stub calls EFI boot services, gets the memory map, exits boot services and enters the decompressor. This is how unified kernel images (UKIs) work.
• initramfs vs. initrd. The old initrd was a block-device image that needed a filesystem driver. The modern initramfs is a cpio archive unpacked into an in-kernel tmpfs called “rootfs”. No block driver is needed, which is why it is the universal mechanism for early userspace. The kernel can also embed an initramfs at build time (CONFIG_INITRAMFS_SOURCE).
• switch_root vs. pivot_root. pivot_root(2) swaps mount points and is used by container runtimes. The rootfs that holds the initramfs cannot be unmounted, so switch_root (util-linux or BusyBox) instead deletes the initramfs files to free the RAM, mount --moves the new root onto /, chroots and execs the real init.
• Why PID 1 is special. It reaps orphaned zombies. It cannot be killed by signals it does not handle. If it exits, the kernel panics (“Attempted to kill init!”). That is why an initramfs /init must exec the next init rather than fork it.

Observe the boot chain on a live system:

dmesg | head -50                                  # earliest kernel messages
journalctl -k -b | grep -iE 'efi|e820|kaslr' | head
systemd-analyze                                    # firmware / loader / kernel / initrd / userspace split
systemd-analyze critical-chain
cat /proc/cmdline
lsinitramfs /boot/initrd.img-$(uname -r) | head   # Debian/Ubuntu; use lsinitrd on Fedora
cat /sys/kernel/debug/x86/pat_memtype_list 2>/dev/null | head   # needs debugfs + root

To see start_kernel run step by step, use the QEMU+GDB setup in Part 5 and set break start_kernel.

2.3 Memory management

Virtual memory and paging on x86-64

Every process sees a private virtual address space. The MMU translates virtual addresses to physical ones by walking a radix tree of page tables rooted at the physical address in CR3.

• 4-level paging: PGD → PUD → PMD → PTE, with 9 bits per level plus a 12-bit offset. That gives 48-bit virtual addresses: 256 TiB total, split into 128 TiB for user and 128 TiB for kernel.
• 5-level paging (LA57): adds a P4D level and gives 57-bit virtual addresses (128 PiB). Kernel developers made 5-level support unconditional on x86-64 by removing the CONFIG_X86_5LEVEL option. It is now detected at boot and can be disabled with no5lvl.¹¹¹²¹³ On 4-level hardware the extra level is “folded” away. Userspace only gets addresses above 47 bits if it explicitly asks with an mmap hint, which protects programs that store tags in high pointer bits.
• The TLB caches translations. PCID tags TLB entries per address space so that context switches need not flush everything. That matters a great deal since the Meltdown mitigation (KPTI) split user and kernel page tables.

The kernel generically abstracts these levels as pgd_t, p4d_t, pud_t, pmd_t and pte_t (include/linux/pgtable.h). An architecture with fewer levels folds the unused ones at compile time.

Physical memory: nodes, zones, pages

 pg_data_t (one per NUMA node)
   ├─ ZONE_DMA      (< 16 MiB, ancient ISA DMA)
   ├─ ZONE_DMA32    (< 4 GiB, 32-bit DMA devices)
   ├─ ZONE_NORMAL   (everything else on 64-bit)
   ├─ ZONE_MOVABLE  (only movable pages → hot-unplug, CMA, compaction friendly)
   └─ ZONE_DEVICE   (pmem, device-private memory)

Every physical page frame is described by a struct page, which is increasingly accessed through struct folio. A folio represents a power-of-two group of pages handled as a unit. Converting the page cache and memory management to folios has been the main mm refactoring since Matthew Wilcox’s folio patches were merged in 5.16 (the page-cache conversion followed in 5.17), because it cuts per-page overhead and makes large pages first-class.

The buddy allocator

The page allocator (mm/page_alloc.c) hands out blocks of 2^order contiguous pages. Order 0 is 4 KiB, order 9 is 2 MiB, and MAX_PAGE_ORDER is the cap. On free, a block merges with its “buddy” whenever the buddy is also free. This keeps external fragmentation in check at O(log n) cost. On top of the buddy lists sit per-CPU page lists (PCP), which make order-0 allocation mostly lock-free. Allocation behaviour is controlled by GFP flags: GFP_KERNEL may sleep and do I/O to reclaim, GFP_ATOMIC must not sleep, GFP_NOWAIT avoids blocking, __GFP_MOVABLE is a placement hint, and so on.

cat /proc/buddyinfo          # free blocks per order, per zone, per node
cat /proc/pagetypeinfo       # same, split by migrate type (root)
cat /proc/zoneinfo | less    # watermarks: min/low/high per zone

The slab layer (SLUB)

Most kernel objects are much smaller than a page. The slab allocator carves pages into caches of fixed-size objects (kmem_cache_create(), kmem_cache_alloc()), and kmalloc() serves generic sizes from size-class caches. Linux once had three implementations. SLOB was removed in 6.4 and SLAB in 6.8, so SLUB is now the only slab allocator, which lets developers optimise it without worrying about the others. The current SLUB work adds “sheaves”, per-CPU arrays of object pointers kept under the allocator’s control, so that subsystems stop building their own caching layers on top.¹⁴

sudo slabtop -o | head -25
sudo cat /proc/slabinfo | head
ls /sys/kernel/slab/ | head          # per-cache tunables and stats

kmalloc vs. vmalloc

kvmalloc() tries kmalloc first and falls back to vmalloc. Use it when you need a large buffer and do not care about physical contiguity.

The page cache, reclaim, and the OOM killer

File data lives in the page cache, indexed per inode by an address_space (an XArray of folios). read(2) copies from it, and mmap(2) maps its folios straight into the process. Writes dirty the folios, and writeback threads flush them. Under memory pressure, kswapd (background) and direct reclaim (synchronous) evict clean page-cache folios and swap out anonymous memory, guided by LRU lists. Recent kernels can use the multi-generational LRU (MGLRU) here. When reclaim fails, the OOM killer chooses a victim by oom_badness(), which is largely resident memory adjusted by oom_score_adj. In containers it acts per cgroup when memory.max is hit.

Huge pages

On x86-64, huge pages are 2 MiB (PMD-level) or 1 GiB (PUD-level). They reduce TLB misses. There are two mechanisms. HugeTLBfs is explicitly reserved (vm.nr_hugepages) and mapped via MAP_HUGETLB or a hugetlbfs mount. Transparent Huge Pages (THP) are opportunistic, with khugepaged collapsing them in the background. Multi-size THP (mTHP) also allows intermediate folio sizes, such as 64 KiB, for anonymous memory.

grep -i huge /proc/meminfo
cat /sys/kernel/mm/transparent_hugepage/enabled
ls /sys/kernel/mm/transparent_hugepage/ | grep hugepages-   # mTHP sizes
numactl --hardware                     # NUMA topology and distances
cat /proc/$$/numa_maps | head          # per-mapping NUMA placement
cat /proc/$$/maps                      # the virtual address space of your shell
sudo cat /proc/$$/smaps_rollup         # RSS / PSS / swap summary

2.4 Processes, scheduling, cgroups and namespaces

task_struct and the process model

The kernel does not distinguish processes from threads in its core data structure. Each schedulable entity is a struct task_struct (include/linux/sched.h), a large structure holding scheduling state, credentials (cred), the memory descriptor (mm_struct, shared by threads), open files (files_struct), filesystem context (fs_struct), namespaces (nsproxy), signal handlers and more. A “thread” is just a task that shares its mm, files and signal handlers with other tasks.

• fork(), vfork() and clone()/clone3() all go through kernel_clone() in kernel/fork.c. The CLONE_* flags decide what is shared and what is copied. Copying is lazy: copy-on-write marks pages read-only and duplicates them on the first write fault.
• execve() (fs/exec.c) replaces the mm. It finds a binary format handler (binfmt_elf, binfmt_script, binfmt_misc), maps the ELF segments, sets up the stack with argv/envp/auxv (including AT_SYSINFO_EHDR, which points at the vDSO), and returns to the new entry point. For dynamic binaries that entry point is the dynamic loader, ld-linux.
• exit() releases resources and leaves a zombie until the parent wait()s.

Scheduling classes

kernel/sched/ implements a strict hierarchy of scheduling classes. A higher class always preempts a lower one:

From CFS to EEVDF. CFS (2.6.23, 2007) picked the runnable task with the smallest weighted virtual runtime from a red-black tree. Linux 6.6 replaced that selection logic with EEVDF (Earliest Eligible Virtual Deadline First),¹⁵¹⁶ and Linux 6.12 completed the transition.¹⁷¹⁸ EEVDF computes each task’s lag, the service it is owed relative to an ideal fair share. Only tasks with non-negative lag are eligible. Among those it picks the one with the earliest virtual deadline, which is derived from the task’s requested time slice. The result is better latency for short-slice tasks without CFS’s pile of heuristics. The weight table, vruntime accounting, group scheduling and load balancing remain largely the same, and the class is still called fair_sched_class.

sched_ext. Merged in 6.12, sched_ext lets a BPF program act as the scheduler. If it misbehaves, the kernel kicks it out and falls back to the fair class. Linux 7.3 completed sub-schedulers: a root BPF scheduler can hand a cgroup subtree to a nested BPF scheduler along with revocable CPU grants.^19202122

Preemption models (chosen in Kconfig under “Preemption Model”):

PREEMPT_DYNAMIC lets distribution kernels choose between none, voluntary, full and lazy at boot with preempt=.

cgroups v2 and namespaces: the container primitives

Containers are not a kernel object. They are a composition of two independent mechanisms:

• Namespaces virtualise what a process can see: mnt, pid, net, ipc, uts, user, cgroup, time. They are created with clone(CLONE_NEW*) or unshare(2), and joined with setns(2).
• cgroups v2 limit and account for what a process can use. The unified hierarchy at /sys/fs/cgroup provides cpu.max, cpu.weight, memory.max, memory.high, io.max, pids.max and pressure-stall information (*.pressure).

# Scheduling observation
chrt -p $$                                   # policy/priority of your shell
cat /proc/$$/sched | head -30                # per-task scheduler stats
cat /proc/sys/kernel/sched_*  2>/dev/null    # tunables (many moved to debugfs)
sudo ls /sys/kernel/debug/sched/             # debug knobs, features
cat /sys/kernel/sched_ext/state 2>/dev/null  # is a BPF scheduler loaded?
perf sched record -- sleep 5 && perf sched latency | head

# Namespaces and cgroups
lsns                                          # all namespaces on the system
ls -l /proc/$$/ns
sudo unshare --pid --fork --mount-proc --uts --net bash   # a 'container' in one line
cat /proc/self/cgroup
systemd-cgls | head -40
cat /sys/fs/cgroup/system.slice/cpu.pressure

2.5 The VFS and filesystems

The Virtual Filesystem Switch (fs/) is the abstraction that makes “everything is a file” work. It defines four core objects. Each has an operations table that a concrete filesystem implements:

Path lookup (fs/namei.c) walks dentries component by component. The fast path, “RCU-walk”, takes no locks or reference counts and drops back to “ref-walk” only when it must. Mounts are struct mount objects grouped into mount namespaces, and propagation (shared, slave, private) decides how mount events spread between namespaces. The new mount API (fsopen, fsconfig, fsmount, move_mount, open_tree) replaces the overloaded mount(2) with file-descriptor-based steps. Linux 7.3 continued this “do it via an fd” direction with fchroot() and a deliberately empty failfs, which give sandboxes a root that has no ambient filesystem at all.

The VFS is tied closely to the page cache. Regular-file read_iter/write_iter normally go through the generic helpers (filemap_read, generic_perform_write), which work on folios in the inode’s address_space. The filesystem only provides address_space_operations to read and write folios against blocks, and the iomap framework increasingly does that work.

Filesystem overview:

• ext4: journaled, extent-based, conservative, the default for LFS and many distros. e2fsprogs provides mkfs and fsck.
• XFS: allocation groups give parallelism, and it scales well for large files and filesystems. It is the RHEL default. Its metadata checksums and online repair are mature.
• Btrfs: copy-on-write B-trees, with snapshots, checksums for data and metadata, built-in RAID profiles, send/receive and compression. Its strengths and tradeoffs both come from CoW.
• Plus tmpfs, overlayfs (the basis of container image layers), FUSE, and pseudo-filesystems (procfs, sysfs, debugfs, tracefs, cgroup2, bpffs).

io_uring. io_uring uses two ring buffers shared between user and kernel memory: a submission queue and a completion queue. It turns syscalls-per-I/O into batched, asynchronous, often syscall-free I/O, and it now covers much more than read/write: networking, openat, statx and more. Its large attack surface means some hardened environments disable it via the io_uring_disabled sysctl or seccomp.

findmnt                          # the mount tree of your namespace
cat /proc/self/mountinfo | head  # propagation flags are here
stat /etc/passwd                 # inode metadata
sudo cat /proc/sys/fs/dentry-state
cat /proc/filesystems            # registered filesystem types
sudo debugfs -R 'stats' /dev/sdXN | head   # ext4 internals (read-only)
vmtouch /some/file 2>/dev/null   # how much of a file is in the page cache
cat /proc/sys/kernel/io_uring_disabled 2>/dev/null

2.6 The system call path

Here is the full path of a write(1, buf, n) from user space on x86-64:

 user:   write() in glibc  → mov $1,%eax (__NR_write); args in rdi,rsi,rdx,r10,r8,r9
         syscall instruction
 cpu:    RIP ← MSR_LSTAR (entry_SYSCALL_64), CS/SS from MSR_STAR, RFLAGS masked by MSR_SYSCALL_MASK
         RCX ← user RIP, R11 ← user RFLAGS; still on the *user* stack, CPL=0
 kernel: arch/x86/entry/entry_64.S  entry_SYSCALL_64:
           swapgs                      (GS base → per-CPU area)
           switch CR3 if KPTI          (user page tables → kernel page tables)
           switch to the task's kernel stack, build struct pt_regs
           call do_syscall_64(regs, nr)
         do_syscall_64():  syscall_enter_from_user_mode()  (ptrace, seccomp, audit, tracepoints)
                           x64_sys_call(regs, nr)          → __x64_sys_write → ksys_write → vfs_write
                           syscall_exit_to_user_mode()     (signals, rescheduling, tracing)
         return via SYSRET (or IRET on the slow path) after restoring CR3/GS

Key points:

• The syscall table comes from arch/x86/entry/syscalls/syscall_64.tbl. Since Linux 6.9, with commit 1e3ad78334a6 (“x86/syscall: Don’t force use of indirect calls for system calls”), x86-64 dispatches through a generated switch in x64_sys_call() instead of an indirect call through sys_call_table[]. This is a Spectre-BHI hardening measure. Syscall numbers are ABI and never change. Linux 7.3 added fchroot at the end of the table.¹
• SYSCALL_DEFINEn macros (include/linux/syscalls.h) generate the wrappers that unpack pt_regs, plus metadata used by ftrace syscall events.
• The vDSO. clock_gettime(), gettimeofday(), time() and getcpu() normally never enter the kernel. The kernel maps a small ELF shared object, the vDSO, into every process. It reads timekeeping data the kernel publishes in a shared page (the vvar). glibc finds it through AT_SYSINFO_EHDR.
• ptrace and seccomp hook in at syscall_enter_from_user_mode(). That is why strace sees every syscall, and why a seccomp filter can return SECCOMP_RET_ERRNO, KILL, TRAP, USER_NOTIF or TRACE before the syscall body runs.

strace -f -tt -e trace=openat,read,write cat /etc/hostname
strace -c ls / >/dev/null                      # syscall histogram
grep -E '^[0-9]+\s+common\s+(read|write|fchroot)\b' \
     /usr/src/linux/arch/x86/entry/syscalls/syscall_64.tbl   # if you have the source
cat /proc/self/auxv | od -A x -t x8 | head     # raw auxv
LD_SHOW_AUXV=1 /bin/true | grep SYSINFO        # where the vDSO is mapped
grep vdso /proc/self/maps
sudo perf trace -s -- sleep 1                  # low-overhead strace alternative
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

2.7 Interrupts, exceptions and deferred work

Exceptions are synchronous: page faults, general protection faults, divide errors, int3. Interrupts are asynchronous, coming from devices via the IOAPIC/MSI/MSI-X, or as IPIs between CPUs. On x86-64, both enter through the IDT (arch/x86/kernel/idt.c, arch/x86/entry/). FRED (Flexible Return and Event Delivery) is the newer replacement for IDT delivery on CPUs that support it. Generic IRQ handling lives in kernel/irq/: irq_desc, irq_chip, the flow handlers (handle_edge_irq, handle_fasteoi_irq) and irqaction chains for shared lines.

A hard interrupt handler must be short, because it runs with that interrupt line, or all local interrupts, masked. The work is therefore split:

cat /proc/interrupts | head -30         # per-CPU counts per IRQ line
watch -n1 -d 'cat /proc/softirqs'       # softirq counts per CPU
ps -eo pid,cls,rtprio,comm | grep -E 'irq/|ksoftirqd|kworker' | head
cat /proc/irq/*/smp_affinity_list 2>/dev/null | head
sudo perf stat -e 'irq:*' -a sleep 2

2.8 The device model

The driver core (drivers/base/) represents hardware as a graph of C objects, and it exports that graph to userspace through sysfs:

• kobject / kset / ktype (lib/kobject.c): reference-counted nodes with a parent pointer and a sysfs directory. Every device, driver, bus and class embeds one.
• bus_type: PCI, USB, platform, I2C, SPI, virtio and so on. Each bus has a match() that compares a device’s identity (PCI vendor/device ID, ACPI _HID, DT compatible string) against each driver’s ID table.
• device_driver: probe() binds the driver to a device and remove() unbinds it. Probing can be deferred (-EPROBE_DEFER) until a dependency, such as a clock or regulator, appears.
• class: a user-facing grouping (net, block, input, tty) that is independent of the bus.

uevents and udev. When a device is added, the kernel emits a uevent over netlink containing ACTION, DEVPATH, SUBSYSTEM and MODALIAS. systemd-udevd receives it. It runs kmod to load the module that matches MODALIAS (via modules.alias), applies rules from /usr/lib/udev/rules.d and /etc/udev/rules.d, creates symlinks such as /dev/disk/by-uuid/..., and sets ownership. The device nodes themselves come from devtmpfs, which the kernel populates (CONFIG_DEVTMPFS, CONFIG_DEVTMPFS_MOUNT).

ACPI vs. Device Tree. Both describe hardware that cannot be discovered on its own. On x86 and most arm64 servers, firmware provides ACPI tables (DSDT/SSDT bytecode, interpreted by the kernel’s ACPICA). On most embedded ARM, RISC-V and PowerPC systems, the bootloader passes a flattened Device Tree blob (.dtb, compiled from .dts in arch/*/boot/dts/). Drivers mostly use the firmware-agnostic fwnode/device_property_* API so the same driver works with either.

ls /sys/bus/                             # all buses
ls /sys/bus/pci/drivers/                 # bound drivers
readlink /sys/class/net/*/device/driver  # which driver backs each NIC
udevadm info -a /sys/class/block/sda | head -40
udevadm monitor --kernel --udev          # then plug in a USB stick
cat /sys/bus/pci/devices/*/modalias | head
modprobe --resolve-alias "$(cat /sys/class/net/eth0/device/modalias 2>/dev/null)"
lspci -k | head -30

2.9 Locking and concurrency

The kernel is massively concurrent: preemption, SMP, interrupts and softirqs all interleave. Choosing a primitive starts with one question: can the code that holds this lock sleep, and from which contexts is it taken?

Memory barriers. CPUs and compilers reorder memory accesses. The kernel memory model (tools/memory-model/, the LKMM) defines smp_mb(), smp_rmb(), smp_wmb(), smp_load_acquire()/smp_store_release(), and READ_ONCE()/WRITE_ONCE() to prevent compiler tearing and fusing. When you write lockless code, check it against LKMM litmus tests with herd7.

lockdep (CONFIG_PROVE_LOCKING) records the order in which lock classes are acquired across the whole run, and reports a potential deadlock the first time an inconsistent order occurs, even if no deadlock actually happened. KCSAN detects data races dynamically, and KASAN detects memory-safety bugs. Run all three in debug builds.

cat /proc/lockdep_stats 2>/dev/null       # only on lockdep-enabled kernels
sudo cat /sys/kernel/debug/rcu/*/rcudata 2>/dev/null | head
sudo perf lock record -- sleep 2 && sudo perf lock report | head

2.10 Modules and the module loader

lsmod | head
modinfo ext4                    # license, depends, vermagic, signer, parameters
sudo modprobe -v dummy          # resolves deps via modules.dep, then insmod
sudo modprobe -r dummy
cat /proc/modules | head
ls /sys/module/ext4/parameters 2>/dev/null
cat /proc/sys/kernel/modules_disabled     # 1 = no more loading until reboot

Here is how the pieces fit:

• insmod passes an ELF .ko to finit_module(2) (or the older init_module(2)). kernel/module/ then verifies the signature if one is required, allocates memory, applies relocations, resolves undefined symbols against the kernel’s exported symbol table, and calls the module’s module_init() function.
• Symbol exports. Only symbols marked EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL() are visible to modules. _GPL exports can only be used by modules declaring a GPL-compatible MODULE_LICENSE. Symbol namespaces (EXPORT_SYMBOL_NS_GPL, MODULE_IMPORT_NS) further limit who may import.
• modprobe (kmod) adds dependency resolution (modules.dep, generated by depmod), alias matching (modules.alias) and configuration (/etc/modprobe.d/, blacklists, options).
• vermagic and modversions. CONFIG_MODVERSIONS records CRCs of exported symbol prototypes, so a module built against an incompatible kernel is rejected. This is how the “no stable internal API” policy is enforced mechanically.
• Module signing. With CONFIG_MODULE_SIG=y the build signs every module with a key (auto-generated as certs/signing_key.pem unless you supply one). CONFIG_MODULE_SIG_FORCE=y, or lockdown mode, refuses unsigned modules. Under Secure Boot, distributions chain this to their shim/MOK keys.

A minimal out-of-tree module, so readers can watch the loader work:

// hello.c
#include <linux/init.h>
#include <linux/module.h>
#include <linux/printk.h>

static int __init hello_init(void)
{
	pr_info("hello: loaded, jiffies=%lu\n", jiffies);
	return 0;
}

static void __exit hello_exit(void)
{
	pr_info("hello: unloaded\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Minimal module to observe the loader");

# Makefile
obj-m := hello.o
KDIR  ?= /lib/modules/$(shell uname -r)/build

all:
	$(MAKE) -C $(KDIR) M=$(CURDIR) modules
clean:
	$(MAKE) -C $(KDIR) M=$(CURDIR) clean

make && sudo insmod hello.ko && sudo dmesg | tail -2 && sudo rmmod hello

2.11 Security subsystems

Kernel security comes in layers, and each layer enforces a different principle:

• DAC and credentials. The classic UID/GID and permission bits, stored in struct cred, which is immutable and RCU-replaced.
• Capabilities (man 7 capabilities). Root’s power is split into roughly 40 bits (CAP_NET_ADMIN, CAP_SYS_ADMIN, CAP_BPF, CAP_PERFMON and so on), held in permitted, effective, inheritable, bounding and ambient sets. CAP_SYS_ADMIN remains an overloaded “new root”.
• LSM framework (security/). Hooks are placed at security-relevant points such as security_file_open and security_bprm_check. Several LSMs can be stacked, listed in boot order via lsm=. Two major LSMs typically sit alongside the minor ones:
  • SELinux: label-based mandatory access control with type enforcement, used by Fedora, RHEL and Android.
  • AppArmor: path-based profiles, used by Ubuntu, Debian and SUSE.
  • Landlock: unprivileged sandboxing, where a process restricts its own filesystem, network-port and IPC access. It is composable and needs no root.
  • Yama (ptrace scope), LoadPin, IPE (Integrity Policy Enforcement, 6.12), lockdown and BPF LSM (attach BPF programs to LSM hooks).
• seccomp. A classic-BPF filter applied to syscall numbers and arguments, as described in 2.6. systemd’s SystemCallFilter=, Docker and Chrome all use it.
• Lockdown (lockdown=integrity|confidentiality). This blocks even root from modifying the running kernel: /dev/mem, unsigned modules, kexec of unsigned images, some debugfs and BPF features. It is tied to Secure Boot on many distros.
• Exploit mitigations:
  • KASLR: randomises the kernel base, physical and virtual.
  • SMEP and SMAP: the CPU refuses to execute or access user pages from ring 0 unless stac/clac explicitly permits it.
  • Stack protector: CONFIG_STACKPROTECTOR_STRONG adds canaries.
  • CONFIG_INIT_STACK_ALL_ZERO, hardened usercopy, CONFIG_FORTIFY_SOURCE, CONFIG_RANDOMIZE_KSTACK_OFFSET.
  • CFI: kCFI with Clang, and FineIBT with Intel IBT.
  • Shadow stacks for userspace (Intel CET, since 6.6).
  • Strict kernel memory permissions (W^X) and refcount_t.

cat /sys/kernel/security/lsm; echo
cat /sys/kernel/security/lockdown 2>/dev/null
grep -E 'smep|smap|ibt|shstk|user_shstk' -o /proc/cpuinfo | sort | uniq -c
grep Cap /proc/self/status && capsh --decode=$(awk '/CapEff/{print $2}' /proc/self/status)
grep Seccomp /proc/self/status
sudo dmesg | grep -iE 'kaslr|lockdown|landlock|selinux|apparmor' | head
zgrep -E 'CONFIG_(STACKPROTECTOR_STRONG|RANDOMIZE_BASE|MODULE_SIG_FORCE|SECURITY_LANDLOCK)=' /proc/config.gz 2>/dev/null \
  || grep -E 'CONFIG_(STACKPROTECTOR_STRONG|RANDOMIZE_BASE|MODULE_SIG_FORCE|SECURITY_LANDLOCK)=' /boot/config-$(uname -r)

2.12 eBPF, tracing and observability

The best way to understand a kernel is to watch it run. Linux exposes several layers of observability:

The BPF verifier is what makes running user-supplied code in the kernel acceptable. Before loading, it explores every path to prove memory safety, bounded loops and correct use of helpers and kfuncs. Since 5.8, CAP_BPF and CAP_PERFMON allow tracing without full CAP_SYS_ADMIN.

A practical tour:

# ftrace: function_graph of a single syscall in a single process
cd /sys/kernel/tracing
echo 0 > tracing_on
echo function_graph > current_tracer
echo ksys_read > set_graph_function
echo $$ > set_ftrace_pid
echo 1 > tracing_on; cat /etc/hostname; echo 0 > tracing_on
head -60 trace
echo nop > current_tracer; echo > set_graph_function; echo > set_ftrace_pid

# trace-cmd is the friendlier front end
sudo trace-cmd record -p function_graph -g do_sys_openat2 -F cat /etc/hostname
sudo trace-cmd report | head -60

# Tracepoints
ls /sys/kernel/tracing/events/ | head
sudo perf list 'sched:*' | head

# perf: where is CPU time going, kernel and user?
sudo perf top -g
sudo perf record -a -g -- sleep 10 && sudo perf report --stdio | head -60

# bpftrace one-liners
sudo bpftrace -e 'kprobe:do_sys_openat2 { printf("%s %s\n", comm, str(arg1)); }'
sudo bpftrace -e 'tracepoint:sched:sched_switch { @[kstack(5)] = count(); }'
sudo bpftrace -e 'kretprobe:vfs_read /retval > 0/ { @bytes[comm] = sum(retval); }'
sudo bpftrace -e 'uprobe:/usr/lib/x86_64-linux-gnu/libc.so.6:malloc { @[comm] = count(); }'

# What BPF is already loaded on this box?
sudo bpftool prog show | head
sudo bpftool map show | head
sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -A20 'struct task_struct {' | head -30

2.13 The networking stack

 NIC ─DMA→ RX ring ─IRQ→ NAPI poll (softirq NET_RX)
          │
          ├─ XDP hook (eBPF, before sk_buff allocation): XDP_DROP / PASS / TX / REDIRECT
          ▼
     sk_buff allocated → GRO → tc ingress (eBPF/qdisc) → netfilter PREROUTING
          ▼
     routing decision (FIB) → INPUT (local) or FORWARD
          ▼
     L4: tcp_v4_rcv / udp_rcv → socket receive queue → wake reader
          ▼
     recvmsg() copies to user (or zero-copy: io_uring, MSG_ZEROCOPY, devmem TCP)
 TX is the mirror: socket → L4 → IP → netfilter OUTPUT/POSTROUTING → qdisc (tc) → driver xmit

• struct sk_buff (include/linux/skbuff.h) is the packet descriptor. It holds pointers into a data buffer (head, data, tail, end), so headers can be pushed and pulled without copying, plus fragments, offload metadata and a reference to the socket.
• NAPI switches from interrupts to polling under load, which prevents livelock.
• netfilter hooks are where nftables, and the legacy iptables via its nft backend, plug in, along with connection tracking (conntrack) and NAT.
• XDP runs BPF in the driver’s receive path before an skb exists, which is how line-rate DDoS filtering and load balancers are built. AF_XDP hands frames to userspace without copying.
• Network namespaces give every container its own complete stack: interfaces, routes, netfilter tables and sockets.

ip -s link; ip route; ss -tanp | head
sudo nft list ruleset | head -40
cat /proc/net/softnet_stat | head          # per-CPU backlog/drops
ethtool -S eth0 2>/dev/null | head
sudo bpftool net show
sudo perf trace -e 'net:*' -a -- sleep 1 | head
sudo bpftrace -e 'tracepoint:skb:kfree_skb { @[kstack(3)] = count(); }'   # who drops packets?

2.14 Rust in the kernel

Rust support was merged in Linux 6.1 (December 2022) as an experiment.²³ At the 2025 Kernel Maintainers Summit in Tokyo (December 2025), the developers concluded that “the experiment is done”.²⁴²⁵ Rust is now a permanent part of the kernel, and the “experimental” wording was removed from Documentation/rust/.²⁵²⁶ Current status as of October 2026:

• Toolchain policy. The minimum Rust version follows Debian stable’s toolchain. Since Linux 7.1 the minimum is Rust 1.85.0, with bindgen 0.71.1.²⁷ Stable/LTS branches keep older minimums: 6.18.y still accepts 1.78.0.²⁸²⁹ make rustavailable tells you why your toolchain is not detected.
• What is written in Rust. Android’s Binder IPC driver is the flagship. During the 7.4 merge window (in progress as of this writing), the C Binder implementation was removed, leaving the Rust driver as the only one.³⁰ Others include NVIDIA’s Nova GPU driver (nova-core and nova-drm, still not ready for end users), Arm Mali Tyr, the QR-code panic screen, the Asahi-originated DRM abstractions, and a growing rust/kernel/ crate of safe abstractions over C APIs (sync, alloc, device/driver, DRM, block, workqueues). Linux 7.3 added Rust support for further architectures, including PowerPC.³³¹
• What Rust does not change. C remains the large majority of the code, and existing C drivers are not being rewritten wholesale.³²³³ Rust code links against the same unstable internal APIs. unsafe blocks remain wherever Rust calls C, and the review effort concentrates on those blocks.

grep -E 'CONFIG_RUST=' /boot/config-$(uname -r) 2>/dev/null
ls /sys/module | grep -i rust 2>/dev/null
# In a kernel tree:
make LLVM=1 rustavailable