Refs

Kernel, Traced

1 — A system call, from syscall to return

• kernel/entry/common.c, which is 404 lines at v6.12; arch/x86/entry/entry_64.S with its comments; Documentation/arch/x86/entry_64.rst.

2 — open(): the path walk

• fs/namei.c:walk_component and lookup_fast; fs/dcache.c:__d_lookup_rcu; Documentation/filesystems/path-lookup.rst, which is one of the best documents in the tree.

3 — fork() and exec(): making a process

• kernel/fork.c:copy_process, read with the CLONE_* flags in hand; fs/exec.c:begin_new_exec; Documentation/userspace-api/ELF.rst and the binfmt_misc doc.

4 — The scheduler picks the next task

• kernel/sched/fair.c:pick_eevdf, update_curr, place_entity; Documentation/scheduler/sched-eevdf.rst and sched-ext.rst.

5 — A page fault

• mm/memory.c:handle_pte_fault, the switch. Read it with the callees side by side.
• mm/memory.c:do_wp_page and wp_can_reuse_anon_folio, the reuse decision.
• mm/filemap.c:filemap_fault, the page cache from the fault’s side.
• Documentation/mm/page_tables.rst, and Documentation/admin-guide/mm/concepts.rst for vocabulary.

6 — The page cache and readahead

• mm/readahead.c, whose comment at the top is a short essay; mm/filemap.c:filemap_read; and Documentation/filesystems/vfs.rst, the address_space_operations section.

7 — connect(): where the SYN is born

• net/ipv4/tcp_ipv4.c:tcp_v4_connect end to end.
• kernel/bpf/cgroup.c:__cgroup_bpf_run_filter_sock_addr and compute_effective_progs.
• net/ipv4/inet_hashtables.c:__inet_hash_connect, and why net.ipv4.ip_local_port_range matters under load.
• program_types.rst and the connect4_prog.c selftest.

8 — A packet out: send() to the NIC

• net/ipv4/tcp_output.c:tcp_write_xmit; net/core/dev.c:__dev_queue_xmit; include/net/dropreason-core.h, which lists every drop reason with comments; and scaling.rst, including XPS.

9 — A packet in: NIC to recv()

• net/core/dev.c:__netif_receive_skb_core; net/ipv4/tcp_input.c:tcp_rcv_established, where the fast-path comment is required reading; napi.rst; redirect.rst.

10 — bpf(): the verifier

• kernel/bpf/verifier.c:do_check, reading check_mem_access and is_state_visited first; verifier.rst; Documentation/bpf/standardization/instruction-set.rst.

11 — LSM hooks and cgroups: where policy attaches

• security/security.c, the call_int_hook macro and any three hooks; kernel/bpf/cgroup.c:update_effective_progs and compute_effective_progs; kernel/cgroup/cgroup.c:cgroup_migrate_execute; Documentation/admin-guide/cgroup-v2.rst; LSM BPF Programs.

12 — Signals, timers and RCU: three mechanisms every path uses

• kernel/signal.c:get_signal; the header comment in kernel/time/hrtimer.c; whatisRCU.rst, read twice; Documentation/core-api/entry.rst.

Linux From the Metal Up: further sources

Kernel documentation and source

• The Linux Kernel documentation: https://docs.kernel.org/
• Minimal requirements to compile the kernel: https://docs.kernel.org/process/changes.html
• Kbuild and Kconfig: https://docs.kernel.org/kbuild/index.html
• The development process: https://docs.kernel.org/process/development-process.html
• Submitting patches: https://docs.kernel.org/process/submitting-patches.html
• x86 boot protocol: https://docs.kernel.org/arch/x86/boot.html
• Memory management: https://docs.kernel.org/mm/index.html
• Scheduler (incl. EEVDF and sched_ext): https://docs.kernel.org/scheduler/index.html
• Locking and the memory model: https://docs.kernel.org/locking/index.html
• Tracing (ftrace, kprobes, tracepoints): https://docs.kernel.org/trace/index.html
• BPF: https://docs.kernel.org/bpf/index.html
• Rust in the kernel: https://docs.kernel.org/rust/index.html and https://rust-for-linux.com/
• Debugging with GDB: https://docs.kernel.org/dev-tools/gdb-kernel-debugging.html
• Active kernel releases: https://www.kernel.org/releases.html
• Bootlin Elixir cross-referencer: https://elixir.bootlin.com/linux/latest/source

Building systems

• Linux From Scratch (13.1-systemd): https://www.linuxfromscratch.org/lfs/view/13.1-systemd/
• LFS news and errata: https://www.linuxfromscratch.org/lfs/news.html
• Beyond Linux From Scratch: https://www.linuxfromscratch.org/blfs/
• Buildroot manual: https://buildroot.org/docs.html
• Yocto Project documentation: https://docs.yoctoproject.org/
• BusyBox: https://busybox.net/

News, analysis and history

• LWN.net kernel index and merge-window summaries: https://lwn.net/Kernel/
• KernelNewbies release notes (e.g., Linux 7.3): https://kernelnewbies.org/Linux_7.3
• Bootlin training materials (kernel, embedded, Yocto, Buildroot): https://bootlin.com/docs/

Writing a kernel from scratch

• OSDev wiki: https://wiki.osdev.org/
• xv6 (RISC-V) and its book: https://pdos.csail.mit.edu/6.1810/
• Writing an OS in Rust: https://os.phil-opp.com/
• Operating Systems: Three Easy Pieces: https://pages.cs.wisc.edu/~remzi/OSTEP/

System Design

1 — What happens in a single request

• What happens when you type a URL, an extremely detailed community write-up.
• High Performance Browser Networking, Ilya Grigorik.

2 — What one machine can do

• Latency numbers every programmer should know, an interactive version by year.
• The USE method, Brendan Gregg, on finding which resource is the limit.

3 — Measuring: latency, throughput, availability

• Service Level Objectives, Google SRE book.
• Embracing Risk, Google SRE book, on error budgets.

4 — Scaling out: stateless servers and load balancers

• The Power of Two Choices in Randomized Load Balancing, Michael Mitzenmacher.
• The Twelve-Factor App, on stateless processes.

5 — Databases: indexes, transactions and isolation

• Transaction Isolation, PostgreSQL documentation.
• Use The Index, Luke, on SQL indexing.
• Designing Data-Intensive Applications, Martin Kleppmann.

6 — Caching

• Scaling Memcache at Facebook, Nishtala et al.

7 — Queues and asynchronous work

• The Log: What every software engineer should know about real-time data, Jay Kreps.
• Kafka delivery semantics, Confluent.

8 — Queueing, tail latency and scalability limits

• The Tail at Scale, Dean and Barroso.
• Addressing Cascading Failures, Google SRE book.

9 — Partitioning

• Consistent Hashing and Random Trees, Karger et al.
• Dynamo, on virtual nodes in production.

10 — Replication and consistency

• Dynamo, DeCandia et al.
• Spanner, Corbett et al.
• Conflict-free Replicated Data Types: An Overview, Preguiça et al.

11 — Clocks, consensus and distributed locks

• Time, Clocks, and the Ordering of Events, Leslie Lamport.
• In Search of an Understandable Consensus Algorithm, Ongaro and Ousterhout.
• How to do distributed locking, Martin Kleppmann.

12 — Storage engines

• B-Tree vs LSM-Tree, TiKV.
• The Log-Structured Merge-Tree, O’Neil et al.

13 — Delivery guarantees and transactions across services

• Transactional outbox and Saga, Chris Richardson.
• Debezium outbox event router.
• Kafka delivery semantics.

14 — Resilience: timeouts, retries, circuit breakers and overload

• Addressing Cascading Failures and Handling Overload, Google SRE book.
• Timeouts, retries and backoff with jitter, Amazon Builders’ Library.
• Circuit Breaker, Martin Fowler.
• Metastable Failures in Distributed Systems, Bronson et al.

15 — Services and APIs

• Monolith First, Martin Fowler.
• Hyrum’s Law.
• Zanzibar, Pang et al.

16 — Running it: observability, deployments and regions

• Site Reliability Engineering, Google.
• The USE Method, Brendan Gregg.
• OpenTelemetry documentation.

17 — Low-level design

• Hexagonal Architecture, Alistair Cockburn.
• Refactoring, Martin Fowler.

18 — A rate limiter, end to end

• How we built rate limiting capable of scaling to millions of domains, Cloudflare.

19 — Worked designs

• Designing Data-Intensive Applications, Martin Kleppmann.
• Dynamo, The Tail at Scale and the Raft paper.

Guides

Big-O complexity, and how to approach it in real code

• Big-O Cheat Sheet.
• TimeComplexity, Python wiki.
• Introduction to Algorithms, Cormen, Leiserson, Rivest and Stein.

Threads, concurrency and parallelism

• Is Parallel Programming Hard?, Paul McKenney.
• The Go Memory Model.
• PEP 779, on free-threaded Python.
• The Art of Multiprocessor Programming, Herlihy and Shavit.

How Kafka works, from 0 to 1000

• Kafka documentation.
• The Log: What every software engineer should know about real-time data, Jay Kreps.
• Kafka 4.0 release announcement.
• Delivery semantics, Confluent.

PostgreSQL indexes: every kind, and when to use which

• Index Types, PostgreSQL documentation.
• CREATE INDEX.
• Multicolumn Indexes.
• Use The Index, Luke.
• pgvector.

Semaphore vs mutex vs lock

• The Little Book of Semaphores, Allen Downey.
• Mutexes and Semaphores Demystified, Michael Barr.
• futex(2).
• What really happened on Mars?, Mike Jones and Glenn Reeves.

Shooting manual on the Nikon D7500

• D7500 Online Manual, Nikon: metering.
• ISO sensitivity and auto ISO, Nikon.
• Exposure modes, Nikon.
• Specifications, Nikon.
• Auto ISO control tutorial, Nikon.

JWTs, access tokens, sessions, OAuth and Zero Trust

• RFC 7519: JSON Web Token.
• RFC 8725: JWT Best Current Practices.
• RFC 9068: JWT profile for OAuth access tokens.
• RFC 6749: OAuth 2.0.
• RFC 7636: PKCE.
• RFC 9700: OAuth security best current practice.
• RFC 8693: token exchange.
• RFC 8705: mutual TLS for OAuth.
• RFC 9449: DPoP.
• RFC 9470: step-up authentication.
• RFC 9901: SD-JWT.
• OAuth 2.1 draft.
• OAuth 2.0 for Browser-Based Applications, IETF draft.
• JWT attacks, PortSwigger.
• Psychic Signatures in Java, Neil Madden.
• Session Management Cheat Sheet, OWASP.
• OWASP API Security.
• NIST SP 800-207: Zero Trust Architecture.
• NIST SP 800-63B: authentication and session guidance.
• CISA Zero Trust Maturity Model 2.0.
• Shared Signals Framework and CAEP, OpenID Foundation.
• Continuous access evaluation, Microsoft.
• SPIFFE overview.

The Kubernetes control plane, Cilium, Flannel and OPA Gatekeeper

• Cluster Architecture, Kubernetes.
• Admission Control in Kubernetes.
• Admission Webhook Good Practices.
• API Priority and Fairness, Kubernetes blog.
• KEP-5343: Make nftables the default kube-proxy backend.
• KEP-5495: Deprecate ipvs mode in kube-proxy.
• Ingress NGINX Retirement, Kubernetes blog.
• Cilium: project documentation index.
• Using Gatekeeper with dry run, Mirantis.

RDMA, RoCE and NVIDIA networking

• RDMA Aware Networks Programming User Manual, NVIDIA.
• Congestion Control for Large-Scale RDMA Deployments: the DCQCN paper.
• RDMA over Commodity Ethernet at Scale, Microsoft.
• Data Center Congestion Management requirements, IETF draft.
• Soft-RoCE configuration, rdma-core.
• GPUDirect RDMA and GPUDirect Storage, NVIDIA.
• Powering next-generation AI networking with NVIDIA SuperNICs, NVIDIA.
• UEC launches Specification 1.0, Linux Foundation.
• Ultra Ethernet’s Design Principles and Architectural Innovations.
• Revisiting Network Support for RDMA: the IRN paper.

Redis internals: the protocol, the event loop and why one thread is fast

• Redis serialization protocol specification.
• OBJECT ENCODING: the memory encodings.
• Redis persistence: RDB and AOF.
• HELLO: selecting RESP3.
• Redis adds the AGPL, SD Times.
• Redis bets big on an open source return, InfoWorld.

How TLS works, and when it can be decrypted

• RFC 8446: TLS 1.3.
• RFC 9850: the key log format.
• RFC 10015: deprecating obsolete key exchanges in TLS 1.2.
• Cloudflare Radar: post-quantum adoption.