6 — The page cache and readahead

For read(fd, buf, 1 MiB), how much is memory copying, how much is I/O, and who decided to read more than you asked for?

Requirements: the machine from chapter 0, with ext4 on a virtio disk. virtiofs and 9p skip the block layer, so these steps won’t apply there.

The path

Step 1: into the VFS. ksys_read calls vfs_read, which calls f_op->read_iter. On ext4 that’s ext4_file_read_iter, then generic_file_read_iter, then filemap_read (mm/filemap.c).

Step 2: look in the cache. filemap_read loops. filemap_get_pages calls filemap_get_read_batch, which looks up folios in the file’s address_space xarray by index.

Step 3: on a miss, read ahead. page_cache_sync_ra decides on a window and reaches page_cache_ra_order or page_cache_ra_unbounded. That allocates folios, inserts them in the xarray, and submits reads through a_ops->readahead. On ext4 that’s ext4_mpage_readpages, and then submit_bio.

The window defaults to 128 KiB (read_ahead_kb), which is a floor a device can raise (blk-settings.c). ext4 has no large-folio support at v6.12, so its folio order stays 0.

Step 4: keep it going. Readahead marks a folio with PG_readahead. Reaching that folio triggers page_cache_async_ra, which starts the next window before you need it.

Step 5: wait and copy. If a folio isn’t ready, the reader waits with folio_put_wait_locked. copy_folio_to_iter is the real copy into your buffer.

Step 6: below the page cache. submit_bio goes to submit_bio_noacct, then blk_mq_submit_bio, an I/O scheduler if one is set, and the driver (virtio_queue_rq for virtio). Completion runs blk_mq_complete_request and bio_endio, and folio_end_read wakes the reader.

Writes mirror this: generic_perform_write, dirty folios, then writeback.

Hands-on: see the window

1. Prepare a cold file on ext4:

   cat /sys/block/vda/queue/read_ahead_kb
   dd if=/dev/urandom of=/root/big bs=1M count=256 && sync
   echo 3 > /proc/sys/vm/drop_caches
   findmnt -no FSTYPE /

2. Watch the window grow. arg1 is nr_to_read, counted in pages:

   bpftrace -e 'kprobe:page_cache_ra_unbounded /comm == "dd"/ { printf("ra=%lu\n", arg1); }
                tracepoint:block:block_rq_issue /comm == "dd"/ { @io = hist(args.bytes); }' &
   dd if=/root/big of=/dev/null bs=4k count=2000

3. Expected: ra= values of 4, 8, 16, 32, then 32 again. If @io is empty, drop the comm filter.

4. Compare with cat:

   echo 3 > /proc/sys/vm/drop_caches
   bpftrace -e 'kprobe:page_cache_ra_unbounded /comm == "cat"/ { @[arg1] = count(); }' &
   cat /root/big > /dev/null

   Expected: mostly 64, with no ramp, because coreutils cat sets POSIX_FADV_SEQUENTIAL.

5. Read it warm:

   cd /sys/kernel/tracing
   echo filemap_read > set_graph_function; echo 6 > max_graph_depth
   echo function_graph > current_tracer
   dd if=/root/big of=/dev/null bs=1M count=1
   cp trace /tmp/warm.txt; echo nop > current_tracer; echo > set_graph_function
   grep -c submit_bio /tmp/warm.txt

   Expected: 0 for a fully cached range.

Hands-on: change it

1. Turn readahead off. Do it before the reader opens the file, because the limit is copied at open (open.c):

   echo 0 > /sys/block/vda/queue/read_ahead_kb
   echo 3 > /proc/sys/vm/drop_caches
   bpftrace -e 'tracepoint:block:block_rq_issue { @io = hist(args.bytes); }' &
   dd if=/root/big of=/dev/null bs=4k count=2000
   echo 128 > /sys/block/vda/queue/read_ahead_kb

   Expected: 4 KiB requests. Write down the throughput you measure, and don’t use a number from a non-KVM run.

2. Try random access. In a small C program, call posix_fadvise(fd, 0, 0, POSIX_FADV_RANDOM), then pread 16 KiB at random offsets, and probe it:

   bpftrace -e 'kprobe:force_page_cache_ra { printf("%lu\n", arg1); }'

   Expected: 4 pages each, with no ramp. This path doesn’t write ra->size, so don’t look for it there.

3. Plot the shape:

   bpftrace -e 'tracepoint:filemap:mm_filemap_add_to_page_cache /comm == "dd"/ {
     printf("%lu %lu %d\n", args.i_ino, args.index, args.order); }' > /tmp/adds.txt

   Plot index against line number. order should be 0 throughout on ext4.

4. Patch it. In page_cache_async_ra in mm/readahead.c, inside the if (index == expected) block and just before goto readit, add:

   pr_info_ratelimited("ra: start=%lu size=%u async=%u\n",
                       ra->start, ra->size, ra->async_size);

   Expected: dmesg values should match what you saw in the window experiment.

If it fails

• ra= is empty: the file is cached, or the filesystem has large-folio support.
• Requests are larger than the window: merging or plugging.
• read_ahead_kb seems ignored: the file was already open.
• No vda: check lsblk.

The takeaway

Predict the next request and pay for it early. Readahead is speculation with a feedback loop: sequential hits grow the window, a miss resets it, and async readahead keeps the pipeline full. Every prefetcher (CPU, CDN, database buffer pool) is this loop, and the knobs are the same two: how far ahead, and how to tell when you were wrong.