9 — A packet in: NIC to recv()

A frame arrives at the NIC. How does it become bytes in a recv() buffer, and where are the earliest points the kernel will run your code on it?

Requirements: the guest from chapter 7, with a virtio network device. This matters, because only native XDP on virtio shows the real behaviour. A plain veth can’t.

The path

Step 1: the interrupt. The hardware interrupts, and the driver schedules NAPI with napi_schedule and quiets further interrupts. net_rx_action, the NET_RX softirq, calls the driver’s poll. On virtio that’s virtnet_poll, which pulls frames from the ring. Each NAPI instance’s budget is usually 64 (NAPI_POLL_WEIGHT), and the softirq’s total is netdev_budget (NAPI docs).

Step 2: XDP. In the driver’s poll, before an skb exists, bpf_prog_run_xdp runs on the raw buffer. It returns XDP_DROP, XDP_TX, XDP_REDIRECT or XDP_PASS. In native mode, this is the cheapest place in the kernel to drop a packet.

Step 3: GRO. napi_gro_receive merges consecutive TCP segments of one flow into one big skb. From here there are two paths:

• The NAPI path: gro_normal_one, netif_receive_skb_list_internal, ip_list_rcv.
• The single-packet path, as used by loopback through process_backlog: __netif_receive_skb_core and ip_rcv.

Step 4: the core. __netif_receive_skb_core (net/core/dev.c) runs ptype_all taps, which is where tcpdump sees the packet, then TC ingress and the protocol demux.

Step 5: IP. ip_rcv or ip_list_rcv runs NF_INET_PRE_ROUTING and does the route lookup. ip_local_deliver runs NF_INET_LOCAL_IN, and ip_protocol_deliver_rcu calls tcp_v4_rcv. Both paths meet here, so tcp_v4_rcv is the best place to probe.

Step 6: TCP. tcp_v4_rcv finds the socket. sk_filter_trim_cap runs cgroup ingress BPF and classic socket filters. If user code owns the socket, the packet goes to the backlog. Otherwise tcp_v4_do_rcv calls tcp_rcv_established, the fast path, which queues data with tcp_queue_rcv. tcp_data_ready then calls sock_def_readable, which wakes the waiter.

Step 7: recv(). tcp_recvmsg and tcp_recvmsg_locked walk the receive queue, copy to your buffer, and may send a window update with tcp_cleanup_rbuf.

RSS, RPS and RFS decide which CPU does all this (scaling.rst).

Hands-on: count packets with XDP

1. Create xdp_count.bpf.c:

   #include "vmlinux.h"
   #include <bpf/bpf_helpers.h>
   #include <bpf/bpf_endian.h>
   
   struct {
       __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
       __uint(max_entries, 256);
       __type(key, __u32);
       __type(value, __u64);
   } counts SEC(".maps");
   
   SEC("xdp")
   int xdp_count(struct xdp_md *ctx)
   {
       void *data = (void *)(long)ctx->data;
       void *data_end = (void *)(long)ctx->data_end;
       struct ethhdr *eth = data;
       struct iphdr *ip = data + sizeof(*eth);
       struct udphdr *udp;
       __u32 key; __u64 *v;
   
       if ((void *)(ip + 1) > data_end || eth->h_proto != bpf_htons(0x0800))
           return XDP_PASS;
       key = ip->protocol;
       v = bpf_map_lookup_elem(&counts, &key);
       if (v) (*v)++;
       if (ip->protocol == 17 && ip->ihl == 5) {
           udp = (void *)(ip + 1);
           if ((void *)(udp + 1) > data_end) return XDP_PASS;
           if (udp->dest == bpf_htons(5555)) return XDP_DROP;
       }
       return XDP_PASS;
   }
   char LICENSE[] SEC("license") = "GPL";

2. Boot QEMU with a forwarded UDP port:

   -netdev user,id=n0,hostfwd=udp::5555-:5555 -device virtio-net-pci,netdev=n0

3. In the guest, build and attach it in native mode. Find your interface name with ip link, as it may not be eth0:

   clang -O2 -g -target bpf -c xdp_count.bpf.c -o xdp_count.bpf.o
   ip link set dev eth0 xdpdrv obj xdp_count.bpf.o sec xdp
   ip -d link show eth0

4. Trace it, recording a quiet baseline first:

   bpftrace -e 'tracepoint:skb:kfree_skb { @=count(); } kprobe:tcp_v4_rcv { @tcp=count(); }'

5. On the host, send UDP to the forwarded port:

   echo hi | nc -u -w1 127.0.0.1 5555

6. Expected: ethtool -S eth0 | grep xdp_drops should rise, and bpftool map dump name counts should show protocol 17. These packets should add nothing to kfree_skb, because no skb was ever allocated.

Hands-on: see the difference in generic mode

1. Switch modes:

   ip link set dev eth0 xdpdrv off
   ip link set dev eth0 xdpgeneric obj xdp_count.bpf.o sec xdp

2. Send UDP again.

3. Expected: now kfree_skb should fire on each drop. Generic XDP runs after an skb already exists, so it has to free it. Same program, different cost.

Hands-on: find the two receive paths

1. With XDP off, check the tracepoint’s fields, then trace:

   bpftrace -lv tracepoint:net:netif_receive_skb
   bpftrace -e 'kprobe:ip_rcv { @ip_rcv=count(); } kprobe:ip_list_rcv { @ip_list_rcv=count(); }
   tracepoint:net:napi_gro_receive_entry { @gro=count(); } kprobe:tcp_v4_rcv { @tcp=count(); }'

2. Compare traffic on eth0 with traffic on lo.

3. Look for: ip_list_rcv on the NAPI path, ip_rcv on loopback, and @gro larger than @tcp during a bulk transfer, when segments are coalesced.

If it fails

• Can't set XDP while host is implementing GRO_HW/CSUM: virtio offloads are on (virtio_net.c). Turn them off in QEMU or with ethtool.
• ICMP doesn’t reach the guest: user-mode QEMU doesn’t pass real pings, which is why this uses UDP.
• No traffic at all: check the interface name.

The takeaway

Do the cheap rejection first. The receive path is a funnel: each stage allocates more state and knows more context. XDP knows almost nothing and costs almost nothing; the socket layer knows everything and has paid for all of it. Put the filter at the earliest stage that has the information it needs and not one stage later. That’s a rule for API gateways and auth middleware as much as for packets.