8 — A packet out: send() to the NIC

After connect(), a write() of 100 KiB: how many packets, decided where, and at which points can something refuse or reshape them?

Requirements: the guest and network namespaces from chapter 7.

The path

Step 1: into TCP. __sys_sendto or sock_write_iter reaches __sock_sendmsg, which runs security_socket_sendmsg and then inet_sendmsg. That calls tcp_sendmsg and tcp_sendmsg_locked (net/ipv4/tcp.c), which copy your data into skbs from tcp_stream_alloc_skb and queue them on the socket.

A socket in a sockmap is different: tcp_bpf_sendmsg replaces tcp_sendmsg, so an sk_msg program runs before TCP.

Step 2: decide what can go now. tcp_push reaches tcp_write_xmit (net/ipv4/tcp_output.c), which checks in this order (tcp_output.c):

1. tcp_pacing_check: is pacing holding us back?
2. tcp_cwnd_test: does the congestion window allow it?
3. tcp_snd_wnd_test: does the receiver’s window allow it?
4. Nagle and TSO deferral.
5. tso_fragment, if needed.
6. tcp_small_queue_check: limits bytes queued below this socket.
7. tcp_transmit_skb.

Step 3: through IP. __tcp_transmit_skb builds the header and calls __ip_queue_xmit. Then __ip_local_out runs NF_INET_LOCAL_OUT, and ip_output runs NF_INET_POST_ROUTING. ip_finish_output runs the cgroup egress BPF hook. ip_finish_output2 handles neighbour resolution and reaches __dev_queue_xmit.

Step 4: the device layer. __dev_queue_xmit (net/core/dev.c) runs TC egress with sch_handle_egress, before the qdisc. Then comes the qdisc, in __dev_xmit_skb. The fq qdisc is one place pacing can be enforced.

Step 5: to the driver. sch_direct_xmit calls validate_xmit_skb_list, and GSO segmentation happens there, if the device can’t do TSO (sch_generic.c). Then dev_hard_start_xmit reaches ndo_start_xmit. virtio_net’s start_xmit adds descriptors, then calls virtqueue_kick_prepare and virtqueue_notify.

Step 6: cleanup. Completion frees the skb with napi_consume_skb, and tcp_wfree returns the memory to the socket. Pure ACKs use __sock_wfree.

Hands-on: trace it

1. Start a receiver and count the path:

   ip netns exec server nc -l 10.0.0.2 9000 >/dev/null &
   bpftrace -e 'kprobe:tcp_sendmsg,kprobe:tcp_write_xmit,kprobe:__tcp_transmit_skb,kprobe:__skb_gso_segment { @[probe]=count(); }
   tracepoint:net:net_dev_xmit { @len=hist(args.len); }'

2. Send 100 KiB from another shell:

   head -c 102400 /dev/zero | ip netns exec client nc -q1 10.0.0.2 9000

3. Look for: large lengths in @len. len is skb->len, so with TSO it shows the big skb, not wire-sized segments (net.h).

4. Turn offload off and repeat:

   ip netns exec client ethtool -k veth-c | grep -E 'segmentation|generic'
   ip netns exec client ethtool -K veth-c tso off gso off

5. Expected: lengths should fall to about MSS plus headers.

Hands-on: change it

This drops one flow and reads the reason the kernel gives.

1. Create drop_port.bpf.c. In a cgroup_skb program, offsets start at the IP header, not Ethernet:

   #include "vmlinux.h"
   #include <bpf/bpf_helpers.h>
   #include <bpf/bpf_endian.h>
   
   SEC("cgroup_skb/egress")
   int drop_port(struct __sk_buff *skb)
   {
       __u8 vihl, proto; __u16 dport;
       if (bpf_skb_load_bytes(skb, 0, &vihl, 1) || (vihl >> 4) != 4) return 1;
       if (bpf_skb_load_bytes(skb, 9, &proto, 1) || proto != 6) return 1;
       if (bpf_skb_load_bytes(skb, (vihl & 0xf) * 4 + 2, &dport, 2)) return 1;
       return dport == bpf_htons(9000) ? 0 : 1;
   }
   char LICENSE[] SEC("license") = "GPL";

2. Build it, attach it, and watch drop reasons:

   clang -O2 -g -target bpf -c drop_port.bpf.c -o drop_port.bpf.o
   bpftool prog load drop_port.bpf.o /sys/fs/bpf/drop_port
   bpftool cgroup attach /sys/fs/cgroup/lab egress pinned /sys/fs/bpf/drop_port multi
   echo 1 > /sys/kernel/tracing/events/skb/kfree_skb/enable
   cat /sys/kernel/tracing/trace_pipe | grep BPF_CGROUP_EGRESS &

3. Join the cgroup and connect:

   echo $$ > /sys/fs/cgroup/lab/cgroup.procs
   ip netns exec client nc -z -w2 10.0.0.2 9000

4. Expected: trace_pipe should show reason: BPF_CGROUP_EGRESS, which is SKB_DROP_REASON_BPF_CGROUP_EGRESS (dropreason-core.h). ip netns exec server tcpdump -ni veth-s port 9000 should show no SYN.

If it fails

• nc just waits: tcp_connect arms the retransmit timer rather than reporting most failures (tcp_output.c).
• Nothing is dropped: the program only applies to sockets created in the cgroup.
• A probe won’t attach: many helpers here are inlined, including tcp_cwnd_test and sch_handle_egress. Use __tcp_transmit_skb rather than tcp_transmit_skb.

The takeaway

Backpressure is designed, not discovered. TSQ, cwnd, the pacing qdisc and socket memory accounting are four separate mechanisms whose only purpose is to stop a fast producer from drowning a slow link, each at a different layer, each with its own notion of “too much”. Systems that lack this fail by queuing until they fall over. When you design a pipeline, put a limit at every layer and a reason code on every drop.