7 — `connect()`: where the SYN is born

When a process calls connect(), where exactly does the kernel decide where the SYN goes, and what gets a say before it does?

Requirements: a Debian 13 guest, set up below. It has 6.12.y with bpf in the LSM list. It also helps to have read chapter 1.

Set up the network lab

Chapters 7 to 9 share this setup.

Install the tools in the guest:

apt-get update
apt-get install -y bpftrace bpftool clang llvm libbpf-dev iproute2 ethtool tcpdump netcat-openbsd iptables
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

Create two network namespaces joined by a virtual cable:

ip netns add client; ip netns add server
ip link add veth-c netns client type veth peer name veth-s netns server
ip -n client addr add 10.0.0.1/24 dev veth-c
ip -n server addr add 10.0.0.2/24 dev veth-s
ip -n client link set veth-c up; ip -n server link set veth-s up
ip -n client link set lo up;    ip -n server link set lo up
mkdir -p /sys/fs/cgroup/lab

Mount the BPF filesystem if /sys/fs/bpf is empty: mount -t bpf bpf /sys/fs/bpf.

The path

Read it top to bottom with the tree open.

Into the socket layer

Step 1. The call arrives through x64_sys_call at __x64_sys_connect, and then __sys_connect (net/socket.c). It uses fdget() and sock_from_file() to get the socket, and move_addr_to_kernel() to copy the address in.

Step 2. __sys_connect_file calls security_socket_connect, the first place policy speaks. LSMs see the address and can return -EPERM, but they can’t change it. Then it calls sock->ops->connect, which is inet_stream_connect for TCP.

Step 3. inet_stream_connect takes the socket lock and calls __inet_stream_connect (net/ipv4/af_inet.c), a switch on the socket’s state. For SS_UNCONNECTED, it does this:

if (BPF_CGROUP_PRE_CONNECT_ENABLED(sk)) {
    err = sk->sk_prot->pre_connect(sk, uaddr, addr_len);

Then it calls sk->sk_prot->connect. If the socket blocks, it sleeps in inet_wait_for_connect. If not, it returns -EINPROGRESS. Either way, the SYN has already gone out, or failed to.

The hook

Step 4. That if matters. tcp_v4_pre_connect is not called at all unless a connect program is attached (bpf-cgroup.h). The trace below shows this.

Step 5. When a program is attached, tcp_v4_pre_connect runs __cgroup_bpf_run_filter_sock_addr (kernel/bpf/cgroup.c), the second place policy speaks. It runs the cgroup’s effective programs, which can read and write ctx->user_ip4 and ctx->user_port. If one returns 0, connect() fails with -EPERM.

This is the only place on this path that can change the destination before any route exists.

Where the destination becomes real

Step 6. tcp_v4_connect (net/ipv4/tcp_ipv4.c) does the substance:

Route lookup. A preliminary lookup comes first, then ip_route_output_flow, and ip_route_newports can add another (route.h). This returns a struct rtable with the output device and next hop. Now “where” is committed.
State. tcp_set_state(sk, TCP_SYN_SENT) runs first.
Port. inet_hash_connect then picks an ephemeral port and inserts the socket into the hash.
Sequence number. It comes from secure_tcp_seq().
Send. tcp_connect(sk).

The SYN

Step 7. tcp_connect (net/ipv4/tcp_output.c) calls tcp_call_bpf first, the sock_ops callback, which is informational. Then tcp_connect_init fills in MSS and window, and it allocates the SYN skb and sends it. It then arms a retransmit timer using icsk_rto, 1 second unless sock_ops overrides it.

Step 8. __tcp_transmit_skb builds the TCP header and calls ip_queue_xmit.

Step 9. Down the IP stack:

__ip_queue_xmit uses the route cached on the socket and builds the IP header.
__ip_local_out runs NF_INET_LOCAL_OUT, netfilter’s first look. iptables REDIRECT lives here, after the route exists.
ip_output runs NF_INET_POST_ROUTING.
ip_finish_output runs the cgroup egress hook, which can drop the packet (chapter 8).
ip_finish_output2 leads to __dev_queue_xmit, the qdisc and the driver.

Coming back

The SYN-ACK arrives through tcp_v4_rcv, tcp_v4_do_rcv and tcp_rcv_state_process, reaching tcp_rcv_synsent_state_process. That calls tcp_finish_connect and then wakes the sleeping connect() with sk->sk_state_change.

The answer

The destination is decided at the route lookup in tcp_v4_connect.
It can be refused at security_socket_connect before that, or at ip_finish_output after.
On this path, the cgroup connect4 hook is the only place that changes it before any route exists.

Netfilter NAT can change it later, as covered below (nf_nat_proto.c).

Hands-on: trace it

Check the probes exist on your kernel:

for f in tcp_v4_connect tcp_v4_pre_connect __tcp_transmit_skb; do
  grep -qw "$f" /sys/kernel/tracing/available_filter_functions && echo "ok $f" || echo "NO $f"
done

Start a listener in a second shell:

ip netns exec server nc -l 10.0.0.2 8000

Trace a connection with no BPF attached:

bpftrace -e '
kprobe:__sys_connect            { @t[tid]=nsecs; printf("connect %s\n", comm); }
kprobe:security_socket_connect  /@t[tid]/ { printf("lsm  +%d\n", nsecs-@t[tid]); }
kprobe:tcp_v4_pre_connect       /@t[tid]/ { printf("cg   +%d\n", nsecs-@t[tid]); }
kprobe:ip_route_output_key_hash /@t[tid]/ { printf("rt   +%d\n", nsecs-@t[tid]); }
kprobe:tcp_connect              /@t[tid]/ { printf("syn  +%d\n", nsecs-@t[tid]); }
kprobe:ip_finish_output         /@t[tid]/ { printf("egr  +%d\n", nsecs-@t[tid]); }
kprobe:__dev_queue_xmit         /@t[tid]/ { printf("dev  +%d\n", nsecs-@t[tid]); delete(@t[tid]); }'

Then, in a third shell: ip netns exec client nc -z -w2 10.0.0.2 8000

Two things stand out. There should be no cg line, because nothing is attached. There should be more than one rt line, because of the preliminary lookup.

If delete(@t[tid]) errors on your bpftrace version, try delete(@t, tid).

Hands-on: change it

This moves a connection’s destination before any route exists, which is the mechanism behind transparent proxies and socket-level load balancers.

Create redirect80.bpf.c:

#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

SEC("cgroup/connect4")
int redirect80(struct bpf_sock_addr *ctx)
{
    if (ctx->user_port == bpf_htons(80)) {
        ctx->user_ip4  = bpf_htonl(0x7f000001);   /* 127.0.0.1 */
        ctx->user_port = bpf_htons(8080);
    }
    return 1;                                     /* 0 would give EPERM */
}
char LICENSE[] SEC("license") = "GPL";

Build it, load it and attach it to the cgroup:

clang -O2 -g -target bpf -c redirect80.bpf.c -o redirect80.bpf.o
bpftool prog load redirect80.bpf.o /sys/fs/bpf/redirect80
bpftool cgroup attach /sys/fs/cgroup/lab connect4 pinned /sys/fs/bpf/redirect80 multi
bpftool cgroup tree /sys/fs/cgroup

Run a client in a namespace with only loopback and no route to the target. 203.0.113.1 is a reserved documentation address:

ip netns add iso; ip netns exec iso ip link set lo up
ip netns exec iso sh -c 'nc -l 127.0.0.1 8080 & echo $$ > /sys/fs/cgroup/lab/cgroup.procs; nc -z -w2 203.0.113.1 80'

Expected: the listener should see a connection, and cg should now appear before the first rt in your trace.
Prove it. Detach the program and repeat:
```
bpftool cgroup detach /sys/fs/cgroup/lab connect4 pinned /sys/fs/bpf/redirect80
```
With no route to 203.0.113.1, the same command should now fail. That proves the rewrite happened before the route lookup.
Compare with iptables. Add kprobe:ip_route_me_harder to your trace and use:
```
iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-ports 8080
```
The effect is similar, but it happens after the route, with a re-route to fix up the first decision. The rule should be removed afterwards.

If it fails

Your socket ignores the cgroup: a socket’s cgroup is fixed at creation, so a socket made before the shell joined keeps its old one.
getpeername() shows 127.0.0.1:8080: that’s expected unless cgroup/getpeername4 is attached too.
nc rejects flags: variants differ. Check nc -h.
A kprobe won’t attach: the function may be inlined, so use the check in step 1.

The takeaway

A path has an order of authority: who sees a decision, who can veto it, who can change it, and the point after which nobody can. On this path the LSM can veto but not change; the cgroup hook can change; the route commits; netfilter can only re-route after the fact; the egress hook can veto the packet but not redirect the socket. Designing a control means choosing the point in the path whose authority matches what you need, and the earliest point is almost always the cheapest and the hardest to bypass. The same analysis applies to any request pipeline: find the one point where the decision is still fluid and put the policy there.

7 — connect(): where the SYN is born