1 — A system call, from `syscall` to `return`

Between the syscall instruction and the next user instruction, what runs, and why is “return to user mode” the most interesting half?

Requirements: the machine from chapter 0, and enough C to read a short program.

This chapter assumes classic SYSCALL entry. Newer Intel FRED hardware takes a different route, which isn’t covered here.

The path

Follow one call in order. Open each file as you go.

Step 1: the CPU jumps. The SYSCALL instruction loads the instruction pointer from the MSR_LSTAR register, which the kernel set at boot (cpu/common.c). It saves the return address in RCX and the flags in R11 (SYSCALL).

Step 2: entry_SYSCALL_64 (arch/x86/entry/entry_64.S). Swap to the kernel GS, switch to the kernel stack and page tables, and push a struct pt_regs holding the user’s registers. Everything a tracer shows you as “args” comes from this struct.

Step 3: do_syscall_64 (arch/x86/entry/common.c). This calls syscall_enter_from_user_mode, the first hook point. Ptrace, seccomp and the raw_syscalls:sys_enter tracepoint all live here. Seccomp runs before the tracepoint, so a filtered call never reaches sys_enter.

Step 4: dispatch. do_syscall_x64 calls x64_sys_call, a generated switch statement. Older write-ups describe sys_call_table[nr](regs), but since 6.9 the source says that table “is no longer used for system calls” (common.c, syscall_64.c).

The list of calls lives in arch/x86/entry/syscalls/syscall_64.tbl. SYSCALL_DEFINEn macros generate __x64_sys_* wrappers that unpack pt_regs into typed arguments.

Step 5: the work. Whatever the call does. Chapters 2 to 9 each follow one of these.

Step 6: the way back. syscall_exit_to_user_mode reaches exit_to_user_mode_loop in kernel/entry/common.c, the second hook point, and the busier one. It loops while work is pending:

_TIF_NEED_RESCHED: call schedule()
_TIF_SIGPENDING or _TIF_NOTIFY_SIGNAL: deliver signals (chapter 12)
_TIF_NOTIFY_RESUME: run task work, such as finishing file closes
_TIF_UPROBE: handle uprobes
_TIF_PATCH_PENDING: live patching

So a syscall that “returns” may run another process first.

Step 7: back to user mode. The fast exit is syscall_return_via_sysret. swapgs_restore_regs_and_return_to_usermode is the slower IRET exit. SYSRET can’t always be used, because it needs a canonical address in RCX (SYSRET).

Hands-on: trace it

Part 1: count syscalls

Count syscalls by number for five seconds:

bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[args.id] = count(); } interval:s:5 { exit(); }'

Look up the numbers in syscall_64.tbl. getpid is 39 and openat is 257.

Part 2: find what can’t be traced

Try to trace the function you just read:

cd /sys/kernel/tracing
grep -w do_syscall_64 available_filter_functions
echo do_syscall_64 > set_graph_function

Expect: the grep prints nothing, and the echo fails with “Invalid argument”.

do_syscall_64 is marked noinstr, because code this early can’t safely run a tracer (compiler_types.h). Tools have blind spots, and the boundary itself is one.

Part 3: trace one layer in

Create gp.c:

#include <unistd.h>
#include <sys/syscall.h>
int main(void) {
    for (;;) { syscall(SYS_getpid); sleep(1); }
}

Build it and graph the call:

gcc -o gp gp.c
cd /sys/kernel/tracing
echo nop > current_tracer
echo __x64_sys_getpid > set_graph_function
echo function_graph > current_tracer
~/gp & echo $! > set_ftrace_pid
sleep 3; head -40 trace

Clean up, or your next experiment will be silently filtered:

echo nop > current_tracer; echo > set_graph_function; echo > set_ftrace_pid; kill %1

Look for: __x64_sys_getpid and anything it calls. Measure the duration yourself.

Part 4: watch exit work

exit_to_user_mode_loop is __always_inline here, so you can’t probe it directly. Probe what it calls.

Write sig.c, a program named sig with a SIGALRM handler and a setitimer timer that fires regularly.

Attach:

bpftrace -e 'kprobe:arch_do_signal_or_restart /comm == "sig"/ { @sig = count(); }
             kprobe:task_work_run { @tw[comm] = count(); }'

Expected: @sig should follow your timer’s rate.

Hands-on: change it

This blocks openat with seccomp and shows where the filter runs.

Create sc.c:

#include <errno.h>
#include <fcntl.h>
#include <stddef.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <linux/audit.h>
#include <linux/filter.h>
#include <linux/seccomp.h>

int main(void) {
    struct sock_filter f[] = {
        BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, arch)),
        BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, AUDIT_ARCH_X86_64, 1, 0),
        BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_KILL_PROCESS),
        BPF_STMT(BPF_LD|BPF_W|BPF_ABS, offsetof(struct seccomp_data, nr)),
        BPF_JUMP(BPF_JMP|BPF_JEQ|BPF_K, __NR_openat, 0, 1),
        BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ERRNO | EPERM),
        BPF_STMT(BPF_RET|BPF_K, SECCOMP_RET_ALLOW),
    };
    struct sock_fprog p = { sizeof f / sizeof f[0], f };
    prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
    if (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &p)) { perror("seccomp"); return 1; }
    sleep(2);
    for (int i = 0; i < 5; i++) {
        int fd = open("/etc/hostname", O_RDONLY);
        dprintf(2, "open=%d errno=%d\n", fd, errno);
    }
    return 0;
}

Build it and run it under this trace:

gcc -o sc sc.c
bpftrace -e 'tracepoint:raw_syscalls:sys_enter /comm == "sc" && args.id == 257/ { @enter = count(); }
             tracepoint:raw_syscalls:sys_exit  /comm == "sc" && args.id == 257/ { @exit = count(); }' &
./sc

Expected: the loader’s openat calls run before the filter, so they’re counted in both maps. The five filtered calls skip sys_enter, so @exit should be exactly 5 higher than @enter.
Expected: remove the filter and run again. The two counts should now match. If it fails: prctl returning EACCES means PR_SET_NO_NEW_PRIVS was skipped.

The takeaway

Boundaries are where the hooks live. Every cross-privilege transition in a system is a place to observe, filter and defer work, and the kernel concentrates its policy at exactly two: entering and leaving. The exit path does more than the entry because that’s the last chance before control is lost. Design your own boundaries (RPC handlers, job runners) the same way: one entry, one exit, and all the cross-cutting concerns at those two points.

1 — A system call, from syscall to return