9 — Namespaces and cgroups

Requirements: root in a disposable VM, /lo/rootfs from chapter 3, and cgroup v2 at /sys/fs/cgroup. Check with stat -fc %T /sys/fs/cgroup, which should say cgroup2fs. Don’t do this on a machine you care about, as mistakes in the root cgroup are awkward.

The parts

A container is four things:

Docker is a convenience over exactly these. This walkthrough covers the first three. User namespaces are what make rootless containers possible, though they aren’t used here (namespaces(7)).

Hands-on

Step 1: check the controllers

cat /sys/fs/cgroup/cgroup.controllers

Expected: it includes memory and pids. If not, those options were dropped from your kernel config.

Step 2: create a cgroup with limits

Enable the controllers for children first, or memory.max won’t exist in the child (cgroup-v2):

echo "+memory +pids" > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/lab
echo 64M > /sys/fs/cgroup/lab/memory.max
echo 16 > /sys/fs/cgroup/lab/pids.max

Expected: cat /sys/fs/cgroup/lab/memory.max prints 67108864, which is 64 × 1024 × 1024.

Step 3: watch tasks move

In a second terminal:

bpftrace -e 'tracepoint:cgroup:cgroup_attach_task { printf("%s pid %d -> %s\n", str(args.comm), args.pid, str(args.dst_path)); }'

Step 4: enter the group and new namespaces

echo $$ > /sys/fs/cgroup/lab/cgroup.procs
unshare --pid --fork --mount-proc --mount --uts --net --ipc /bin/sh

--mount-proc matters. Without it, /proc still shows the outer PIDs, and ps won’t show what you expect.

Expected, inside: echo $$ prints 1, ps shows very few processes, ip link shows only lo, and hostname inner doesn’t change the name outside. Your outer shell joined the cgroup too, which is fine in a lab.

From outside, the same process has a different PID. Compare the two views.

Step 5: switch root

Still inside:

mount --make-rprivate /
mount --bind /lo/rootfs /lo/rootfs
cd /lo/rootfs
mkdir -p .old
pivot_root . .old
umount -l /.old
mount -t proc proc /proc

new_root must be a mount point, hence the bind mount, and it can’t be shared, hence --make-rprivate (pivot_root(2)).

Expected: ls / shows only your system.

Step 6: hit pids.max

Inside:

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do sleep 30 & done

Expected: the shell reports fork failures once the limit is reached. Outside, cat /sys/fs/cgroup/lab/pids.events shows a non-zero max, and forks there still work. The limit applies only inside.

Step 7: hit memory.max

Run a memory hog from a child you can afford to lose, since the shell itself might be killed:

busybox dd if=/dev/zero of=/dev/null bs=128M count=1

Attach tracepoint:oom:mark_victim first.

Expected: the process is killed, grep oom_kill /sys/fs/cgroup/lab/memory.events shows a non-zero count, and dmesg records an OOM in the cgroup.

Step 8: clean up

exit
echo $$ > /sys/fs/cgroup/cgroup.procs
rmdir /sys/fs/cgroup/lab

Expected: rmdir succeeds. If it says busy, a process remains, so check cgroup.procs.

If it fails

• No memory.max: Step 2’s subtree_control failed.
• Device or resource busy on subtree_control: this is the no-internal-process rule. A non-root cgroup can’t distribute resources while it holds processes itself.
• Invalid argument from pivot_root: a mount point or sharing rule was broken.
• fork: Cannot allocate memory in a PID namespace: its init exited (pid_namespaces(7)).
• unshare ignores Ctrl-C: documented behaviour. Signal the child.

Principle: isolation is a set of kernel objects, not a product

Each part of a container is a kernel object you can create, inspect and trace independently. Understanding isolation as objects rather than a product means you can reason about what a sandbox actually guarantees, and what it doesn’t. Your container here still shares the cgroup, time and user namespaces with its host.