The Kubernetes control plane, Cilium, Flannel and OPA Gatekeeper
Kubernetes is often described as a container orchestrator. A more useful description is a database with an API, surrounded by programs that each watch for changes and try to make the world match what the database says.
Nothing in Kubernetes runs a central script of “create the pod, then attach the network, then start the container”. Separate components each do one job when they see work to do. That one design explains most of its behaviour, including how it recovers from failure and why debugging it can be confusing.
This article goes through the control plane, then what happens on the network, then how policy is enforced at the door. Details reflect Kubernetes v1.36.
The parts
CONTROL PLANE
+-----------------------------------------------------+
| kube-scheduler kube-controller-manager |
| | | |
| +-------+-------------+ |
| v |
| kube-apiserver <-------> etcd |
+----------------^------------------------------------+
| HTTPS: everything talks to the API server
+----------+-----------+
| |
+-----+--------+ +-----+--------+
| kubelet | | kubelet |
| kube-proxy | | kube-proxy |
| runtime, CNI | | runtime, CNI |
+--------------+ +--------------+
NODE NODE
| Component | Job |
|---|---|
kube-apiserver | The front door. Authenticates, authorises, validates and stores objects, and serves watches |
etcd | The consistent key-value store that holds all cluster state |
kube-scheduler | Chooses a node for each unscheduled pod |
kube-controller-manager | Runs the built-in controllers |
cloud-controller-manager | Talks to the cloud provider, on cloud clusters |
kubelet | Runs on each node, and makes that node’s pods real |
kube-proxy | Programs Service routing on each node, unless a CNI replaces it |
One rule matters most: only the API server talks to etcd. The scheduler, controllers and kubelets never touch the database, and they do not talk to each other. All coordination happens through objects in the API.
Declarative state and the reconcile loop
Most objects have two halves. spec is what you want, and status is what currently exists. A controller is a loop that closes the gap:
forever:
desired = read spec
actual = observe the world
make one change that moves actual towards desired
write status
The next program is a tiny model. A Store plays the API server: every write increases a resourceVersion and adds to an event log. One function reconciles a ReplicaSet.
class Store:
"""A tiny API server: every write bumps resourceVersion and emits an event."""
def __init__(self):
self.objects, self.events, self.rv = {}, [], 0
def write(self, kind, name, obj):
self.rv += 1
event = "MODIFIED" if (kind, name) in self.objects else "ADDED"
self.objects[(kind, name)] = obj
self.events.append((self.rv, event, kind, name))
def delete(self, kind, name):
self.rv += 1
del self.objects[(kind, name)]
self.events.append((self.rv, "DELETED", kind, name))
def watch(self, since):
return [e for e in self.events if e[0] > since]
def pods_of(store, rs):
return [n for (k, n), o in store.objects.items() if k == "Pod" and o["owner"] == rs]
serial = 0
def reconcile(store, rs):
global serial
want = store.objects[("ReplicaSet", rs)]["replicas"]
have = pods_of(store, rs)
for _ in range(want - len(have)):
serial += 1
store.write("Pod", f"{rs}-{serial}", {"owner": rs})
for name in have[want:]:
store.delete("Pod", name)
return want - len(have)
store = Store()
store.write("ReplicaSet", "web", {"replicas": 3})
seen, rounds = 0, 0
def drain():
global seen, rounds
while events := store.watch(seen):
seen = events[-1][0]
rounds += 1
print(f" round {rounds}: {len(events)} event(s), diff {reconcile(store, 'web'):+d}")
print("create replicas=3"); drain()
print("kill one pod"); store.delete("Pod", pods_of(store, "web")[0]); drain()
print("scale to 5, then to 2, before the controller wakes up")
store.write("ReplicaSet", "web", {"replicas": 5})
store.write("ReplicaSet", "web", {"replicas": 2}); drain()
print("final pods:", sorted(pods_of(store, "web")), "rv", store.rv)
Output:
create replicas=3
round 1: 1 event(s), diff +3
round 2: 3 event(s), diff +0
kill one pod
round 3: 1 event(s), diff +1
round 4: 1 event(s), diff +0
scale to 5, then to 2, before the controller wakes up
round 5: 2 event(s), diff -1
round 6: 1 event(s), diff +0
final pods: ['web-2', 'web-3'] rv 9
Round 2 shows a controller reacting to its own writes, seeing nothing to do, and stopping. That is normal.
Round 5 is the important one. Two scale changes arrived while the controller was busy. It never acted on replicas: 5. It read the current spec, saw 3 pods and wanted 2, then deleted one. It did not need to process the history correctly, or even see it.
This is level-triggered behaviour: act on the current state, not on each event. Events only say “something changed, go and look”. So a controller that crashes and misses a thousand events recovers by listing the current state and reconciling once.
Real controllers add machinery around this: an informer keeps a local cache from a watch, and a work queue holds object keys, not events. Five changes to one object collapse into one queue entry, and failed items are retried with backoff.
The life of an API request
A write to the API server passes through a fixed pipeline:
TLS and authentication who are you?
API Priority and Fairness do you get a seat now?
Authorization may you do this? (RBAC, Node, webhook)
Mutating admission change the object
Schema validation is it well-formed?
Validating admission is it allowed?
Write to etcd
Notify watchers
Authentication and authorisation
Users commonly present client certificates or OIDC tokens. Pods use short-lived, audience-bound service account tokens. Authorisation is usually RBAC, plus the Node authorizer, which limits a kubelet to objects related to its own pods.
Admission
Admission runs on create, update and delete requests. It cannot block reads, because get, list and watch skip that layer.
Mutation comes first. Mutating webhooks run one after another, since each can change the object. Validating webhooks then run in parallel, against the final object.
Admission has built-in plugins, external webhooks, and CEL policies evaluated inside the API server. ValidatingAdmissionPolicy became GA in 1.30, and MutatingAdmissionPolicy in 1.36. The project recommends CEL-based admission where it fits, since it avoids a network call for every request.
API Priority and Fairness
One badly behaved client can overload an API server, and a retry storm makes that worse. Without protection, requests that keep the cluster alive, such as node heartbeats, would compete with a script listing every pod.
APF classifies each request with a FlowSchema, which maps it to a PriorityLevelConfiguration. Each level has a share of the server’s concurrency, measured in seats. Within a level, requests are split into flows, usually per user or namespace, and shuffle sharding spreads the flows across queues. A noisy client fills its own queues and mostly hurts itself.
A client that is rejected receives HTTP 429. You can inspect the configuration with:
kubectl get flowschemas
kubectl get prioritylevelconfigurations
Optimistic concurrency
Every object carries a resourceVersion. An update includes the version it read. If the object has changed since, the API server returns 409 Conflict, and the client reads again and retries.
There are no long-held locks on objects. It also means two controllers that fight over a field will not corrupt anything, though they can loop forever.
etcd
etcd is a key-value store replicated with Raft, a consensus protocol. A write commits when a majority of members have it on disk.
| Members | Majority | Failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 7 | 4 | 3 |
Four members tolerate no more failures than three, so clusters use odd numbers. More members improve availability, but slow writes, since more disks must acknowledge. Three or five is typical.
Objects are stored under paths like /registry/pods/default/web-1, usually as protobuf rather than JSON.
Three properties shape Kubernetes:
- Every change has a revision, one global number that only rises. This is the source of
resourceVersion. - It supports watches from a given revision, so nothing is missed.
- It keeps old versions until compaction removes them. Fragmented space is reclaimed separately by defragmentation.
etcd’s database has a size quota, 2 GB by default, and its documentation suggests up to 8 GB. If the quota fills, etcd raises an alarm and refuses writes, so the cluster can no longer change. Its other sensitivity is disk: every commit is an fsync, so slow storage directly causes API latency and leader elections.
The most important backup in a self-managed cluster is an etcd snapshot, since it contains the state of everything.
List and watch
If every component polled the database, the load would be enormous. Instead, a component lists once, then opens a long-lived watch:
GET /api/v1/pods?resourceVersion=0 the initial list
GET /api/v1/pods?watch=1&resourceVersion=48213 a stream of changes
{"type":"ADDED", "object":{...}}
{"type":"MODIFIED", "object":{...}}
{"type":"DELETED", "object":{...}}
The API server keeps a watch cache in memory. Most watchers are served from it, so thousands of clients do not each become an etcd watch. That is why the API server scales out far more easily than etcd.
If a client reconnects from a version that has been compacted away, it gets 410 Gone, and must list again. Bookmark events periodically advance a client’s position, reducing how often this happens.
You can watch this stream yourself:
kubectl get pods -v=8 # shows the HTTP calls
kubectl get pods --watch --output-watch-events
kubectl get --raw "/api/v1/namespaces/default/pods?watch=1"
The scheduler
The scheduler watches for pods with no spec.nodeName. For each one, it decides on a node, then writes a Binding. It does not start anything.
Scheduling runs as a pipeline of plugins, with two important phases:
- Filter removes nodes that cannot run the pod: too few resources, a taint the pod does not tolerate, a failed affinity rule, or a missing volume.
- Score ranks the survivors.
This model shows the two phases with two filters and the LeastAllocated scoring idea. The real scheduler combines many weighted plugins.
nodes = {
"node-a": {"cpu": 4000, "mem": 8192, "used_cpu": 3500, "used_mem": 2048, "taints": set()},
"node-b": {"cpu": 4000, "mem": 8192, "used_cpu": 1000, "used_mem": 2048, "taints": set()},
"node-c": {"cpu": 8000, "mem": 16384, "used_cpu": 2000, "used_mem": 4096, "taints": {"gpu"}},
"node-d": {"cpu": 4000, "mem": 8192, "used_cpu": 2000, "used_mem": 6144, "taints": set()},
}
pod = {"cpu": 1000, "mem": 1024, "tolerates": set()}
def feasible(n):
if n["taints"] - pod["tolerates"]:
return "untolerated taint"
if n["cpu"] - n["used_cpu"] < pod["cpu"]:
return "insufficient cpu"
if n["mem"] - n["used_mem"] < pod["mem"]:
return "insufficient memory"
def score(n): # LeastAllocated: prefer the emptier node
cpu = (n["cpu"] - n["used_cpu"] - pod["cpu"]) / n["cpu"]
mem = (n["mem"] - n["used_mem"] - pod["mem"]) / n["mem"]
return round(100 * (cpu + mem) / 2)
for name, n in nodes.items():
why = feasible(n)
print(f"{name}: {'filtered, ' + why if why else f'score {score(n)}'}")
print("bind to", max((n for n in nodes if not feasible(nodes[n])), key=lambda n: score(nodes[n])))
Output:
node-a: filtered, insufficient cpu
node-b: score 56
node-c: filtered, untolerated taint
node-d: score 19
bind to node-b
The scheduler uses a pod’s requests, not its actual usage. A node can be idle and still be full, because requests have been promised. Limits are enforced later, on the node, by cgroups.
If no node fits, preemption may evict lower-priority pods to make room. A pod stuck in Pending usually has its reason in its events, as the filter results:
0/4 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {gpu: true}...
The default scheduler places pods one at a time. That breaks down for distributed AI jobs, which need every pod placed or none, and is why tools such as Kueue exist.
Controllers
kube-controller-manager is one binary containing many loops. Each owns a narrow job.
| Controller | Job |
|---|---|
| Deployment | Manages ReplicaSets, and rolling updates |
| ReplicaSet | Keeps the pod count correct |
| StatefulSet, DaemonSet, Job | Their own pod patterns |
| Node | Tracks node health, and taints failed nodes |
| EndpointSlice | Lists the ready pods behind each Service |
| ServiceAccount, Namespace | Lifecycle work |
| Garbage collector | Deletes objects whose owner is gone |
Controllers compose by writing objects for each other. A Deployment writes a ReplicaSet, which writes pods.
Two metadata fields hold this together:
ownerReferenceslinks a child to its parent, so deleting a parent can cascade.finalizersblock deletion until a controller has finished cleanup. An object stuck inTerminatingusually has a finalizer whose controller is gone.
Custom resources extend this. A CRD adds a new object type to the API server, and an operator is a controller watching it. Gatekeeper, below, is built this way.
High availability
| Component | How it runs |
|---|---|
| API server | Active-active behind a load balancer |
| etcd | Raft, needs a majority |
| Scheduler, controller manager | Active-passive, by leader election |
Leader election uses a Lease object. The holder keeps renewing it, and a standby takes over when renewals stop. Only one scheduler acts at a time, so two cannot make conflicting decisions.
kubectl -n kube-system get lease
If the entire control plane dies, running pods keep running, and Services keep routing, since nodes hold their last configuration. What you lose is change: no scheduling, no scaling, no self-healing. A node failure during an outage will not be repaired until the control plane returns.
The kubelet and the node
The kubelet watches for pods bound to its node, then makes them exist, through two standard interfaces:
- CRI, the Container Runtime Interface, to containerd or CRI-O.
- CNI, the Container Network Interface, used by the runtime to set up pod networking.
It also runs probes, enforces resource limits through cgroups, reports status, and renews a node Lease as a heartbeat. If heartbeats stop, the node controller marks the node unhealthy and taints it. Pods have a default toleration of 300 seconds for this, which explains why they take about five minutes to be replaced after a sudden node failure.
The kubelet can also run static pods from files on disk with no API server at all. kubeadm uses this to start the control plane: the API server is a static pod in /etc/kubernetes/manifests/, which solves the bootstrapping problem.
From kubectl apply to a running pod
kubectlsends the Deployment to the API server, where it passes the pipeline and lands in etcd.- The Deployment controller sees it, and creates a ReplicaSet.
- The ReplicaSet controller sees that, and creates a pod with no node.
- The scheduler sees the pod, and binds it to a node.
- That node’s kubelet sees the binding.
- The kubelet asks the runtime to create the pod sandbox.
- The runtime calls the CNI plugin, which gives the pod an IP.
- Containers start, and probes run.
- The kubelet reports
Ready. - The EndpointSlice controller adds the pod’s IP.
- kube-proxy or the CNI updates Service routing on every node.
No component called another. Each saw an object change.
The Kubernetes network model
Kubernetes sets the rules, and leaves the implementation to plugins:
- Every pod gets its own IP address.
- Pods can reach each other across nodes without NAT.
- Agents on a node can reach all pods on that node.
There are four separate problems here, often solved by different components:
| Problem | Usually solved by |
|---|---|
| Pod-to-pod connectivity | CNI plugin |
| Service virtual IPs | kube-proxy, or the CNI |
| Name resolution | CoreDNS |
| Traffic from outside | Ingress or Gateway controller, LoadBalancer |
What a CNI plugin does
A CNI plugin is an executable. The runtime reads a config from /etc/cni/net.d/, runs the binary from /opt/cni/bin/, and passes it a command such as ADD or DEL, with the pod’s network namespace.
A typical ADD:
- Creates a
vethpair: a virtual cable with two ends. - Puts one end in the pod’s network namespace as
eth0. - Assigns an IP, through an IPAM step.
- Adds routes, so traffic leaves through the host.
The remaining question is how a packet gets from a pod on one node to a pod on another.
Services and kube-proxy
A Service gets a stable virtual IP that exists on no interface. kube-proxy watches Services and EndpointSlices, then programs each node to rewrite traffic to a real pod.
| Mode | Status |
|---|---|
iptables | The long-standing default, with rule evaluation that grows with Service count |
nftables | The successor, with faster map lookups and quicker updates |
ipvs | Deprecated since v1.35 |
KEP-5343 plans to make nftables the default in v1.40.
Readiness connects here. A pod that fails its readiness probe leaves the EndpointSlice, and stops getting traffic, with no restart.
Flannel: the simple overlay
Flannel solves one problem, pod connectivity, and does little else. Each node gets a subnet from the cluster range, and a flanneld agent writes the details to /run/flannel/subnet.env.
The VXLAN backend
VXLAN wraps the pod’s whole Ethernet frame inside a UDP packet between node IPs, so the physical network only ever sees node addresses.
+----------+----------+-----+-------+---------+----------+---------+
| Outer | Outer IP | UDP | VXLAN | Inner | Inner IP | Payload |
| Ethernet | node A>B | | | Eth | pod>pod | |
| 14 | 20 | 8 | 8 | 14 | 20 | ... |
+----------+----------+-----+-------+---------+----------+---------+
The sending path on node A:
- The pod sends to
10.244.2.5, and the packet leaves through itsveth. - The host routing table sends
10.244.2.0/24to theflannel.1VXLAN device. - The kernel encapsulates the frame, and addresses it to node B.
- Node B’s kernel strips the wrapper, and routes the packet to the pod.
Flannel’s agent writes the routes, ARP entries and forwarding entries, so this all happens in the kernel’s ordinary VXLAN code.
Encapsulation has a cost you can calculate:
import struct
outer = 14 + 20 + 8 + 8 # Ethernet + IP + UDP + VXLAN
print("outer overhead", outer, "bytes")
print("pod MTU on a 1500 byte network:", 1500 - (20 + 8 + 8 + 14))
hdr = struct.pack(">B3xI", 0x08, 1 << 8)
print("vxlan header:", hdr.hex(" "))
for payload in (64, 512, 1400):
print(f"{payload} byte pod packet: {100 * outer / (payload + outer):.1f}% overhead")
outer overhead 50 bytes
pod MTU on a 1500 byte network: 1450
vxlan header: 08 00 00 00 00 00 01 00
64 byte pod packet: 43.9% overhead
512 byte pod packet: 8.9% overhead
1400 byte pod packet: 3.4% overhead
A pod MTU of 1450 is why overlay networks often show a classic failure: small requests work, and large transfers hang. If a pod’s MTU is still 1500, full-size packets no longer fit after encapsulation. If the “fragmentation needed” ICMP messages are filtered, the sender never learns this.
The host-gw backend
If all nodes share a layer-2 network, no wrapper is needed. Flannel adds plain routes, with each node as the next hop for the others’ pod subnets:
ip route
# 10.244.2.0/24 via 192.168.1.12 dev eth0
This avoids the encapsulation overhead, but it does not work when nodes are separated by a router that does not know those pod routes.
What Flannel leaves out
Flannel does not enforce NetworkPolicy. If you create one on a Flannel-only cluster, the API server accepts it, and nothing happens. Silent non-enforcement is a real risk, and the reason to pair Flannel with an enforcer if you need policy.
Cilium: networking in eBPF
Cilium takes a different approach: it loads eBPF programs into the kernel, and attaches them to network hooks. Forwarding, load balancing and policy decisions run there, using eBPF maps as lookup tables.
Pod -> veth -> eBPF at tc hook -> policy check, service lookup, conntrack -> out
^
eBPF maps (identities, services, endpoints, policies)
^
cilium-agent (watches the API server, writes the maps)
Replacing kube-proxy
Cilium can provide ClusterIP, NodePort and LoadBalancer Services itself, replacing kube-proxy. A hash map lookup does not grow with Service count the way sequential rules do.
For traffic from a pod, it can act at connect() time in the socket layer, and choose the backend before any packet exists. The packet then goes straight to the real pod IP, with no per-packet translation for that path.
Identity instead of IP
Traditional firewalls filter by IP. In Kubernetes, pod IPs change constantly, so rules would need updating everywhere on each pod start.
Cilium derives a numeric security identity from a pod’s security-relevant labels. All pods with the same labels share an identity, across nodes. Policy is checked by identity, so a new pod with existing labels needs no policy change.
Layer 7 policy
Standard NetworkPolicy works at layers 3 and 4. A CiliumNetworkPolicy can go up to HTTP. L7 rules route the traffic through an Envoy proxy managed by Cilium.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: api-read-only
spec:
endpointSelector:
matchLabels:
app: api
ingress:
- fromEndpoints:
- matchLabels:
app: frontend
toPorts:
- ports:
- port: "8080"
protocol: TCP
rules:
http:
- method: GET
path: "/products.*"
This lets frontend call GET /products, while a POST is refused.
Hubble
Since eBPF programs see every flow, Hubble can report them with their identities and policy verdicts:
hubble observe --namespace default --verdict DROPPED
That answers the most common network question in Kubernetes, why a connection fails, with an explicit reason rather than a timeout.
Comparison
| Flannel | Cilium | |
|---|---|---|
| Datapath | Kernel VXLAN or routes | eBPF |
| Pod connectivity | Yes | Yes, overlay or native routing |
| NetworkPolicy | No | Yes |
| L7 policy | No | Yes |
| Replaces kube-proxy | No | Optional |
| Flow visibility | No | Hubble |
| Operational weight | Very light | Heavier, and needs a suitable kernel |
Flannel suits small clusters, labs and simple needs. Cilium suits clusters that need policy, visibility, or large Service counts. Calico is the other common choice, with BGP routing and strong policy support.
Ingress and Gateway API
The Ingress object is limited, and much of its behaviour lives in vendor annotations. The widely used Ingress NGINX project reached the end of maintenance in March 2026, with no further releases or security fixes planned after that. The project recommends migrating to Gateway API or another controller.
Gateway API separates roles: a GatewayClass for the implementation, a Gateway for the listener, and HTTPRoute objects for routing rules, which application teams can own.
OPA Gatekeeper
RBAC says who may create a pod. It cannot say that pods must have an owner label, must not run as root, or may only use images from your registry. Those are rules about an object’s content, and that is what admission policy is for.
OPA, Open Policy Agent, is a general policy engine with a language called Rego. Gatekeeper is a Kubernetes project that runs OPA as a validating webhook and manages policies as Kubernetes objects.
kubectl apply
|
API server --- AdmissionReview ---> Gatekeeper webhook
^ |
| evaluates Rego
+------- allowed, or denied <---------+
Template and Constraint
Gatekeeper separates the logic from its use.
A ConstraintTemplate holds the Rego, and defines a new custom resource kind. Gatekeeper creates that CRD for you.
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("missing required labels: %v", [missing])
}
Reading the Rego: violation is a set of results. Each line in the body must be true for a violation to exist. It builds the set of provided labels, the set of required ones, and takes the difference. If the difference is empty, the rule produces nothing, and the object is allowed.
A Constraint applies the template, with parameters and a scope.
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: namespaces-need-owner
spec:
enforcementAction: dryrun
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
excludedNamespaces: ["kube-system", "gatekeeper-system"]
parameters:
labels: ["owner"]
A platform team can write one template, and apply it many times with different parameters.
Enforcement actions
| Action | Effect |
|---|---|
dryrun | Records violations, blocks nothing |
warn | Allows the request, and returns a warning to the client |
deny | Rejects the request |
Audit
Admission only sees objects as they are created or changed. A resource that existed before a policy would never be checked. Gatekeeper’s audit controller periodically evaluates existing resources, and writes violations to each Constraint’s status:
kubectl get k8srequiredlabels namespaces-need-owner -o yaml
So a safe rollout is: deploy with dryrun, read the audit results, fix the violators, then move to warn, and finally deny.
Mutation and data
Gatekeeper can also change objects at admission, using resources such as Assign and AssignMetadata, for example to add a default securityContext.
Policies that compare an object with others, such as “ingress hostnames must be unique”, need cluster data. A Config resource tells Gatekeeper which kinds to sync into OPA’s cache, where Rego reads them from data.inventory.
The webhook risk
An admission webhook is in the path of every matching write. If it is slow or down, what happens depends on its failurePolicy:
failurePolicy | When the webhook is unreachable |
|---|---|
Fail | Writes are rejected, which can block repairs to the cluster |
Ignore | Writes are allowed, so policy is silently skipped |
This is a real trade-off between safety and availability. Run several replicas, exclude system namespaces, set a short timeout, and check which setting your install uses.
Gatekeeper, CEL or Kyverno
| Gatekeeper | ValidatingAdmissionPolicy | Kyverno | |
|---|---|---|---|
| Language | Rego, and CEL in recent versions | CEL | YAML policies, and CEL |
| Runs in | A webhook | The API server | A webhook |
| Audit of existing objects | Yes | Limited | Yes |
| Mutation | Yes | Through MutatingAdmissionPolicy | Yes |
| Cluster data in policy | Yes | Limited | Yes |
For simple field checks, built-in CEL policy has less to operate. For audit, complex logic, or shared policy across systems, Gatekeeper remains a strong option.
Hands on with a local cluster
kind runs a Kubernetes cluster in Docker containers. A cluster with a real control plane and no default CNI is a good base for Cilium.
# kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
networking:
disableDefaultCNI: true
kubeProxyMode: none
kind create cluster --name lab --config kind.yaml
kubectl get nodes # NotReady: there is no CNI yet
Nodes stay NotReady until a network plugin is installed, which shows that pod networking is not built in.
Look at the control plane, which runs as static pods:
kubectl -n kube-system get pods
docker exec lab-control-plane ls /etc/kubernetes/manifests
kubectl -n kube-system get lease
Read etcd directly:
kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/server.crt \
--key /etc/kubernetes/pki/etcd/server.key \
get /registry/namespaces --prefix --keys-only
Install Cilium, with its service handling in place of kube-proxy:
helm repo add cilium https://helm.cilium.io
helm install cilium cilium/cilium --namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost=lab-control-plane \
--set k8sServicePort=6443 \
--set hubble.relay.enabled=true
kubectl get nodes # now Ready
kubectl -n kube-system exec ds/cilium -- cilium-dbg status
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
kubectl -n kube-system exec ds/cilium -- cilium-dbg identity list
Install Gatekeeper and test the policy:
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
--namespace gatekeeper-system --create-namespace
kubectl apply -f template.yaml
kubectl apply -f constraint.yaml
Change enforcementAction to deny, apply it again, then:
kubectl create namespace test
# denied: missing required labels: {"owner"}
To remove the lab:
kind delete cluster --name lab
When something breaks
| Symptom | First place to look |
|---|---|
Pod Pending | kubectl describe pod, for scheduler events |
Pod ContainerCreating | Events, often the CNI or a volume |
Node NotReady | The kubelet, and whether a CNI is installed |
| Writes rejected with a webhook error | The webhook’s pods and failurePolicy |
database space exceeded | The etcd quota, compaction and defragmentation |
| API requests return 429 | API Priority and Fairness, and the noisy client |
| Service has no endpoints | Labels, selectors and readiness |
| Large transfers hang, small ones work | MTU |
Object stuck in Terminating | Its finalizers |
What to take away
- All state is in etcd, and only the API server touches it.
- Components coordinate by watching objects, never by calling each other.
- Controllers act on current state, not on events, which is why they recover so well.
- The control plane manages change, and workloads survive without it for a while.
- A CNI plugin connects pods, and something else may handle Services, DNS and policy.
- Flannel keeps networking simple, and Cilium moves it into programmable eBPF.
- Gatekeeper turns content rules into Kubernetes objects, with an audit of what already exists.