Ashok G

Draft

The Kubernetes control plane, Cilium, Flannel and OPA Gatekeeper

Kubernetes is often described as a container orchestrator. A more useful description is a database with an API, surrounded by programs that each watch for changes and try to make the world match what the database says.

Nothing in Kubernetes runs a central script of “create the pod, then attach the network, then start the container”. Separate components each do one job when they see work to do. That one design explains most of its behaviour, including how it recovers from failure and why debugging it can be confusing.

This article goes through the control plane, then what happens on the network, then how policy is enforced at the door. Details reflect Kubernetes v1.36.

The parts

                        CONTROL PLANE
  +-----------------------------------------------------+
  |  kube-scheduler     kube-controller-manager         |
  |        |                     |                      |
  |        +-------+-------------+                      |
  |                v                                    |
  |         kube-apiserver  <------->  etcd             |
  +----------------^------------------------------------+
                   |  HTTPS: everything talks to the API server
        +----------+-----------+
        |                      |
  +-----+--------+       +-----+--------+
  | kubelet      |       | kubelet      |
  | kube-proxy   |       | kube-proxy   |
  | runtime, CNI |       | runtime, CNI |
  +--------------+       +--------------+
       NODE                   NODE
ComponentJob
kube-apiserverThe front door. Authenticates, authorises, validates and stores objects, and serves watches
etcdThe consistent key-value store that holds all cluster state
kube-schedulerChooses a node for each unscheduled pod
kube-controller-managerRuns the built-in controllers
cloud-controller-managerTalks to the cloud provider, on cloud clusters
kubeletRuns on each node, and makes that node’s pods real
kube-proxyPrograms Service routing on each node, unless a CNI replaces it

One rule matters most: only the API server talks to etcd. The scheduler, controllers and kubelets never touch the database, and they do not talk to each other. All coordination happens through objects in the API.

Declarative state and the reconcile loop

Most objects have two halves. spec is what you want, and status is what currently exists. A controller is a loop that closes the gap:

forever:
    desired = read spec
    actual  = observe the world
    make one change that moves actual towards desired
    write status

The next program is a tiny model. A Store plays the API server: every write increases a resourceVersion and adds to an event log. One function reconciles a ReplicaSet.

class Store:
    """A tiny API server: every write bumps resourceVersion and emits an event."""
    def __init__(self):
        self.objects, self.events, self.rv = {}, [], 0
    def write(self, kind, name, obj):
        self.rv += 1
        event = "MODIFIED" if (kind, name) in self.objects else "ADDED"
        self.objects[(kind, name)] = obj
        self.events.append((self.rv, event, kind, name))
    def delete(self, kind, name):
        self.rv += 1
        del self.objects[(kind, name)]
        self.events.append((self.rv, "DELETED", kind, name))
    def watch(self, since):
        return [e for e in self.events if e[0] > since]

def pods_of(store, rs):
    return [n for (k, n), o in store.objects.items() if k == "Pod" and o["owner"] == rs]

serial = 0
def reconcile(store, rs):
    global serial
    want = store.objects[("ReplicaSet", rs)]["replicas"]
    have = pods_of(store, rs)
    for _ in range(want - len(have)):
        serial += 1
        store.write("Pod", f"{rs}-{serial}", {"owner": rs})
    for name in have[want:]:
        store.delete("Pod", name)
    return want - len(have)

store = Store()
store.write("ReplicaSet", "web", {"replicas": 3})
seen, rounds = 0, 0

def drain():
    global seen, rounds
    while events := store.watch(seen):
        seen = events[-1][0]
        rounds += 1
        print(f"  round {rounds}: {len(events)} event(s), diff {reconcile(store, 'web'):+d}")

print("create replicas=3"); drain()
print("kill one pod"); store.delete("Pod", pods_of(store, "web")[0]); drain()
print("scale to 5, then to 2, before the controller wakes up")
store.write("ReplicaSet", "web", {"replicas": 5})
store.write("ReplicaSet", "web", {"replicas": 2}); drain()
print("final pods:", sorted(pods_of(store, "web")), "rv", store.rv)

Output:

create replicas=3
  round 1: 1 event(s), diff +3
  round 2: 3 event(s), diff +0
kill one pod
  round 3: 1 event(s), diff +1
  round 4: 1 event(s), diff +0
scale to 5, then to 2, before the controller wakes up
  round 5: 2 event(s), diff -1
  round 6: 1 event(s), diff +0
final pods: ['web-2', 'web-3'] rv 9

Round 2 shows a controller reacting to its own writes, seeing nothing to do, and stopping. That is normal.

Round 5 is the important one. Two scale changes arrived while the controller was busy. It never acted on replicas: 5. It read the current spec, saw 3 pods and wanted 2, then deleted one. It did not need to process the history correctly, or even see it.

This is level-triggered behaviour: act on the current state, not on each event. Events only say “something changed, go and look”. So a controller that crashes and misses a thousand events recovers by listing the current state and reconciling once.

Real controllers add machinery around this: an informer keeps a local cache from a watch, and a work queue holds object keys, not events. Five changes to one object collapse into one queue entry, and failed items are retried with backoff.

The life of an API request

A write to the API server passes through a fixed pipeline:

TLS and authentication      who are you?
API Priority and Fairness   do you get a seat now?
Authorization               may you do this? (RBAC, Node, webhook)
Mutating admission          change the object
Schema validation           is it well-formed?
Validating admission        is it allowed?
Write to etcd
Notify watchers

Authentication and authorisation

Users commonly present client certificates or OIDC tokens. Pods use short-lived, audience-bound service account tokens. Authorisation is usually RBAC, plus the Node authorizer, which limits a kubelet to objects related to its own pods.

Admission

Admission runs on create, update and delete requests. It cannot block reads, because get, list and watch skip that layer.

Mutation comes first. Mutating webhooks run one after another, since each can change the object. Validating webhooks then run in parallel, against the final object.

Admission has built-in plugins, external webhooks, and CEL policies evaluated inside the API server. ValidatingAdmissionPolicy became GA in 1.30, and MutatingAdmissionPolicy in 1.36. The project recommends CEL-based admission where it fits, since it avoids a network call for every request.

API Priority and Fairness

One badly behaved client can overload an API server, and a retry storm makes that worse. Without protection, requests that keep the cluster alive, such as node heartbeats, would compete with a script listing every pod.

APF classifies each request with a FlowSchema, which maps it to a PriorityLevelConfiguration. Each level has a share of the server’s concurrency, measured in seats. Within a level, requests are split into flows, usually per user or namespace, and shuffle sharding spreads the flows across queues. A noisy client fills its own queues and mostly hurts itself.

A client that is rejected receives HTTP 429. You can inspect the configuration with:

kubectl get flowschemas
kubectl get prioritylevelconfigurations

Optimistic concurrency

Every object carries a resourceVersion. An update includes the version it read. If the object has changed since, the API server returns 409 Conflict, and the client reads again and retries.

There are no long-held locks on objects. It also means two controllers that fight over a field will not corrupt anything, though they can loop forever.

etcd

etcd is a key-value store replicated with Raft, a consensus protocol. A write commits when a majority of members have it on disk.

MembersMajorityFailures tolerated
110
321
532
743

Four members tolerate no more failures than three, so clusters use odd numbers. More members improve availability, but slow writes, since more disks must acknowledge. Three or five is typical.

Objects are stored under paths like /registry/pods/default/web-1, usually as protobuf rather than JSON.

Three properties shape Kubernetes:

  • Every change has a revision, one global number that only rises. This is the source of resourceVersion.
  • It supports watches from a given revision, so nothing is missed.
  • It keeps old versions until compaction removes them. Fragmented space is reclaimed separately by defragmentation.

etcd’s database has a size quota, 2 GB by default, and its documentation suggests up to 8 GB. If the quota fills, etcd raises an alarm and refuses writes, so the cluster can no longer change. Its other sensitivity is disk: every commit is an fsync, so slow storage directly causes API latency and leader elections.

The most important backup in a self-managed cluster is an etcd snapshot, since it contains the state of everything.

List and watch

If every component polled the database, the load would be enormous. Instead, a component lists once, then opens a long-lived watch:

GET /api/v1/pods?resourceVersion=0                  the initial list
GET /api/v1/pods?watch=1&resourceVersion=48213      a stream of changes
{"type":"ADDED",    "object":{...}}
{"type":"MODIFIED", "object":{...}}
{"type":"DELETED",  "object":{...}}

The API server keeps a watch cache in memory. Most watchers are served from it, so thousands of clients do not each become an etcd watch. That is why the API server scales out far more easily than etcd.

If a client reconnects from a version that has been compacted away, it gets 410 Gone, and must list again. Bookmark events periodically advance a client’s position, reducing how often this happens.

You can watch this stream yourself:

kubectl get pods -v=8                                # shows the HTTP calls
kubectl get pods --watch --output-watch-events
kubectl get --raw "/api/v1/namespaces/default/pods?watch=1"

The scheduler

The scheduler watches for pods with no spec.nodeName. For each one, it decides on a node, then writes a Binding. It does not start anything.

Scheduling runs as a pipeline of plugins, with two important phases:

  • Filter removes nodes that cannot run the pod: too few resources, a taint the pod does not tolerate, a failed affinity rule, or a missing volume.
  • Score ranks the survivors.

This model shows the two phases with two filters and the LeastAllocated scoring idea. The real scheduler combines many weighted plugins.

nodes = {
    "node-a": {"cpu": 4000, "mem": 8192, "used_cpu": 3500, "used_mem": 2048, "taints": set()},
    "node-b": {"cpu": 4000, "mem": 8192, "used_cpu": 1000, "used_mem": 2048, "taints": set()},
    "node-c": {"cpu": 8000, "mem": 16384, "used_cpu": 2000, "used_mem": 4096, "taints": {"gpu"}},
    "node-d": {"cpu": 4000, "mem": 8192, "used_cpu": 2000, "used_mem": 6144, "taints": set()},
}
pod = {"cpu": 1000, "mem": 1024, "tolerates": set()}

def feasible(n):
    if n["taints"] - pod["tolerates"]:
        return "untolerated taint"
    if n["cpu"] - n["used_cpu"] < pod["cpu"]:
        return "insufficient cpu"
    if n["mem"] - n["used_mem"] < pod["mem"]:
        return "insufficient memory"

def score(n):                        # LeastAllocated: prefer the emptier node
    cpu = (n["cpu"] - n["used_cpu"] - pod["cpu"]) / n["cpu"]
    mem = (n["mem"] - n["used_mem"] - pod["mem"]) / n["mem"]
    return round(100 * (cpu + mem) / 2)

for name, n in nodes.items():
    why = feasible(n)
    print(f"{name}: {'filtered, ' + why if why else f'score {score(n)}'}")
print("bind to", max((n for n in nodes if not feasible(nodes[n])), key=lambda n: score(nodes[n])))

Output:

node-a: filtered, insufficient cpu
node-b: score 56
node-c: filtered, untolerated taint
node-d: score 19
bind to node-b

The scheduler uses a pod’s requests, not its actual usage. A node can be idle and still be full, because requests have been promised. Limits are enforced later, on the node, by cgroups.

If no node fits, preemption may evict lower-priority pods to make room. A pod stuck in Pending usually has its reason in its events, as the filter results:

0/4 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {gpu: true}...

The default scheduler places pods one at a time. That breaks down for distributed AI jobs, which need every pod placed or none, and is why tools such as Kueue exist.

Controllers

kube-controller-manager is one binary containing many loops. Each owns a narrow job.

ControllerJob
DeploymentManages ReplicaSets, and rolling updates
ReplicaSetKeeps the pod count correct
StatefulSet, DaemonSet, JobTheir own pod patterns
NodeTracks node health, and taints failed nodes
EndpointSliceLists the ready pods behind each Service
ServiceAccount, NamespaceLifecycle work
Garbage collectorDeletes objects whose owner is gone

Controllers compose by writing objects for each other. A Deployment writes a ReplicaSet, which writes pods.

Two metadata fields hold this together:

  • ownerReferences links a child to its parent, so deleting a parent can cascade.
  • finalizers block deletion until a controller has finished cleanup. An object stuck in Terminating usually has a finalizer whose controller is gone.

Custom resources extend this. A CRD adds a new object type to the API server, and an operator is a controller watching it. Gatekeeper, below, is built this way.

High availability

ComponentHow it runs
API serverActive-active behind a load balancer
etcdRaft, needs a majority
Scheduler, controller managerActive-passive, by leader election

Leader election uses a Lease object. The holder keeps renewing it, and a standby takes over when renewals stop. Only one scheduler acts at a time, so two cannot make conflicting decisions.

kubectl -n kube-system get lease

If the entire control plane dies, running pods keep running, and Services keep routing, since nodes hold their last configuration. What you lose is change: no scheduling, no scaling, no self-healing. A node failure during an outage will not be repaired until the control plane returns.

The kubelet and the node

The kubelet watches for pods bound to its node, then makes them exist, through two standard interfaces:

  • CRI, the Container Runtime Interface, to containerd or CRI-O.
  • CNI, the Container Network Interface, used by the runtime to set up pod networking.

It also runs probes, enforces resource limits through cgroups, reports status, and renews a node Lease as a heartbeat. If heartbeats stop, the node controller marks the node unhealthy and taints it. Pods have a default toleration of 300 seconds for this, which explains why they take about five minutes to be replaced after a sudden node failure.

The kubelet can also run static pods from files on disk with no API server at all. kubeadm uses this to start the control plane: the API server is a static pod in /etc/kubernetes/manifests/, which solves the bootstrapping problem.

From kubectl apply to a running pod

  1. kubectl sends the Deployment to the API server, where it passes the pipeline and lands in etcd.
  2. The Deployment controller sees it, and creates a ReplicaSet.
  3. The ReplicaSet controller sees that, and creates a pod with no node.
  4. The scheduler sees the pod, and binds it to a node.
  5. That node’s kubelet sees the binding.
  6. The kubelet asks the runtime to create the pod sandbox.
  7. The runtime calls the CNI plugin, which gives the pod an IP.
  8. Containers start, and probes run.
  9. The kubelet reports Ready.
  10. The EndpointSlice controller adds the pod’s IP.
  11. kube-proxy or the CNI updates Service routing on every node.

No component called another. Each saw an object change.

The Kubernetes network model

Kubernetes sets the rules, and leaves the implementation to plugins:

  • Every pod gets its own IP address.
  • Pods can reach each other across nodes without NAT.
  • Agents on a node can reach all pods on that node.

There are four separate problems here, often solved by different components:

ProblemUsually solved by
Pod-to-pod connectivityCNI plugin
Service virtual IPskube-proxy, or the CNI
Name resolutionCoreDNS
Traffic from outsideIngress or Gateway controller, LoadBalancer

What a CNI plugin does

A CNI plugin is an executable. The runtime reads a config from /etc/cni/net.d/, runs the binary from /opt/cni/bin/, and passes it a command such as ADD or DEL, with the pod’s network namespace.

A typical ADD:

  1. Creates a veth pair: a virtual cable with two ends.
  2. Puts one end in the pod’s network namespace as eth0.
  3. Assigns an IP, through an IPAM step.
  4. Adds routes, so traffic leaves through the host.

The remaining question is how a packet gets from a pod on one node to a pod on another.

Services and kube-proxy

A Service gets a stable virtual IP that exists on no interface. kube-proxy watches Services and EndpointSlices, then programs each node to rewrite traffic to a real pod.

ModeStatus
iptablesThe long-standing default, with rule evaluation that grows with Service count
nftablesThe successor, with faster map lookups and quicker updates
ipvsDeprecated since v1.35

KEP-5343 plans to make nftables the default in v1.40.

Readiness connects here. A pod that fails its readiness probe leaves the EndpointSlice, and stops getting traffic, with no restart.

Flannel: the simple overlay

Flannel solves one problem, pod connectivity, and does little else. Each node gets a subnet from the cluster range, and a flanneld agent writes the details to /run/flannel/subnet.env.

The VXLAN backend

VXLAN wraps the pod’s whole Ethernet frame inside a UDP packet between node IPs, so the physical network only ever sees node addresses.

+----------+----------+-----+-------+---------+----------+---------+
| Outer    | Outer IP | UDP | VXLAN | Inner   | Inner IP | Payload |
| Ethernet | node A>B |     |       | Eth     | pod>pod  |         |
| 14       | 20       | 8   | 8     | 14      | 20       | ...     |
+----------+----------+-----+-------+---------+----------+---------+

The sending path on node A:

  1. The pod sends to 10.244.2.5, and the packet leaves through its veth.
  2. The host routing table sends 10.244.2.0/24 to the flannel.1 VXLAN device.
  3. The kernel encapsulates the frame, and addresses it to node B.
  4. Node B’s kernel strips the wrapper, and routes the packet to the pod.

Flannel’s agent writes the routes, ARP entries and forwarding entries, so this all happens in the kernel’s ordinary VXLAN code.

Encapsulation has a cost you can calculate:

import struct
outer = 14 + 20 + 8 + 8     # Ethernet + IP + UDP + VXLAN
print("outer overhead", outer, "bytes")
print("pod MTU on a 1500 byte network:", 1500 - (20 + 8 + 8 + 14))
hdr = struct.pack(">B3xI", 0x08, 1 << 8)
print("vxlan header:", hdr.hex(" "))
for payload in (64, 512, 1400):
    print(f"{payload} byte pod packet: {100 * outer / (payload + outer):.1f}% overhead")
outer overhead 50 bytes
pod MTU on a 1500 byte network: 1450
vxlan header: 08 00 00 00 00 00 01 00
64 byte pod packet: 43.9% overhead
512 byte pod packet: 8.9% overhead
1400 byte pod packet: 3.4% overhead

A pod MTU of 1450 is why overlay networks often show a classic failure: small requests work, and large transfers hang. If a pod’s MTU is still 1500, full-size packets no longer fit after encapsulation. If the “fragmentation needed” ICMP messages are filtered, the sender never learns this.

The host-gw backend

If all nodes share a layer-2 network, no wrapper is needed. Flannel adds plain routes, with each node as the next hop for the others’ pod subnets:

ip route
# 10.244.2.0/24 via 192.168.1.12 dev eth0

This avoids the encapsulation overhead, but it does not work when nodes are separated by a router that does not know those pod routes.

What Flannel leaves out

Flannel does not enforce NetworkPolicy. If you create one on a Flannel-only cluster, the API server accepts it, and nothing happens. Silent non-enforcement is a real risk, and the reason to pair Flannel with an enforcer if you need policy.

Cilium: networking in eBPF

Cilium takes a different approach: it loads eBPF programs into the kernel, and attaches them to network hooks. Forwarding, load balancing and policy decisions run there, using eBPF maps as lookup tables.

Pod -> veth -> eBPF at tc hook -> policy check, service lookup, conntrack -> out
                     ^
              eBPF maps (identities, services, endpoints, policies)
                     ^
               cilium-agent (watches the API server, writes the maps)

Replacing kube-proxy

Cilium can provide ClusterIP, NodePort and LoadBalancer Services itself, replacing kube-proxy. A hash map lookup does not grow with Service count the way sequential rules do.

For traffic from a pod, it can act at connect() time in the socket layer, and choose the backend before any packet exists. The packet then goes straight to the real pod IP, with no per-packet translation for that path.

Identity instead of IP

Traditional firewalls filter by IP. In Kubernetes, pod IPs change constantly, so rules would need updating everywhere on each pod start.

Cilium derives a numeric security identity from a pod’s security-relevant labels. All pods with the same labels share an identity, across nodes. Policy is checked by identity, so a new pod with existing labels needs no policy change.

Layer 7 policy

Standard NetworkPolicy works at layers 3 and 4. A CiliumNetworkPolicy can go up to HTTP. L7 rules route the traffic through an Envoy proxy managed by Cilium.

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-read-only
spec:
  endpointSelector:
    matchLabels:
      app: api
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: "/products.*"

This lets frontend call GET /products, while a POST is refused.

Hubble

Since eBPF programs see every flow, Hubble can report them with their identities and policy verdicts:

hubble observe --namespace default --verdict DROPPED

That answers the most common network question in Kubernetes, why a connection fails, with an explicit reason rather than a timeout.

Comparison

FlannelCilium
DatapathKernel VXLAN or routeseBPF
Pod connectivityYesYes, overlay or native routing
NetworkPolicyNoYes
L7 policyNoYes
Replaces kube-proxyNoOptional
Flow visibilityNoHubble
Operational weightVery lightHeavier, and needs a suitable kernel

Flannel suits small clusters, labs and simple needs. Cilium suits clusters that need policy, visibility, or large Service counts. Calico is the other common choice, with BGP routing and strong policy support.

Ingress and Gateway API

The Ingress object is limited, and much of its behaviour lives in vendor annotations. The widely used Ingress NGINX project reached the end of maintenance in March 2026, with no further releases or security fixes planned after that. The project recommends migrating to Gateway API or another controller.

Gateway API separates roles: a GatewayClass for the implementation, a Gateway for the listener, and HTTPRoute objects for routing rules, which application teams can own.

OPA Gatekeeper

RBAC says who may create a pod. It cannot say that pods must have an owner label, must not run as root, or may only use images from your registry. Those are rules about an object’s content, and that is what admission policy is for.

OPA, Open Policy Agent, is a general policy engine with a language called Rego. Gatekeeper is a Kubernetes project that runs OPA as a validating webhook and manages policies as Kubernetes objects.

kubectl apply
     |
API server --- AdmissionReview ---> Gatekeeper webhook
     ^                                     |
     |                                 evaluates Rego
     +------- allowed, or denied <---------+

Template and Constraint

Gatekeeper separates the logic from its use.

A ConstraintTemplate holds the Rego, and defines a new custom resource kind. Gatekeeper creates that CRD for you.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package k8srequiredlabels

      violation[{"msg": msg}] {
        provided := {label | input.review.object.metadata.labels[label]}
        required := {label | label := input.parameters.labels[_]}
        missing := required - provided
        count(missing) > 0
        msg := sprintf("missing required labels: %v", [missing])
      }

Reading the Rego: violation is a set of results. Each line in the body must be true for a violation to exist. It builds the set of provided labels, the set of required ones, and takes the difference. If the difference is empty, the rule produces nothing, and the object is allowed.

A Constraint applies the template, with parameters and a scope.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: namespaces-need-owner
spec:
  enforcementAction: dryrun
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Namespace"]
    excludedNamespaces: ["kube-system", "gatekeeper-system"]
  parameters:
    labels: ["owner"]

A platform team can write one template, and apply it many times with different parameters.

Enforcement actions

ActionEffect
dryrunRecords violations, blocks nothing
warnAllows the request, and returns a warning to the client
denyRejects the request

Audit

Admission only sees objects as they are created or changed. A resource that existed before a policy would never be checked. Gatekeeper’s audit controller periodically evaluates existing resources, and writes violations to each Constraint’s status:

kubectl get k8srequiredlabels namespaces-need-owner -o yaml

So a safe rollout is: deploy with dryrun, read the audit results, fix the violators, then move to warn, and finally deny.

Mutation and data

Gatekeeper can also change objects at admission, using resources such as Assign and AssignMetadata, for example to add a default securityContext.

Policies that compare an object with others, such as “ingress hostnames must be unique”, need cluster data. A Config resource tells Gatekeeper which kinds to sync into OPA’s cache, where Rego reads them from data.inventory.

The webhook risk

An admission webhook is in the path of every matching write. If it is slow or down, what happens depends on its failurePolicy:

failurePolicyWhen the webhook is unreachable
FailWrites are rejected, which can block repairs to the cluster
IgnoreWrites are allowed, so policy is silently skipped

This is a real trade-off between safety and availability. Run several replicas, exclude system namespaces, set a short timeout, and check which setting your install uses.

Gatekeeper, CEL or Kyverno

GatekeeperValidatingAdmissionPolicyKyverno
LanguageRego, and CEL in recent versionsCELYAML policies, and CEL
Runs inA webhookThe API serverA webhook
Audit of existing objectsYesLimitedYes
MutationYesThrough MutatingAdmissionPolicyYes
Cluster data in policyYesLimitedYes

For simple field checks, built-in CEL policy has less to operate. For audit, complex logic, or shared policy across systems, Gatekeeper remains a strong option.

Hands on with a local cluster

kind runs a Kubernetes cluster in Docker containers. A cluster with a real control plane and no default CNI is a good base for Cilium.

# kind.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
networking:
  disableDefaultCNI: true
  kubeProxyMode: none
kind create cluster --name lab --config kind.yaml
kubectl get nodes                # NotReady: there is no CNI yet

Nodes stay NotReady until a network plugin is installed, which shows that pod networking is not built in.

Look at the control plane, which runs as static pods:

kubectl -n kube-system get pods
docker exec lab-control-plane ls /etc/kubernetes/manifests
kubectl -n kube-system get lease

Read etcd directly:

kubectl -n kube-system exec etcd-lab-control-plane -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/namespaces --prefix --keys-only

Install Cilium, with its service handling in place of kube-proxy:

helm repo add cilium https://helm.cilium.io
helm install cilium cilium/cilium --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=lab-control-plane \
  --set k8sServicePort=6443 \
  --set hubble.relay.enabled=true

kubectl get nodes                # now Ready
kubectl -n kube-system exec ds/cilium -- cilium-dbg status
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list
kubectl -n kube-system exec ds/cilium -- cilium-dbg identity list

Install Gatekeeper and test the policy:

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
  --namespace gatekeeper-system --create-namespace

kubectl apply -f template.yaml
kubectl apply -f constraint.yaml

Change enforcementAction to deny, apply it again, then:

kubectl create namespace test
# denied: missing required labels: {"owner"}

To remove the lab:

kind delete cluster --name lab

When something breaks

SymptomFirst place to look
Pod Pendingkubectl describe pod, for scheduler events
Pod ContainerCreatingEvents, often the CNI or a volume
Node NotReadyThe kubelet, and whether a CNI is installed
Writes rejected with a webhook errorThe webhook’s pods and failurePolicy
database space exceededThe etcd quota, compaction and defragmentation
API requests return 429API Priority and Fairness, and the noisy client
Service has no endpointsLabels, selectors and readiness
Large transfers hang, small ones workMTU
Object stuck in TerminatingIts finalizers

What to take away

  • All state is in etcd, and only the API server touches it.
  • Components coordinate by watching objects, never by calling each other.
  • Controllers act on current state, not on events, which is why they recover so well.
  • The control plane manages change, and workloads survive without it for a while.
  • A CNI plugin connects pods, and something else may handle Services, DNS and policy.
  • Flannel keeps networking simple, and Cilium moves it into programmable eBPF.
  • Gatekeeper turns content rules into Kubernetes objects, with an audit of what already exists.