How Kafka works, from 0 to 1000

Apache Kafka is a distributed, durable log. That one idea explains nearly everything about it. This article starts from what a log is, and goes down to how replication keeps data safe, and how transactions work.

Level 0: the problem

A company with ten systems that share data has a wiring problem. Connecting each pair directly means up to 90 one-way links, each with its own format and failure modes.

Kafka puts one central pipeline in the middle. Systems publish events to it, and others read them. Producers don’t need to know who’s consuming, and consumers can arrive later. This is the same problem LinkedIn built Kafka to solve.

Level 100: the log

A log is an append-only sequence of records. New records go at the end, and existing ones never change.

offset:  0    1    2    3    4    5
       [a]  [b]  [c]  [d]  [e]  [f]  <- new records appended here

Each record gets an offset, its position in the log. Consumers track their own offset, so reading doesn’t delete anything. This differs from a traditional queue, where a message disappears once it’s consumed, and it’s what makes replay and multiple independent readers possible.

A record contains a key, a value, a timestamp and optional headers. Kafka treats the key and value as plain bytes.

Level 200: topics and partitions

A topic is a named stream, such as orders. One log on one machine wouldn’t scale, so a topic is split into partitions, each its own log.

Topic: orders
Partition 0:  [0][1][2][3][4]
Partition 1:  [0][1][2]
Partition 2:  [0][1][2][3]

Ordering holds within a partition only. There’s no global order across a topic.

A producer picks a partition with the record’s key. Records with the same key hash to the same partition, so all of one customer’s orders stay in order. Records with no key are spread around, in batches.

Partitions also set the limit on parallelism: a consumer group can’t have more active members than partitions, in the classic model.

Level 300: brokers and clusters

A broker is a Kafka server, and several form a cluster. Each partition lives on a few brokers, for safety. Clients connect to a few bootstrap brokers, fetch metadata that says which broker leads which partition, and then talk to those directly.

Level 400: producers

A producer doesn’t send each record alone. It collects them into batches per partition, which is key to Kafka’s throughput.

Waiting a few milliseconds gives larger batches, better compression and higher throughput, at the price of latency. Compression applies to the whole batch, so repeated field names compress very well.

Acknowledgements

Modern producers default to acks=all, with idempotence on.

Level 500: consumers

Consumers pull data. A consumer asks a broker for records from a given offset, and the request can wait until data arrives, which is called long polling. Pulling lets each consumer go at its own pace, and the broker doesn’t have to track delivery state for each one.

Consumer groups

A consumer group is a set of consumers sharing a topic. Each partition goes to one member at a time.

Partitions:  P0  P1  P2  P3
              \  /    \  /
Consumers:     C1      C2     (group: billing)

Different groups read independently, so billing and analytics can each read all the data.

Offsets

A group stores its progress in an internal topic called __consumer_offsets. When a record is committed relative to processing decides the delivery guarantee.

Most systems process first, and make processing idempotent.

Rebalancing

When members join or leave, partitions have to be redistributed. This is a rebalance. A broker acting as group coordinator manages it.

Old-style rebalances stopped the whole group, so a rolling restart of consumers could cause repeated pauses. Improvements arrived in steps:

• Cooperative rebalancing moves only the partitions that need to move.
• Static membership, using group.instance.id, lets a consumer restart without a rebalance.
• KIP-848, generally available in Kafka 4.0, moves assignment logic to the server.

Two timeouts are often confused. session.timeout.ms detects a dead consumer through heartbeats. max.poll.interval.ms detects one that’s stuck: if a consumer takes too long between polls, it’s removed from the group, its partitions go elsewhere, and work gets repeated. Slow processing causing endless rebalances is a classic production problem.

Level 600: how Kafka is so fast

Kafka handles huge throughput on ordinary hardware, using disks. Several design choices explain why.

Sequential I/O. Appending to a log, and reading it in order, avoids seeks. Sequential disk access is much faster than random access.

Log segments. A partition is a directory of segment files.

/data/orders-0/
  00000000000000000000.log        records
  00000000000000000000.index      offset to file position
  00000000000000000000.timeindex  timestamp to offset
  00000000000000367812.log        next segment

The filename is the first offset in the file. To find an offset, Kafka picks the segment, then uses a sparse index to jump close, and scans forward. Old data is removed by deleting whole segments, which is cheap.

Page cache. Kafka relies on the operating system’s page cache instead of its own. Recent data stays in memory, and consumers reading near the end of the log hit that cache, so reads don’t touch the disk.

Zero copy. Using sendfile, data can go from the page cache straight to the network socket, with no copy into the application. This doesn’t apply when TLS encrypts data in user space.

Batching everywhere. Producers send batches, brokers store them as they arrived, and consumers fetch them. Brokers don’t decompress or rewrite them in the normal path.

Level 700: replication

Each partition has a replication factor, often 3. One replica is the leader, which handles reads and writes, and the rest are followers, which fetch from it.

Partition 0:  Broker 1 (leader)   Broker 2 (follower)   Broker 3 (follower)

In-sync replicas

The in-sync replica set (ISR) is the set of replicas caught up with the leader. A follower that falls behind for longer than replica.lag.time.max.ms is removed from it, and it can rejoin once caught up.

The high watermark

The high watermark is the offset up to which every in-sync replica has the data. Consumers can only read below it.

Leader:     [0][1][2][3][4][5]
Follower A: [0][1][2][3][4]
Follower B: [0][1][2][3]
                        ^ high watermark

This prevents a consumer reading a record that could vanish if the leader dies. The offset after the leader’s last record is the log end offset.

Durability settings

acks=all waits for the whole ISR. But the ISR can shrink. If it shrinks to just the leader, acks=all means one copy. So min.insync.replicas sets a floor.

The usual safe setup is:

With these, one broker can fail with no data loss and no downtime. If two fail, producers get errors rather than a false success. This is a choice of consistency over availability.

Unclean leader election decides what happens if every in-sync replica is gone. Allowing an out-of-sync replica to lead restores availability, and loses data. It’s off by default.

Replicas should be spread across racks or availability zones. Kafka supports this with broker.rack.

Level 800: KRaft

For years, Kafka depended on ZooKeeper for cluster metadata. KRaft replaced it with a built-in Raft quorum, and Kafka 4.0 removed ZooKeeper completely.

A few nodes act as controllers, and they keep metadata in an internal log, __cluster_metadata, replicated with Raft. One is the active controller. Brokers fetch updates from that log. Treating metadata as a log means that a failover needs no huge reload, which allows far more partitions per cluster.

Level 900: exactly-once

Retries cause duplicates: a producer can time out after the broker stored a batch. Kafka handles this in layers.

Idempotent producers

Each producer gets a producer ID, and numbers its batches per partition. A broker rejects a batch whose sequence number it’s already seen, so a retry can’t duplicate. This also keeps order during retries.

Transactions

Transactions make writes across several partitions atomic. This is what makes read-process-write correct.

1. A producer has a transactional.id, and registers with a transaction coordinator, which logs state in __transaction_state.
2. It writes records to any partitions.
3. It also writes its consumer offsets as part of the transaction.
4. On commit, the coordinator writes control records into each partition.

Consumers using isolation.level=read_committed only see committed records. They read up to the last stable offset, so an open transaction holds them back, and a stuck producer can add latency.

The boundary matters. As the Kafka docs note, these guarantees cover Kafka-to-Kafka pipelines. Calling an external API or writing to a separate database is outside them, so those still need idempotency.

Level 950: retention, compaction and tiered storage

Time and size retention delete old segments, with retention.ms, 7 days by default, and retention.bytes.

Log compaction is different. It keeps at least the latest value for each key, and removes older ones.

Before: [k1=a][k2=b][k1=c][k3=d][k2=e]
After:  [k1=c][k3=d][k2=e]

A compacted topic works like a table’s changelog: replaying it rebuilds the current state. A record with a null value is a tombstone, which deletes a key. __consumer_offsets is itself compacted.

Tiered storage (KIP-405) moves older segments to object storage such as S3. Brokers keep recent data locally. That makes long retention much cheaper, and it speeds up recovery, since there’s less local data to re-replicate.

Share groups (KIP-932), introduced in Kafka 4.0, add queue-like consumption. Consumers share partitions, and acknowledge records one by one, so there can be more consumers than partitions. It was early access in 4.0, so check the current status before relying on it.

Level 975: the ecosystem

Schemas matter more than they might seem. Avro or Protobuf, with compatibility checks, stop a producer breaking its consumers.

Stream processing has its own concepts. Windows group events by time, and event time differs from processing time, so late events need handling. Stateful operations keep local stores, backed by changelog topics.

Level 1000: running it

Choosing partitions

More partitions give more parallelism, but cost open files, memory and rebalance time. Increasing partitions later changes key-to-partition mapping, which breaks ordering guarantees for existing keys. So a keyed topic needs some thought in advance. A rough method is to divide target throughput by what one consumer can handle, then add room to grow.

What to monitor

Common failures

Hot partitions. A skewed key overloads one broker. The fix is a better key.

Rebalance storms. Slow consumers keep getting kicked out. Longer max.poll.interval.ms, smaller batches, or moving work off the polling thread all help.

Poison pills. A bad record blocks a partition, because order forces it to be handled first. A dead letter topic is the usual answer.

Disk full. Retention by time alone doesn’t bound disk use when traffic spikes.

Ordering with retries. Idempotence keeps order. Without it, max.in.flight.requests.per.connection above 1 can reorder records.

Large messages. Kafka isn’t built for big values. The usual pattern stores the blob in object storage, and sends a reference.

Cross-zone costs. Replication between zones can be a big cloud bill. Follower fetching lets consumers read from a nearby replica.

When not to use Kafka

Kafka is a heavy tool. For a few background jobs, a simple queue or a database table is much easier to run. It’s a poor fit for per-message routing, long delays, or request-response.

The takeaway

Kafka is a partitioned, replicated, append-only log. Sequential writes, the page cache, batching and zero copy make it fast. Partitions give scale, and order within each. ISR, the high watermark and min.insync.replicas give durability. Consumers own their offsets, which enables replay. Exactly-once holds inside Kafka, and past that, idempotency does the work.