How Apache Kafka Works

Apache Kafka is a distributed, durable log of events. Applications write events to it, Kafka stores them in order across a cluster of servers, and any number of other applications read them back — now, or days later. That one idea, an append-only log that many readers can share, is what lets Kafka sit at the centre of payments systems, analytics pipelines and microservice architectures.

This guide walks through every moving part. Each section maps to something you can watch happen in the interactive simulator.

The big picture

Kafka has three roles:

• Producers — applications that write events (for example a checkout-service publishing each order).
• Brokers — the Kafka servers that store events. A group of brokers is a cluster.
• Consumers — applications that read and process events (for example a billing service).

Producers and consumers never talk to each other directly. They only know about Kafka. That decoupling is the point: you can add a new consumer next month and it can read the full history without the producer changing a line of code.

Topics and partitions

Events are organised into topics — a named stream such as orders or payments.

A topic is split into partitions. Each partition is an ordered, append-only log: new messages are added to the end and each one receives the next offset (0, 1, 2, …). Messages are never edited in place; they’re removed only when retention expires.

Partitions are how Kafka scales. Different partitions can live on different brokers and be read by different consumers in parallel. The trade-off: ordering is guaranteed only within a single partition, not across a whole topic.

You can add partitions to a topic later, but you can never remove them.

Partition keys: who goes where

When a producer sends a message it can attach a key — a customer ID, an order ID, an account number. Kafka picks the partition like this:

partition = murmur2(key) % number_of_partitions

So the same key always lands in the same partition, and therefore all events for that key stay in order. If there’s no key, the producer spreads messages across partitions to balance load.

This is also where things go wrong. If one key is far busier than the others — say one enormous customer — its partition becomes a hot partition: one broker and one consumer do most of the work while the rest sit idle. Choosing a key with enough variety is one of the most important Kafka design decisions. Try the Hot partition scenario in the simulator to see it.

Brokers, replication and leaders

Each partition is copied to several brokers. The number of copies is the replication factor (commonly 3).

• One copy is the leader. All reads and writes for that partition go to it.
• The others are followers. They continuously copy the leader’s log.
• Followers that are fully caught up form the in-sync replica set (ISR).

Consumers can only read up to the high watermark — the last offset that every in-sync replica has copied. That guarantees a consumer never sees a message that could disappear if the leader crashed.

What happens when a broker crashes

If a broker dies, every partition it was leading needs a new leader. Kafka’s controller picks one from that partition’s ISR, and producers and consumers redirect to it. Because the new leader was in sync, no acknowledged data is lost. If no replica of a partition is left alive, that partition goes offline until a broker comes back.

The Broker crash scenario shows the election as it happens.

Consumer groups

Consumers that share a group id cooperate to read a topic:

• Each partition is assigned to exactly one consumer in the group.
• A consumer can own several partitions.
• If there are more consumers than partitions, the extras sit idle — the number of partitions is the maximum parallelism of a group.

Different groups are completely independent. A billing group and a shipping group each receive every message on the topic. That’s how Kafka acts as both a queue (within a group) and pub/sub (across groups).

Partitions are handed out by an assignment strategy such as range (contiguous blocks) or round-robin (dealt out one by one).

Offsets, commits and lag

Kafka doesn’t delete a message when it’s read. Instead, each consumer group commits how far it has got in each partition — its committed offset.

• Lag = log end offset − committed offset. It tells you how far behind a group is, and it’s the single most useful metric to alert on.
• If a consumer crashes after processing a message but before committing, the next owner of that partition starts from the last commit and processes the message again. This is at-least-once delivery, and it’s why consumers should be idempotent.
• A brand-new group chooses where to start with auto.offset.reset: earliest replays the whole retained history, latest reads only new messages.

The Many groups scenario adds an audit group with earliest and you can watch it replay the full log.

Heartbeats and rebalancing

Consumers send regular heartbeats to the group coordinator. If a consumer stops heartbeating for longer than the session timeout, Kafka assumes it’s dead and triggers a rebalance: the group’s partitions are redistributed among the remaining members. The same happens when a consumer joins or leaves.

During a classic (eager) rebalance every consumer briefly stops processing, so frequent rebalances hurt throughput. Watch it happen in the Consumer crash scenario.

Retention

Kafka keeps messages for a configured retention period (or size), whether or not anyone has read them. This is what makes replay possible: a new service, a bug fix or a backfill can re-read history straight from Kafka.

Kafka in 10 concepts

Concept	In one line
Producer	Writes events to a topic and picks the partition using the key.
Topic	A named stream of events, split into partitions.
Partition	An ordered, append-only log; ordering is guaranteed only here.
Partition key	hash(key) % partitions — same key, same partition, same order.
Broker	A Kafka server storing partition replicas.
Leader & followers	One replica serves traffic; in-sync followers take over if it fails.
High watermark	Consumers see only messages copied to all in-sync replicas.
Consumer group	Partitions are split among members; each group gets every message.
Offset & lag	Groups commit progress per partition; lag is how far behind they are.
Rebalance	Partitions are redistributed when consumers join, leave or crash.

Try it yourself

The Kafka simulator runs a real model of all of the above in your browser — murmur2 key hashing, replication, high watermarks, ISR-based leader election, heartbeats, rebalances, committed offsets, redelivery and retention. Pick a guided scenario (Kafka 101, Consumer crash, Broker crash, Hot partition, Scaling consumers, Many groups) or build your own cluster in the sandbox. Press Space to pause at any point.