Kafka

Message Queues drew the line between a traditional queue and a log-based system, and named Kafka as the log-based example without going further. This lesson is that log-based model made concrete — the shape almost every design reaches for when "we'd put a queue in front of it" turns into "which one, specifically."

Topics, partitions, and offsets

A Kafka topic is split into partitions, and each partition is an append-only, ordered log — a producer writes a message, it gets an offset (its position in that partition), and it stays there for a configured retention window rather than being deleted the moment it's read. That last part is the key difference from a traditional queue: reading a message doesn't remove it, so multiple independent readers can each consume the same partition at their own pace, tracking their own offset.

A second consumer group — analytics, an audit pipeline added months later — can read the exact same partition from the beginning, independently of group A's progress, because nothing was ever removed. This replayability is the entire reason to reach for Kafka over a simpler queue: it's only worth the operational cost when more than one consumer genuinely needs the same stream.

Partitioning is how ordering and scale coexist

Kafka only guarantees order within a single partition, not across an entire topic. Producers pick a partition per message — commonly by hashing a key, the same mechanism the Chat System case study relies on by partitioning on conversationId — so every message for that key lands on the same partition and is processed in order, while different keys spread across partitions for parallelism. More partitions means more consumers can read in parallel; fewer means stronger ordering guarantees across a wider set of keys. That's the actual trade-off, not a default to memorize.

Consumer groups

Within one consumer group, each partition is read by exactly one consumer at a time — adding more consumers to a group (up to one per partition) increases parallelism, and losing a consumer triggers a rebalance where its partitions get reassigned to the survivors. This is also why partition count is a real capacity decision made up front: it caps how many consumers in a single group can ever do useful work in parallel, and increasing it later is possible but disruptive.

Replication and the durability/latency trade-off

Each partition has a leader broker plus replica brokers holding copies; producers write to the leader, and replicas that are caught up form the in-sync replica (ISR) set. A producer can ask for an acknowledgment as soon as the leader has the write, or wait until every in-sync replica has it too — the same Reliability trade-off named for redundancy generally: waiting for the full ISR is slower per write but survives a leader failure without losing that message, while acknowledging on the leader alone is faster but can lose the most recent writes if the leader dies before replicating them.