Distributed Consensus

Kafka named a partition leader and an in-sync replica set without explaining how brokers agree on who the leader is. Cassandra avoids the question entirely by being masterless. This lesson is the piece both of those lean on implicitly: how a distributed system gets multiple independent nodes to agree on one fact — who's the leader, what the current config is — when any of them could be slow, unreachable, or disagreeing about the current state.

Why this is hard

A single node deciding "I'm the leader" is trivial. The hard part is every other node agreeing, including ones that can't currently talk to each other — the same network-partition reality Consistency and CAP names generally. Two nodes that both briefly believe they're the leader — a "split brain" — is the concrete failure mode a consensus protocol exists to prevent: without it, both could accept writes independently, and the system diverges in a way nothing can cleanly reconcile afterward.

What a coordination service actually does

Rather than every system reimplementing consensus from scratch, most production systems delegate it to a dedicated coordination service — ZooKeeper or etcd, most commonly. A coordination service provides a small, strongly-consistent set of primitives — leader election, distributed locks, and a place to store small amounts of configuration that every node needs to agree on — that everything else is built from. It works because it's small and rarely written to; it would be a poor choice as a general-purpose database, but that narrow scope is exactly what makes it practical to keep strongly consistent.

Raft and Paxos, at interview depth

Paxos is the original, famously difficult-to-fully-explain consensus algorithm; Raft was designed later specifically to be more understandable while providing the same guarantee. Both solve the same problem: get a majority of nodes to agree on a single value — e.g. "node B is the new leader" — even if some nodes are down or slow, and make that agreement durable against a minority of failures. The interview-relevant skill here isn't reproducing either algorithm step by step — it's naming which real systems use which, correctly: Raft backs etcd (and, through etcd, Kubernetes' own cluster state); ZooKeeper uses ZAB, a Raft-like protocol built specifically for it, and has historically backed Kafka's controller election and Hadoop HDFS's NameNode failover.

Where this actually shows up in a design

Most designs in this track never call a consensus algorithm directly — they use a building block that already has one underneath. Kafka's partition-leader election has historically been backed by ZooKeeper (newer versions use a built-in Raft-based mode, KRaft, to remove that external dependency); a Kubernetes control plane's notion of cluster state is backed by etcd's Raft log. Naming that a design's "leader election" line really means "delegate to etcd/ZooKeeper, don't build it" is usually a stronger answer than trying to describe building consensus from scratch.