Twitter

Understanding the Problem

A user follows other users. Anyone they follow can post. The user's feed shows those posts, most recent first — think of it as one mailbox that automatically collects a copy of every letter from everyone you've subscribed to, always sorted so the newest sits on top. This case study intentionally scopes out ranking — reverse-chronological only — so we can focus on the harder structural question underneath any feed: how do you make reading a feed fast when a single post can, in principle, need to reach millions of readers?

Let's turn that into concrete requirements before touching any architecture.

Functional Requirements

Non-Functional Requirements

With what the system does settled, the harder question is how fast and how reliably it has to do it — especially the read side, since a feed gets checked far more often than it gets posted to.

• A feed read must return in under 200ms at p99 — it's the most-loaded read path in the system, since every user checks their feed far more often than they post.
• Assume 100 million users, averaging 200 follows each, with a long tail of "celebrity" accounts that have 10 million or more followers.
• Assume a 10:1 read-to-write ratio on posts — far more feed views happen than posts get created. Concretely: each user posts twice a day and checks their feed 20 times a day, which works out to:
  • 100,000,000 × 2 / 86,400 sec ≈ 2,300 writes/sec
  • 100,000,000 × 20 / 86,400 sec ≈ 23,000 reads/sec, the same 10:1 ratio, with bursts well above that at peak hours Stated assumptions to defend, not given facts.
• A new post doesn't need to appear in every follower's feed the instant it's written — a few seconds of lag while it's still being fanned out is fine. What isn't fine is that lag growing unbounded under load, so the design needs a way to actually watch it (see Operations & Observability, below).

Capacity Estimation

Precomputed feed storage is the number worth deriving, since it's what fan-out-on-write actually costs: if each of 100 million users keeps a capped list of, say, the 1,000 most recent feed post IDs (roughly 30 bytes each in Redis, including overhead), that's 100M × 1,000 × 30B ≈ 3TB of precomputed feed data — a real, sizeable infrastructure cost, not a rounding error, and part of why the design doesn't fan out every account's posts unconditionally. A single post from an average user (200 followers) costs 200 small writes — trivial — but a post from an account above the celebrity threshold would cost one write per follower instead, which is exactly the arithmetic ("10 million individual feed writes for one post") the hybrid design in this lesson is built to avoid.

Here's the number that actually sets where that threshold belongs: fanning out one post costs one write per follower, so the real question is at what follower count that stops being "trivial." Twitter has operated its own real cutover closer to 10,000 followers than 10 million — a sign that "celebrity," for fan-out purposes, means something far more common than the word suggests. At 10,000 followers, one post is still only 10,000 writes, easily absorbed by a small worker pool in well under a second; the arithmetic only turns genuinely scary in the tens-of-millions range this lesson's own NFR describes, which is exactly the tail the hybrid design below exists to catch.

Those QPS numbers also explain the read-side hot spot the caching section below exists to solve: at 23,000 feed reads/sec and roughly 20 posts rendered per feed page, that's about 460,000 post-content lookups/sec landing on the Post Cache — the concrete load behind why a single cache shard holding a viral post would get hammered, and why the redundant-cache design later in this lesson spreads that load across every instance instead.

Core Entities

type Post = { id: string; authorId: string; createdAt: Date; body: string };
type Follow = { followerId: string; followeeId: string };

API Interface

POST /posts                 { body: string }  -> 201 { postId }
GET  /feed?cursor=...                          -> { posts: Post[], nextCursor }

High-Level Design

The core decision is when a feed gets assembled — at write time or at read time. Let's put the two options side by side before choosing between them:

The mail-delivery analogy from Understanding the Problem splits cleanly along this exact line: fan-out-on-write pre-delivers a copy of every post into each follower's mailbox the moment it's created; fan-out-on-read leaves everyone's original where it is and reads each mailbox live, merging them into a feed only when someone actually checks.

Fan-out-on-write makes reads cheap (a feed read is just a list fetch) at the cost of expensive writes — and it breaks down for celebrity accounts: a single post from a 10-million-follower account means 10 million individual feed writes. This is a well-documented real problem — large-scale feed systems (Twitter has written publicly about exactly this) commonly resolve it with a hybrid: fan out writes for ordinary accounts, but for accounts above a follower-count threshold, leave their posts out of precomputed feeds and merge them in at read time instead, since read-time merging for a single celebrity's posts is cheap regardless of how many followers they have.

That's the mechanism at a glance. Now let's make sure the write side that makes it possible for ordinary accounts doesn't itself become a bottleneck, and that the read side doesn't quietly recreate the exact problem it was built to avoid.

Fanning out without melting the write path

Precomputing a million feeds sounds simple until you ask how, mechanically, one post creation turns into that many writes. The naive answer — have the Post Service loop over every follower and write to their feed itself, synchronously, before returning a response to the poster — doesn't survive contact with real follower counts. A request that has to make 200 downstream writes before it can respond is already slow for an average account, and for anyone crossing into five- or six-figure follower counts, that same request either times out or exhausts every outbound connection the Post Service has open.

The fix looks like other places you've probably already decoupled a slow side effect from a fast response: publish one small event ("post X exists") to a queue the instant the post is written, return to the caller immediately, and let a pool of workers pull from that queue and do the actual fan-out asynchronously — in parallel, across as many machines as the backlog needs.

Each follower's precomputed list is naturally a sorted set, not a plain array — scored by recency (a timestamp or the post's own ID), so a worker's write is one insert, and keeping the list capped at 1,000 entries is one trim off the low-scoring end rather than a full re-sort of everything the follower has.

This is also the piece that has to skip celebrity accounts entirely — a worker fanning out a below-threshold post writes a few hundred feeds and moves on; the same worker asked to fan out an above-threshold post would still be running an hour later, which is the whole reason the hybrid split in Trade-offs exists in the first place.

Reading a hot post without hammering one cache shard

Fan-out solves the write side. There's a second, easy-to-miss hot spot on the read side: even with feeds precomputed, actually rendering a feed means fetching the full post content for each post ID in it, and a viral post gets fetched by everyone who has it in their feed, all at once. Cache that post the obvious way — one cache, sharded by post ID, so a given post always lives on the same shard — and you've just recreated the celebrity problem in miniature: the one shard holding the viral post gets hammered while every other shard sits idle.

The fix is almost the opposite move from normal cache scaling: instead of splitting the keyspace across shards, run several full, identical cache instances and let the load balancer send a given read to any of them. No single instance owns Post X, so no single instance can be hammered by it. The cost — covered in Trade-offs below — is that every instance now needs enough memory to hold a broadly useful working set on its own, not a 1/Nth slice of one.

Trade-offs

Pure fan-out-on-write vs. hybrid. Pure fan-out-on-write is simpler to reason about — one code path, no read-time merge logic. But it has a genuine failure mode at the celebrity tail: a single post can trigger a write storm disproportionate to normal traffic.

Chosen: hybrid, with a follower-count threshold deciding which path a given author's posts take. That threshold is a config value, not a fixed constant — Twitter has operated it as low as roughly 10,000 followers in production, tuned at runtime rather than hardcoded, since "how many followers is too many to fan out synchronously" depends on current fleet capacity, not on some inherent property of the number itself.

Naive synchronous fan-out vs. async queue-based workers. Looping over every follower and writing their feed before responding to the poster is genuinely simpler to build and debug — no queue, no worker fleet, no question of whether an event got dropped somewhere. But that simplicity has a hard ceiling: any account with more than a few hundred followers turns "create a post" into a request that's still running when the client gives up waiting.

Chosen: publish one small event to a queue and let a worker pool fan out asynchronously (see High-Level Design). That trades a fire-and-forget event and a small, NFR-bounded amount of eventual-consistency lag for a write path that stays fast no matter how large the fan-out gets.

Sharded vs. redundant post cache. Sharding a cache by post ID is the default move for scaling a cache past one instance's memory, and it's the right call when read traffic is spread evenly across keys — total capacity grows linearly with the number of shards, with no wasted duplication. But post reads aren't evenly spread: a viral post's shard gets every reader of that post while its neighbors sit idle, recreating the celebrity problem at the caching layer.

Chosen: run several redundant, unsharded cache instances instead and let the load balancer pick any of them per request. The real cost — total cache capacity no longer scales with instance count the same way, since every instance needs a broadly useful working set on its own rather than a 1/Nth slice, and a cold instance sees more initial misses than a warm shard would.

Cursor-based vs. offset-based pagination. GET /feed?cursor=... above already commits to cursor-based paging, and it's worth saying why. Offset paging (LIMIT 20 OFFSET 40) is simpler and needs no extra state, but a feed keeps getting new posts while someone scrolls — so the same offset can land on a different row than it did a page ago, showing duplicates or skipping posts.

Chosen: a cursor built from the last post's timestamp and ID, since it means "everything older than what I already saw" instead of a row count that shifts under load. The cost is that jumping straight to an arbitrary page number isn't possible the way offset allows — not something a feed product actually needs.

Ranking as a future extension. Reverse-chronological is explicitly out of scope, but the hybrid design leaves a natural seam for it — a ranking step could re-order the merged candidate set at read time without changing how posts get produced or stored.

Final Design

Put every piece from above together and this is the shape it settles into:

Not every arrow here is a synchronous round trip, which is why this diagram stays one-way rather than two-way: the Post Service → queue hop is fire-and-forget — that's the entire point of the async fan-out design above — even though the Feed Service's reads from the caches and stores below it are genuinely synchronous request/response.

Each piece scales on its own axis, the same argument the caching and fan-out sections above already made individually:

• Post Service and Feed Service see very different load — Feed Service absorbs the 10:1 read-heavy traffic and needs to scale far more aggressively.
• The fan-out worker pool scales with write volume and backlog, independently of both services above it.
• Post Cache and Precomputed Feed Store are both Redis-shaped but serve different purposes — one redundant, to avoid a hot shard; one a plain per-user capped list — and can be sized independently.
• Post DB and Follow DB are the two sources of truth everything above can be rebuilt from.

Operations & Observability

A design that only handles the happy path isn't finished — you also need to know when it's breaking before a follower notices their feed went stale. Raising this unprompted is one of the clearest signals that separates a senior or staff-level answer from one that stops at "and then it works."

On the write side, watch fan-out backlog / worker queue depth first — it's the direct, mechanical readout of the staleness NFR above: a growing backlog means posts are taking longer than a few seconds to reach precomputed feeds, and it's the earliest possible warning, well before any single feed actually looks stale to a user. Pair it with fan-out completion time p99, split by whether a post went through the async-write path or the read-time celebrity merge — the two paths have completely different cost shapes, and averaging them together would hide a regression in either one.

On the read side, watch cache hit ratio, tracked separately for the Post Cache and the Precomputed Feed Store. A drop on the Post Cache specifically, right as read volume spikes, is the exact hot-key signature the redundant-cache design above exists to prevent; a drop on the Precomputed Feed Store instead points at eviction pressure or a cold cache after a deploy, the same cold-cache scenario Level Expectations calls out below. If the celebrity threshold ever gets tuned (see Trade-offs), watch read-time-merge query latency for above-threshold accounts too — more accounts crossing into read-time merging means more per-request query fan-out on the read path, which can quietly erode the 200ms p99 NFR even though nothing about the system technically "broke."

Level Expectations

Mid-level: proposes fan-out-on-write and can explain why it makes reads fast at the cost of write amplification.

Senior: proactively identifies the celebrity-account problem with pure fan-out-on-write and proposes a hybrid before being pushed on it, and separately raises the hot-post read problem — that even a precomputed feed still needs a cache for the actual post content, and a naive cache sharded by post ID just recreates the celebrity problem one layer down. Names fan-out backlog and cache hit ratio as what they'd actually watch in production, not just what they'd build.

Staff: reasons about where a ranking model would plug into this architecture without a rewrite, can discuss how feed staleness (a follow that hasn't propagated yet) gets tolerated or masked, and treats the fan-out threshold itself as a runtime-tunable operational dial rather than a number fixed once at design time — including what breaks first (worker fleet throughput, then read-time-merge latency) as that dial moves.

Follow-Up Questions

Try It Yourself