Building Blocks
Message Queues
At-least-once vs. exactly-once delivery, ordering via partition keys, and dead-letter queues.
A queue shows up in nearly every design that needs to decouple "accepting work" from "doing work." The Notification System and the event-driven half of What Interviewers Assess's trade-off example both lean on one — this lesson is the underlying concepts those examples assume.
Delivery guarantees
| Guarantee | Meaning | Cost |
|---|---|---|
| At-most-once | A message is delivered zero or one times | Simple, but messages can silently vanish on failure |
| At-least-once | A message is delivered one or more times | Never silently drops a message, but consumers must handle duplicates |
| Exactly-once | A message is delivered exactly one time | Requires coordination between the queue and the consumer's storage — genuinely hard to guarantee end-to-end |
At-least-once plus an idempotent consumer (see API Design's idempotency-key pattern) is the practical default almost everywhere — it gets the same user-visible outcome as exactly-once without the coordination cost.
Ordering
A plain queue with multiple consumers gives no ordering guarantee across
consumers — two messages for the same entity can be processed out of order
if they land on different consumers. Partitioning by a key (e.g. Kafka
partitioning by conversationId, the same mechanism the Chat
System case study relies
on) fixes this for messages sharing that key, since a given key's messages
always land on the same partition and get processed in order by a single
consumer — at the cost of no ordering guarantee across different keys,
which is usually fine since nothing outside that key's own state depends
on it.
Producer
Queue
partitioned by key
Consumer
processes messages, ACKs on success
Repeated failure
message routed to Dead Letter Queue
Dead letter queues
A message that fails processing repeatedly — a malformed payload, a bug triggered only by that specific input — shouldn't block every message behind it in the same partition forever. After a bounded number of retries, it gets moved to a separate dead-letter queue instead of retried indefinitely, so the rest of the queue keeps flowing and the failed message is preserved for manual inspection rather than silently lost.
Queue vs. log
A traditional queue (SQS-style) removes a message once it's been successfully processed — one consumer, one delivery. A log-based system (Kafka-style) instead retains messages for a configured window and lets multiple independent consumer groups each read the same stream at their own pace, tracking their own read position. The choice matters: if a second use case will need to read the same events later (auditing, a new consumer added after the fact), a log's replayability is worth the extra operational complexity over a simpler queue.