Reliability

Every design eventually gets the question "what happens when this component goes down." Having a real answer — not "we'd add more servers" — is what separates a design that sounds robust from one that actually is.

Redundancy

A single instance of anything is a single point of failure. The standard fix is running at least two — a database primary with a replica, two availability zones instead of one — so that losing any single machine doesn't take the system down. The cost is real: two replicas of a database means reasoning about replication lag (see Consistency and CAP), and two availability zones means cross-zone network cost and latency. Redundancy is bought, not free — naming what it costs is worth as much as naming that you'd add it.

Health checks

A load balancer (see Load Balancers) can only route around a failed instance if it knows the instance failed. A health check is a periodic ping — often distinguished as liveness ("is the process still running") and readiness ("is it currently able to serve traffic," which can be false even while the process is alive, e.g. during startup or while it's overloaded). Routing only to instances passing a readiness check, not just a liveness check, is the detail that separates a design that sounds right from one that actually avoids sending traffic to a struggling instance.

Retries, backoff, and jitter

A naive retry — try again immediately on failure — is often what turns a brief blip into an outage: every client retries at once, and the resulting spike is the same thundering-herd problem the Redis lesson named for cache misses. Exponential backoff (wait longer between each retry) helps, but if every client backs off on the same schedule they still retry in lockstep. Jitter — adding a small random delay on top of the backoff — is what actually spreads the retries out.

Circuit breakers and graceful degradation

If a downstream dependency is failing consistently, continuing to call it on every request just adds latency to every caller while waiting for a timeout that's going to fail anyway — this is the pattern popularized by Netflix's Hystrix library. A circuit breaker tracks the failure rate and, past a threshold, "opens" — failing fast locally instead of calling the downstream service at all — then periodically lets a small number of requests through to check if it's recovered. Graceful degradation is the product-level version of the same idea: if the recommendations service is down, show the page without recommendations rather than failing the whole page load. Naming which parts of a design are allowed to degrade instead of fail entirely is a strong signal of thinking about reliability as a first-class requirement, not an afterthought.

Observability: how would you actually know

Everything above assumes you already know something failed. In practice that knowledge comes from three layers worth naming as distinct, not one blob called "monitoring": metrics (numeric time series — request rate, error rate, p99 latency — cheap to store, good for "is something wrong right now"), logs (a record of individual events — good for "what exactly happened to this one request," expensive to store at full volume), and traces (a request's path across every service it touched — good for "which of these five hops added the latency"). An alert fires when a metric crosses a threshold — the failure mode worth naming unprompted is alerting on a raw resource metric instead of a symptom: alerting on "CPU above 80%" pages someone for a non-problem; alerting on "error rate above 1%" or "p99 latency above the SLO" pages someone for something that actually affects a user.

Backups and disaster recovery

Redundancy protects against losing one machine; it does nothing against a bug that corrupts data correctly, on every replica, because the corrupted write itself gets replicated. That's what backups are for — a point-in-time copy that can be restored independently of the live, possibly-corrupted replicas. Two numbers are worth having ready: RPO (recovery point objective — how much data you can afford to lose, i.e. how old your last usable backup can be) and RTO (recovery time objective — how long restoring from it is allowed to take). A design that backs up nightly has an RPO of up to 24 hours; if that's not acceptable for the data in question — a payment ledger, say, see Payment System — the answer is more frequent backups or continuous write-ahead-log shipping, not "we'd add backups" as an unqualified line item.

Multi-region failover is the same idea at a larger scope: if an entire region goes down, not just a machine, traffic and data need a path to a second region. This is expensive — cross-region replication has real latency, and keeping a second region "warm" costs nearly as much as running it — so it's worth reserving for designs where the non-functional requirements actually justify it, not proposing by default.