Case Studies
Web Crawler
Politeness, per-domain rate limiting, and a memory-bounded way to avoid re-crawling the same URL.
Designing a crawler that discovers and fetches pages starting from seed URLs — a good vehicle for testing whether a candidate can reason about politeness, memory-bounded duplicate detection, and frontier prioritization, not just raw fetch throughput.
What this lesson covers
A domain-partitioned URL frontier enforces politeness with a per-domain rate limiter and robots.txt crawl delay; fetchers normalize URLs and check a Bloom filter for memory-bounded dedup with rare false positives. Order the frontier BFS by default, priority as the escalation.
- It sounds like 'fetch a lot of URLs fast', but politeness, deduplication, and crawl traps each force a real design decision.
- Politeness means never overloading one site: respect robots.txt and a per-domain crawl delay no matter how many workers you run.
- Remembering every URL exactly gets expensive at billions of pages, so dedup has to be memory-bounded and accept a small error.
- A crawl trap is a site generating endless new-looking URLs; a depth cap is the concrete mitigation to name.
- Staff answers escalate to priority ordering, near-duplicate content detection, and sharding the frontier across machines.
Included in Plus and Pro
Continue reading “Web Crawler”
The full lesson works the design through end to end — diagrams, trade-offs, and what interviewers expect at each level. About 20 minutes.
- Understanding the Problem
- Functional Requirements
- Non-Functional Requirements
- Capacity Estimation
- Core Entities
- API Interface
- High-Level Design
- Trade-offs
- Final Design
- Operations & Observability
- Level Expectations
- Follow-Up Questions
- Try It Yourself