LLM Inference Service

Understanding the Problem

Think of the GPU here like a short-order cook working a diner counter, not a chef who finishes one full meal, start to finish, before even glancing at the next ticket. A cook who insists on completing one plate at a time leaves most of the grill empty most of the shift — that's exactly what a GPU running one generation request at a time looks like, and fixing it is the entire point of this design.

A language model generates a response one token at a time. Each new token depends on every token generated before it, so a 500-token response costs roughly 500 sequential forward passes through the model, not one — you can't skip ahead or compute the whole response in a single shot.

That per-token cost is real money on scarce GPU hardware. Generating one token for one request barely uses a GPU's available compute, because the operation is memory-bound — mostly reading data back out of GPU memory rather than crunching numbers — so a single request run in isolation leaves most of a GPU's throughput sitting idle, the same empty-grill problem the diner cook above has.

That's why this design is almost entirely about serving many requests together, efficiently, while still getting each individual user their first token back quickly and streaming the rest as they're produced.

Functional Requirements

With the core inefficiency named, let's turn it into a concrete list of what this service actually has to do.

Non-Functional Requirements

Knowing what the service does still leaves the harder question open: how fast, and at what scale? That's what these numbers pin down, and they're what the rest of this lesson is designed around.

• Time-to-first-token (TTFT) under roughly 300ms at p99 — for an interactive experience, this is the latency users actually notice, more than the total time a long generation takes to fully finish.
• Time-per-output-token (TPOT) — the per-token latency once decode is underway, separate from the one-time TTFT above — well under 100ms at p99, comfortably faster than a reader can actually read the words as they stream in.
• Assume 10,000 concurrent requests, each generating around 500 tokens on average.
• GPU capacity is the scarce, expensive resource this design optimizes for — the goal is maximizing useful generated tokens per GPU-second, not just correctness.

Capacity Estimation

With TTFT and concurrency numbers in hand, we can work out the one number that actually decides how far this system scales: how much GPU memory a single request costs.

Every in-flight request holds a KV cache — the model's attention keys and values for every token generated so far — in GPU memory for as long as that request is generating, so it doesn't have to recompute attention over the entire prompt from scratch on every new token. If a single request's KV cache runs to roughly 50MB by the time it has generated 500 tokens (a plausible ballpark for a mid-size model), an 80GB GPU can hold on the order of 80GB ÷ 50MB ≈ 1,600 concurrent requests' KV caches before running out of memory to batch further.

That's the real ceiling on how many requests a single GPU can serve simultaneously — not raw compute FLOPs — and it's the concrete reason the batching strategy below has to manage memory dynamically rather than just packing in as many requests as arrive.

That 1,600 figure is also optimistic in one specific way: it treats the whole 80GB as available for KV cache, when the model's own weights and the serving runtime's working memory are sitting in that same 80GB first.

The KV-cache pool a GPU can actually offer is whatever's left once the model itself is loaded — smaller than the raw GPU size, and smaller still for a larger model, on top of the per-request cost already driving the math above.

For context, a real mid-size (13-billion-parameter) model's KV cache for a single request can grow past 1.5GB once a long, multi-turn conversation's accumulated history fills its context window — well above the 50MB estimate above, which only assumes one 500-token response. The real ceiling moves with how a product actually gets used, not just with model size.

If you've never had to size GPU memory for a model yourself, that's fine — the arithmetic pattern here (total capacity ÷ cost per unit) is the same one you'd use to size anything with a fixed budget per request, and that transferable pattern is what's actually being tested.

Core Entities

With the shape of the constraint clear, let's pin down the actual data every request carries as it moves through the system.

type GenerationRequest = {
  id: string;
  prompt: string;
  status: "queued" | "generating" | "done";
  generatedTokens: string[];
};

API Interface

That's the shape of a request in memory — now let's define the wire contract a client actually calls.

POST /generate   { prompt: string }
  -> a streamed response (SSE or chunked transfer), one token per event,
     terminated by a final "done" event

High-Level Design

One endpoint that only ever returns a stream is a good hint that everything hard about this design lives inside what happens between that request landing and tokens streaming back out — let's build that.

A GPU worker runs a continuous batching scheduler: rather than waiting to accumulate a fixed batch of requests before starting, new requests join the in-flight batch as soon as GPU memory allows, and a request leaves the batch — freeing its KV-cache memory for a new one — the instant it finishes generating. Each token is streamed back to its own client immediately over its own connection, the same incremental-delivery shape Chat System uses for messages, applied here to a single response's own tokens.

Scheduling tokens across many requests

Here's what one scheduling step actually looks like from the GPU worker's side — not one request's journey, but several at once.

At the instant this step starts, Request A is already mid-generation, Request C is about to finish and free its slot, and Request B — which had been waiting — gets admitted into that freed slot before the very next forward pass even runs. One pass produces exactly one new token for every request currently in the batch, whether it joined a millisecond ago or has been generating for the last ten seconds.

That's what "continuous" means here, as opposed to just "batching." Static batching waits to fill a fixed-size batch, runs it start to finish, and only then accepts the next one — the diner-counter equivalent of refusing to seat anyone new until every current customer has finished and left. In practice this gap is large: production LLM-serving benchmarks have measured continuous batching alone delivering roughly 8x higher throughput than naive one-request-at-a-time serving, and over 20x when it's combined with the memory-management technique below.

That memory-management technique is the other half of the story, and it's about the KV cache the diner-counter analogy hinted at from the start.

Managing KV-cache memory without wasting it

Every in-flight request's KV cache has to live somewhere in GPU memory, and the naive way to manage that memory repeats the diner-counter host's mistake from the top of this lesson: reserve one contiguous slab per request, sized for the longest response the model could possibly generate, whether or not the request ever gets that long.

Most requests finish well short of that ceiling, so most of the reserved memory sits empty for the whole request. Real production systems built this way have measured wasting 60-80% of GPU memory to exactly this kind of fragmentation and over-reservation.

The fix borrows directly from how an operating system manages memory for several running programs at once: cut each request's KV cache into small, fixed-size blocks, hand out a block only once a request actually generates enough tokens to need it, and let those blocks live anywhere in GPU memory rather than one contiguous slab, tracked with a small per-request lookup table. Systems built this way have been measured wasting under 4% of GPU memory, instead of 60-80% — very roughly two to four times more usable capacity out of the same hardware, which is the difference between the 1,600-request ceiling above and something closer to a third or a half of that.

If you've never worked with GPU-serving infrastructure directly, that's fine — the idea an interviewer actually wants is the reasoning (a scarce, shared resource that has to be paged and evicted, same as any other memory system), not hands-on familiarity with a specific serving framework.

There's still a hard limit, though: if every block is spoken for and a new token needs one that doesn't exist, something has to give. The scheduler evicts a request's blocks to make room — either recomputing that request from scratch once it's readmitted, or copying its cache out to host memory and back later — both of which cost the evicted request real added latency. Watching how often that happens is one of the things worth watching once this is live, covered below.

We've now covered how a GPU worker schedules requests token by token and keeps its memory from being wasted — next, which of the choices along the way were actually close calls, and why we landed where we did.

Trade-offs

Static batching vs. continuous batching. Static batching gets solid GPU utilization once a batch fills, but every request pays for both the wait to fill it and the pace of its slowest member. The gap this leaves on the table is large: a real benchmark measured naive one-at-a-time serving losing to continuous batching by roughly 8x on throughput alone.

Continuous batching lets requests join and leave every step instead of waiting on a batch boundary, keeping utilization high without making anyone wait on a stranger's slow response. The cost is a genuinely more complex scheduler, one that tracks per-request state and KV-cache memory dynamically instead of running one static tensor batch.

Chosen: continuous batching, since the NFRs explicitly prioritize time-to-first-token, and a static batch window works directly against that.

Naive contiguous KV-cache allocation vs. paged, block-based allocation. Reserving one contiguous slab per request is simple: no lookup table, no block-table indirection, every token's memory access is a straight offset. Its real cost is waste — 60-80% of GPU memory lost on requests that never reach their reserved maximum.

Paged allocation hands out small fixed-size blocks only as they're actually needed, cutting that waste to under 4%. The cost is a lookup table on every attention read, plus a trickier eviction path once memory genuinely runs out.

Chosen: paged allocation — GPU memory is this design's whole bottleneck, so the extra bookkeeping is worth roughly doubling or tripling how many concurrent requests fit on the same hardware.

Streaming tokens vs. returning the full completion at once. Returning the full response only once generation completes is simpler for a client to consume, but for a 500-token response, a user would wait through the entire generation before seeing anything at all — a poor interactive experience. Chosen: stream each token as it's produced, so perceived latency tracks time-to-first-token rather than total generation time.

Final Design

With continuous batching and paged KV-cache memory both decided, we can put the whole system together into one picture.

Requests flow top to bottom the same way the High-Level Design diagram above did, but two things are new here. The Request Router isn't a dumb round-robin — it needs a live read on which GPU worker actually has free KV-cache room, the same capacity number Capacity Estimation worked out earlier, or it'll route a request straight into a worker that has to evict someone else's cache just to make space.

The KV Cache tier under the GPU Worker isn't a separate network hop the way a call to an external cache service would be — it's the same GPU's own memory. It's drawn as its own tier here because it's a genuinely stateful component with its own hard capacity limit, the exact thing the paging mechanism above manages, not because a request ever actually leaves the worker to reach it.

Operations & Observability

A design this dependent on a scarce, dynamically-shared resource doesn't stay healthy on its own — here's what we'd actually watch once it's live, unprompted, which is a large part of what separates a senior answer from one that stops once the design works.

Tokens generated per second, per GPU is the direct readout of how well the continuous-batching scheduler above is actually packing the GPU — a sustained drop usually means either the batch has thinned out (too few concurrent requests to keep every slot full) or an unusual share of traffic is stuck in prefill, which is far more compute-heavy than steady-state, one-token-at-a-time decoding.

Time-to-first-token p99, tracked separately from total generation time, is the NFR's own SLA made observable. Because it's split out from per-token decode latency, a TTFT spike alongside normal decode speed points straight at the Request Router's queueing, not at the model.

KV-cache eviction/preemption rate, straight from the paging mechanism above, is the earliest warning that GPU workers are oversubscribed relative to the memory they actually have — climbing evictions mean requests are losing progress and paying real added latency well before anything times out or actually errors.

Request-queue depth at the router is the simplest signal and the earliest one: sustained growth means requests are arriving faster than GPU worker capacity can absorb them — the moment to add workers, not the moment to wait for TTFT to actually breach its SLA.

Level Expectations

Here's how that depth of reasoning maps onto what we'd actually expect from a candidate at each level.

Mid-level: proposes routing requests to GPU workers with free capacity and can explain why autoregressive, one-token-at-a-time generation makes latency scale with response length.

Senior: proactively proposes batching multiple requests together for GPU efficiency, identifies token streaming as necessary for acceptable perceived latency, and, unprompted, names time-to-first-token p99 and tokens/sec per GPU as what they'd actually watch once this is running in production.

Staff: reasons about continuous batching specifically (not just "batching"), correctly identifies KV-cache memory as the real constraint on batch size rather than raw compute, reasons about paged, block-based allocation as the fix for the waste naive contiguous allocation causes — including what has to happen (eviction, via recompute or a memory copy) when GPU memory genuinely runs out — and can discuss prefix/prompt caching across requests as a further optimization.

Follow-Up Questions

A few questions an interviewer is likely to push on next, if you've made it this far with time to spare.

Try It Yourself

One more to work through on your own before moving on.