How do you scale LLM inference without the cost running away?

To scale LLM inference, serve through an engine with continuous batching and paged KV caching, route and autoscale on tokens, queue depth and cache occupancy rather than request counts, shrink the model with quantisation you have checked against your own evaluation set, and measure time to first token, inter-token latency and cost per thousand tokens. Those five decisions explain most of the difference between a model that serves ten users and one that serves ten thousand.

Serving a large language model to ten users is a tutorial; serving it to ten thousand is a systems design problem. Inference at scale sits where GPU economics, queueing theory and user experience meet, and the decisions interact in ways that surprise teams coming from conventional web scaling. This piece works through what makes it hard, the techniques that buy performance back, and the measurement that keeps the whole thing honest.

Why does LLM inference scale differently from a web app?

Conventional web requests are short, cheap and uniform; LLM requests are none of these. Three properties reshape the problem.

First, generation is sequential. A response is produced token by token, each step depending on the last, so a single request holds resources for the full length of its output: seconds to minutes, not milliseconds. Capacity planning has to think in concurrent long-lived generations, not requests per second.

Second, the memory bill grows with context. Beyond the model weights, each active request keeps a KV cache, the stored attention values for every token in its context, which grows with prompt length and output length. On a busy server this per-request cache, not the weights, is what runs out of GPU memory and caps concurrency. Long-context features multiply the pressure.

Third, work per request varies wildly. A three-word answer and a two-thousand-token essay differ enormously in cost, and you rarely know which one is arriving. Load balancing and autoscaling built on request counts misjudge real load; token throughput is the truer currency.

These properties are why two latency numbers dominate LLM serving: time to first token, how long before the user sees anything (the responsiveness people feel, since responses stream), and inter-token latency, how fast text flows once it starts. Systems are designed and tuned against both, not against one response-time figure.

Which optimisations make the biggest difference?

A handful of techniques, now standard in mature inference engines, account for most of the gap between naive and optimised serving.

Continuous batching is the biggest single win. GPUs are efficient when they process many sequences at once, but requests arrive raggedly and finish at different times. Naive batching assembles a batch, runs it to completion, then starts the next, leaving the GPU idle as sequences finish early. Continuous batching admits new requests into the running batch at every generation step as slots free up. The effect on throughput is large, which is why purpose-built inference servers implement it and generic web stacks struggle to compete.

KV cache management comes next. Paged attention manages cache memory in small blocks, the way an operating system pages memory, which removes fragmentation and fits far more concurrent sequences on each GPU. Prefix caching extends the idea: when many requests share a long common prefix (a system prompt, few-shot examples, a document under discussion), the cache for that prefix is computed once and shared.

Quantisation shrinks the model by storing weights at lower numerical precision. Smaller weights mean less memory bandwidth per token and more room for cache, usually a large serving win for a small quality cost. That cost has to be measured against your own evaluation set, not assumed.

Speculative decoding attacks sequential latency: a small draft model proposes several tokens cheaply, and the large model verifies them in one parallel pass, keeping the correct prefix. When the draft model guesses well, generation speeds up with no change in quality.

How should routing and autoscaling work around the engine?

The inference engine handles one replica; scaling is what happens around it.

Routing deserves more design than it usually gets. A load balancer that spreads requests evenly by count overloads replicas that happen to hold long generations, so token-aware or least-loaded routing performs better. Session affinity pays off with prefix caching: send a user's follow-up turns to the replica already holding their conversation's cache. Above routing sits admission control: a queue with backpressure, per-client rate and token limits, and an honest rejection when the system is saturated. An overloaded service that accepts everything degrades for everyone; one that queues openly and sheds excess load protects both latency and cost.

Autoscaling needs signals that fit LLMs. CPU utilisation means nothing here, and even GPU utilisation misleads. The signals that track real capacity are concurrent sequences, KV cache occupancy, queue depth and a rising time to first token. Scaling out is also slow, because a new replica has to pull a many-gigabyte model and warm up, so policies should scale early on leading indicators and keep enough headroom to ride out spikes, with the cost of that headroom weighed explicitly.

Two more patterns appear in mature deployments. Disaggregated serving splits the prefill phase (processing the prompt: compute-bound and parallel) from the decode phase (generating tokens: memory-bound and sequential) onto different hardware pools, so each scales to its own bottleneck. Model routing accepts that not every request needs the largest model: a classifier or rule sends simple requests to a small, cheap model and keeps the large one for hard cases. That cuts fleet cost at the same perceived quality, provided the routing decision is itself evaluated and monitored.

What should you measure?

Scaled inference without measurement drifts into waste, degradation or both. A serving stack needs three layers of telemetry.

Request level: time to first token and inter-token latency as percentile distributions (the tail is where users suffer), queue wait, tokens in and out per request, and rejection and timeout rates. Fleet level: KV cache occupancy, batch size achieved, token throughput per GPU, and cost per thousand tokens served, the number that ties engineering decisions to the bill. Quality: sampled output evaluation, error and refusal rates, and drift in prompt and output lengths, because a change in how the system is used quietly changes how it performs and what it costs.

The insight lives in correlating the layers: a tail-latency regression traced to cache saturation after a new long-context feature, or a cost spike explained by a traffic mix that defeats the model router. Build dashboards around questions like these before scale forces them at an inconvenient hour.

How do you practise this?

The concepts reward hands-on work: serving a small open model on one GPU teaches batching, caching and latency trade-offs faster than reading. The LLMOps bootcamp is built around exactly this: serving, observing and paying for models in production, with each project graded against a rubric you can read. To find your starting point first, take the free 3-minute skill check.