Latency vs Throughput: Sizing LLM Capacity for Production

"How many GPUs do we need" and "how fast will each response feel" are two different questions with two different answers, and sizing for one without the other is how teams end up either overpaying for idle capacity or shipping a product that queues badly under real load. Latency and throughput trade against each other through batching — improving one without understanding the mechanism usually makes the other worse.

Definitions That Actually Matter

• TTFT (time to first token). How long a user waits before anything streams back — dominated by queueing and the prefill pass over the input.
• TPOT (time per output token) / inter-token latency. How fast tokens stream once generation starts — dominated by decode speed, which depends on how much the GPU is shared across concurrent requests.
• Total latency. Roughly TTFT plus (TPOT × output tokens). A short TTFT with slow streaming can feel worse to a user than a slightly longer TTFT with fast, steady streaming — measure both, not just the total.
• Throughput. Aggregate tokens generated per second across all concurrent requests on a given instance or fleet — the number that determines how many simultaneous users one GPU can actually serve.
• p50 / p95 / p99. Percentile latencies. p50 tells you the typical experience; p95 and p99 tell you what your slowest real users experience — and interactive products should be designed against p95, not the average, since the average hides exactly the tail that annoys people enough to notice.

The Core Trade-Off: Batching

A GPU serving one request at a time gives that request the best possible latency, but wastes most of its compute — decode is memory-bandwidth bound, not compute bound, so a single sequence leaves most of the chip idle. Batching multiple requests together lets the GPU process their decode steps in parallel, dramatically raising aggregate throughput. The cost is that any individual request in the batch can experience more queueing and slightly slower per-token generation as the batch grows, since compute is now shared.

Modern inference servers use continuous batching (dynamically adding and removing requests from a running batch as they arrive and complete, rather than waiting to fill a fixed batch before starting) specifically to capture most of the throughput gain of batching without the latency cliff of static, wait-for-a-full-batch approaches. If you're self-hosting, whether your serving stack uses continuous batching is one of the highest- leverage engineering decisions you'll make on the latency/throughput curve — the difference between naive and continuous batching commonly spans several multiples of effective throughput at a given latency target.

What Actually Caps Concurrency: Memory, Not Compute

The intuitive assumption is that GPU compute (FLOPs) is the bottleneck on how many concurrent requests you can serve. In practice, it's usually GPU memory — specifically the KV cache, which stores the attention keys and values for every token of every active sequence so the model doesn't have to recompute them at each decode step. KV cache size scales with context length × batch size × model architecture (layer count, hidden dimension, attention heads). Longer conversations or larger retrieved contexts mean each concurrent sequence eats more memory, which directly reduces how many sequences you can run at once — this is why a chatbot with long context windows per user often hits a much lower practical concurrency limit than its raw compute would suggest, and why agent trajectories that keep growing are a capacity-planning concern, not just a cost one.

A Capacity-Planning Walkthrough

1. 1. Set a latency target at p95, not average — this is the number users actually judge you by.
2. 2. Estimate output tokens per request from real traffic (or your best pre-launch estimate).
3. 3. Estimate peak requests per second, not average — see below for why this distinction matters.
4. 4. Compute required aggregate throughput: peak RPS × average output tokens per request.
5. 5. Compare against your serving stack's measured (not assumed) throughput per instance at your target latency, and size instance count to cover it with headroom.

Worked Example

Illustrative numbers — treat the per-instance throughput figure as a directional stand-in for your own load-tested value, not a benchmark claim for any specific model or GPU:

• Target: p95 latency acceptable at this load, sustained peak of 50 requests/second
• Average output length: 400 tokens/request
• Required aggregate throughput: 50 × 400 = 20,000 tokens/second
• Assumed sustainable throughput per instance at your latency target (measure this — don't assume it): ~800 tokens/second
• Instances needed: 20,000 / 800 → 25 instances, before adding headroom for failover and traffic variability

The one number in this walkthrough you should never take from a blog post — including this one — is per-instance throughput at your latency target. It depends on your specific model, serving stack, batching configuration, and average context length. Load-test it directly, or use our Latency & Throughput Calculator for directional sizing across common instance types and providers.

Peak RPS, Not Average, Is the Sizing Input

This is a different percentile from the p95 latency target above, and conflating the two is a common planning error. Latency percentiles describe the distribution of response times; traffic percentiles describe the distribution of load over time. In this example, average traffic might run 15 requests/second, but real usage patterns spike — business-hours peaks, a marketing push, a viral moment — and the 50 RPS figure used for sizing above is a 3.3x multiple of that average. Sizing capacity to the average instead of a realistic peak is one of the most common causes of production latency degradation: everything looks fine in a load test built around average traffic, then falls over the first time real peak load hits.

Managed API vs Self-Hosted

A managed API abstracts away batching, KV-cache management, and instance sizing entirely — you trade that control for provider-imposed rate limits (requests per minute, tokens per minute), which become your new capacity-planning constraint instead of GPU count. Self-hosting gives full control over the latency/throughput curve but makes you responsible for getting batching, memory sizing, and autoscaling right — mistakes here show up directly as degraded p95 latency or wasted spend on idle capacity. Neither path removes the need for the capacity-planning steps above; it just changes which team does the planning. Whichever path you choose, the per-token or per-instance pricing that feeds into the cost side of this decision is covered in LLM pricing explained and, for self-hosted GPU costs specifically, our GPU training and hosting cost guide.

Common Mistakes

1. 1. Sizing for average RPS instead of peak. As above — this is the single most common cause of a system that works in testing and falls over in production.
2. 2. Assuming throughput scales linearly with instance count. It often does, approximately, for stateless horizontal scaling — but shared dependencies (a rate-limited upstream, a database, a shared cache) can cap real-world scaling well below the naive multiple.
3. 3. Ignoring KV cache memory as the real concurrency limit. Teams size by GPU compute specs and are surprised when memory, not FLOPs, caps how many concurrent sequences they can actually run.
4. 4. Treating total latency as one number. A product with a fast TTFT and slow token streaming needs a different fix (decode throughput, batch tuning) than one with a slow TTFT and fast streaming (queueing, prefill cost, routing).
5. 5. Never load-testing the actual serving stack. Published or assumed throughput numbers — including any in this article — are a starting point for planning, not a substitute for measuring your own configuration under realistic traffic.

FAQ

Should I design to p95 or p99 latency?

p95 is the standard default for interactive products — it captures the experience of your slower-than-typical users without letting rare extreme outliers dictate your entire capacity plan. Reserve p99 targets for cases where a slow response is a hard failure, like a real-time voice interface.

How much headroom should I add over the raw calculation?

A common starting point is 20-30% above your calculated peak-load instance count, to absorb traffic variability and give room for a rolling deploy or an instance failure without an immediate latency hit — treat this as a planning starting point to refine against your own traffic variance, not a fixed rule.

Does a bigger, more capable model always mean worse throughput?

Generally yes, per instance — larger models need more memory and compute per token generated, which reduces achievable concurrency and tokens/second at a given latency target. This is a real factor in the cost/quality tradeoff on top of the per-token pricing difference, and it's part of why cascading to a smaller model for routine requests (see picking the cheapest model that's still good enough) can improve both cost and latency at once.

Is a managed API ever the wrong choice for latency-sensitive workloads?

It can be, if the provider's p95/p99 latency under your traffic pattern doesn't meet your target and you have no control over their batching or queueing. For most products the convenience and elastic capacity of a managed API outweighs this; it becomes a real consideration mainly for very latency-sensitive, high-volume, or highly variable workloads where fine-grained control over batching pays off.

For the token-level cost mechanics that sit alongside this capacity math, see LLM pricing explained.