AI Latency & Throughput Calculator
Size your infrastructure for any AI workload. Estimate latency, throughput, and cloud costs before you build.
Frequently Asked Questions
What does requests per second (RPS) mean for LLM APIs?▼
RPS measures how many inference requests your system sends to the model per second. For a chatbot serving 1,000 users simultaneously, each sending a message every 20 seconds, that's 50 RPS. Throughput directly affects infrastructure cost, queue depth, and perceived latency.
What is the difference between p50, p95, and p99 latency?▼
Percentile latencies tell you how fast your slowest requests are. p50 (median) means 50% of requests complete in that time or less. p95 is more important for user experience — 95% of requests complete in that time or less. p99 captures near-worst-case scenarios. Always design to p95 for interactive AI applications.
When should I self-host instead of using the OpenAI or Anthropic API?▼
Self-hosting makes financial sense when your token volume exceeds roughly $5,000–$10,000/month on managed APIs. It also makes sense when you need data privacy, custom fine-tuning, predictable latency, or freedom from rate limits. Below that threshold, managed APIs offer lower operational overhead and faster iteration.
How accurate are these infrastructure estimates?▼
These are directional estimates based on published GPU throughput benchmarks and public cloud pricing (May 2026). Real-world performance varies based on batch size, prompt length, system prompt reuse, KV-cache hit rate, and network conditions. Use these numbers for planning and order-of-magnitude decisions, not SLA commitments. Load-test your specific workload before production.
How this calculator works
For managed APIs (OpenAI, Anthropic), the calculator estimates latency from each model's typical time-to-first-token and output tokens-per-second, then adds a queueing delay that grows as your requests-per-second approaches the API's assumed shared capacity. Monthly cost is calculated the same way as the other calculators — tokens in and out, priced at the current rates shown in the calculator — with no infrastructure to size or provision.
For self-hosted deployments (AWS, GCP), it instead works out how many GPU instances you need: it converts your requests-per-second and tokens-per-request into required output tokens per second, picks the smallest GPU instance type with enough VRAM for the model, and sizes the instance count to keep utilization around 65% for headroom. Monthly cost is that instance count times its hourly rate times 730 hours. Latency and cost estimates both come from published GPU throughput benchmarks and public cloud pricing, and are directional planning numbers rather than SLA guarantees.
Self-hosted sizing
outputTokensPerSec = requestsPerSecond × tokensPerRequest × 0.3
instancesNeeded = ceil(outputTokensPerSec ÷ (modelTokensPerSecPerGPU × 0.65))
monthlyCost = instanceHourlyCost × instanceCount × 730
How to use it
- 1.Enter your expected requests per second — peak load, not average, if uptime under load matters.
- 2.Pick a latency target (e.g. under 1 second) so the calculator can flag whether a deployment option meets it.
- 3.Choose the model and typical tokens per request.
- 4.Select a deployment: managed API (OpenAI/Anthropic) or self-hosted (AWS/GCP).
- 5.Compare the provider table for cost, p50/p95 latency, and whether each option meets your latency target.
Common mistakes
- ⚠Sizing for average load instead of peak — infrastructure that handles your average requests-per-second comfortably can fall over during traffic spikes — size to your expected peak, then add headroom.
- ⚠Ignoring p95 in favor of p50 — median latency looks great even when a meaningful share of users wait far longer — design and budget around p95, not the average case.
- ⚠Assuming self-hosting is always cheaper at scale — self-hosting adds GPU provisioning, scaling, and ops overhead that a managed API doesn't — the crossover point is usually in the thousands of dollars per month, not immediately.
- ⚠Treating warning messages as optional — a "utilization above 80%" or "exceeds latency target" warning means the sizing shown will likely degrade under real traffic — add capacity before it becomes a production incident.
Frequently asked questions
What does requests per second (RPS) mean here?▾
It's how many inference requests your system sends to the model per second. A chatbot serving 1,000 simultaneous users, each messaging roughly every 20 seconds, is about 50 RPS. Throughput at this level directly drives both infrastructure cost and queueing delay.
What is the difference between p50, p95, and p99 latency?▾
These are percentiles: p50 (median) means half of requests finish in that time or less. p95 means 95% do — a much better signal for real user experience, since it captures the slower tail. p99 captures near-worst-case requests. Design interactive applications around p95, not the average.
When should I self-host instead of using a managed API?▾
Self-hosting tends to make financial sense once your managed API spend reaches the higher end of typical production budgets, or when you need data privacy, custom fine-tuning, predictable latency, or freedom from rate limits — compare the provider table above for your specific volume.
How accurate are these infrastructure estimates?▾
They're directional, based on published GPU throughput benchmarks and public cloud pricing. Real-world performance varies with batch size, prompt length, system-prompt reuse, KV-cache hit rate, and network conditions — use these numbers for planning, not as an SLA commitment.
Why does self-hosted cost look like a step function as I increase RPS?▾
Because GPU instances come in fixed sizes — the calculator rounds up to the next whole instance whenever your required throughput exceeds what the current instance count can handle at 65% utilization, so cost jumps in steps rather than scaling perfectly smoothly.