The oversizing usually happens before anyone touches a spec sheet. A team picks a model, picks a GPU type, multiplies by the peak request rate, adds 50% for safety, and buys a commitment sized for a number the fleet will hit for ninety minutes a day. The bill is then fixed for a year while the utilization chart shows a flat line at 22%.
Sizing AI infrastructure is a capacity-planning problem with two unusual properties: the throughput metric is tokens per second per model, not requests per second per machine, and the bottleneck is memory bandwidth and cache capacity rather than raw floating-point throughput. Both properties have to be in the spreadsheet before any number purchased means anything.
Why FLOPS is the wrong starting point
Generated tokens come out of a memory-bound process. Each decode step reads the model's parameters and the key-value cache, then produces one token per sequence in the batch. An NVIDIA H100 SXM provides 80 GB of HBM3 and 3.35 TB/s of bandwidth, per NVIDIA's specifications. For a single request, the ceiling is bandwidth divided by bytes read per token, which is why one request per GPU runs at tens of tokens per second no matter how many tensor cores are idle.
Batching breaks that ceiling because the weights are read once per step and used by every sequence in the batch. Ten concurrent sequences do not read ten copies of the model. That is the whole economic argument for serving infrastructure, and it means the right unit for planning is aggregate tokens per second at a defined latency target, not per-request speed.
Count the cache before you count the GPUs
Memory is the budget that limits batching. Weights take their share, and the key-value cache takes the rest. For a 70B-class model with 80 layers, 8 key-value heads, a head dimension of 128, and 16-bit cache entries, the cache costs 327,680 bytes per token, or roughly 320 KiB. An 8,000-token conversation therefore holds about 2.4 GiB, and a batch of 16 such conversations holds about 39 GiB.
Weights at 16-bit precision occupy about 140 GB, so the model alone needs two 80 GB GPUs. At 8-bit precision the weights drop to roughly 70 GB, which fits one 80 GB card with little room for cache; many production deployments use two GPUs per replica so the cache has headroom. Either way, the sequence limit comes from the cache, and the sequence limit sets throughput. A team that plans capacity from FLOPs alone will buy cards that are memory-starved and slow.
Little's law is the sizing formula
Concurrency equals arrival rate times time in system. At 8 requests per second and 3 seconds of end-to-end latency, 24 requests are in flight on average. If each replica can hold 15 concurrent sequences at the required context length, one replica is not enough for the peak and two are more than enough. Notice that this calculation never mentions the model's parameter count; it is about how long work stays in the system.
Queue depth is the dial between cost and latency. Allowing requests to wait raises utilization and lowers cost per token, at the price of p95 latency. A useful policy splits traffic into classes: interactive requests get capacity sized for 60% average utilization so a burst finds headroom, while batch and background jobs can run at 85% or higher because a two-minute wait is acceptable. Without that split, everything gets sized for the strictest class, and the batch jobs inherit an expensive neighborhood.
The throughput number in any capacity model deserves a benchmark rather than a vendor claim. Send a representative mix of prompt and output lengths at increasing concurrency, and record aggregate output tokens per second alongside p95 latency. Most serving stacks hit peak throughput well before they hit acceptable latency, and the point where the latency curve turns is the number to plan around. Measuring that curve once, on your own traffic, prevents a purchase decision from being based on a demonstration that used short prompts.
Reserved, on-demand, and burst
Reserved capacity is cheapest per hour and worst per idle hour. On-demand capacity costs more but can be returned. Serverless model endpoints and API calls cost the most per token and require no commitment at all. The right mix follows the load curve, and most load curves have three sections: a flat base that runs all month, a daily and weekly shoulder, and a thin spike.
Reserve the base, cover the shoulder with on-demand or short commitments, and absorb the spike with burst capacity or by queueing. The mistake is reserving the spike because it is the number leadership remembers. A spike that lasts 90 minutes a day is 6% of the month, and committing a year of capacity to it converts a small overspend into a structural one.
Worked example: a 2027 support assistant
Every figure below is a labeled assumption for illustration, not a vendor quote. The point is the method and the shape of the answer.
| Assumption | Value |
|---|---|
| Sessions per month | 7 million |
| Output tokens per session | 300 |
| Peak request rate | 8 per second |
| Average request rate | 2.7 per second |
| Latency target | 3 seconds at p95 |
| Model | 70B parameters at 8-bit, two GPUs per replica |
| Sustained throughput per replica | 500 output tokens per second aggregate |
| Committed GPU rate | $3.00 per GPU-hour |
| Burst or API rate | $12.00 per million output tokens |
Peak output demand is 8 requests per second times 300 tokens, or 2,400 tokens per second. At 500 tokens per second per replica that is 4.8 replicas, rounded to five, plus one for redundancy, so six replicas and 12 GPUs. At $3.00 per GPU-hour and 730 hours, the committed cost is about $26,300 a month. Monthly output is 7 million sessions times 300 tokens, or 2.1 billion tokens, so the peak-sized fleet costs roughly $12.50 per million output tokens.
The peak-sized fleet is idle most of the month, because average demand is 2.7 requests per second times 300 tokens, or 810 tokens per second, which 1.6 replicas could serve. Reserving two replicas costs $8,760 a month and handles demand up to 1,000 tokens per second. Sending the roughly 35% of monthly tokens that arrive above that rate to a burst endpoint adds about $8,800, for a total near $17,600 a month, or about $8.40 per million tokens. Sending everything to an API at the assumed rate costs about $25,200 a month.
| Strategy | Monthly cost | Cost per million output tokens |
|---|---|---|
| Peak-sized reserved fleet (12 GPUs) | $26,300 | $12.50 |
| Base reserved plus burst (4 GPUs) | $17,600 | $8.40 |
| All burst or API | $25,200 | $12.00 |
The result is not that owning hardware is always wrong. It is that peak-sized reservations are the most expensive of the three options, and they lose to doing nothing at all. Owned capacity competes when the pool you commit stays busy: here the two-replica base runs at roughly half its capacity on average and still beats the API rate, because committed GPU hours cost $3.00. A pool that idles at 15% average utilization costs more than three times as much per token and loses outright. That is the difference between reserving a base and reserving a spike.
The mistake that doubles the bill
Sizing for peak is the visible error. The hidden one is sizing per model. Three models, each with its own dedicated replica set sized for its own peak, triple the fixed cost while each set sits idle most of the day. Multi-model serving on a shared pool, with routing at the gateway and cache-aware scheduling, is the difference between a fleet and a collection of pet deployments.
The second hidden error is ignoring time-to-first-token. Long prompts make prefill the dominant cost in the user's perception of speed, and a fleet tuned only for decode throughput will feel slow while reporting healthy token rates. Track prefill and decode separately, and set different latency objectives for each.
What to do next
- Measure your load curve for two weeks: requests per second, tokens per second in and out, and p95 latency, segmented by traffic class.
- Compute the memory budget for one replica before choosing a GPU count: weights plus cache at your real context length.
- Size for average load plus a stated utilization target, then add burst capacity for the shoulder instead of reserving it.
- Model three strategies in one spreadsheet, as above, and compare cost per million output tokens rather than monthly bills.
- Re-check the decision when a new model changes throughput per GPU; a two-times improvement in serving efficiency invalidates the previous answer.