Every quarter the price of a million tokens falls and every quarter the AI invoice lands higher. Finance reads the combination as a billing error. It is not. Cheaper tokens change what product teams attempt, and providers keep finding ways to serve those tokens at a lower cost, which is exactly why the headline price keeps falling. Price and bill are measuring different things.
The useful question is not which model has the lowest rate. It is what one completed task costs, how that cost behaves when the task gets harder, and which three or four assumptions actually drive the answer. This piece walks through the mechanics that set inference cost, then builds a 2027 budget model with every assumption labeled.
Two meters on the same invoice
Every provider bills two quantities: input tokens processed, called prefill, and output tokens generated, called decode. Input is usually cheaper, often by a factor of three to five, and the gap is not arbitrary. Prefill is one large parallel matrix multiplication over the prompt. Decode produces one token at a time, and each token requires reading the model's weights plus the accumulated cache of keys and values for everything generated so far.
Cost per completed task is turns times the sum of input tokens and output tokens at their respective prices, plus retries. The multiplier most teams miss is turns. A chat product resends accumulated history on every user message. In a ten-turn conversation with a 1,000-token system prompt and 400-token replies, the system processes roughly 28,000 input tokens to produce 4,000 output tokens, because each turn resends everything before it. The visible product feature is a conversation. The billing unit is a growing prompt.
Agentic loops make that ratio worse. Each tool call adds a schema, a result, and another model turn, so a single user request can become six or ten requests. Retries, self-critique passes, and fallbacks multiply the total again. None of this appears in a per-token price sheet, and all of it appears on the invoice.
Prefill is compute; decode is memory bandwidth
The hardware explains why output is expensive and why providers keep cutting prices anyway. A decode step must move the weights through the processor for every generated token. An NVIDIA H100 SXM has 80 GB of HBM3 memory and 3.35 TB/s of memory bandwidth, per NVIDIA's published specifications. A 70B-parameter model at 16-bit precision occupies about 140 GB of weights, so it needs at least two of those GPUs, and each generated token requires streaming that model's shard through memory again.
An ideal single-stream ceiling is the aggregate bandwidth divided by the bytes read per token: about 6.7 TB/s across two GPUs against roughly 140 GB of weights, or near 48 tokens per second, before attention, communication between GPUs, and kernel overheads take their share. That number is roughly fixed for a given model and GPU. It is also the reason a single request cannot get much faster, and the reason batching is the entire business model.
The key-value cache is the other half of the memory budget. For the Llama 3 family, Meta documents 80 layers, 64 attention heads, and 8 key-value heads in the 70B model. Each token in an 8,000-token conversation therefore holds 80 layers times 8 heads times 128 dimensions times two tensors for keys and values times two bytes, which is 327,680 bytes, or about 320 KiB per token. The full conversation occupies roughly 2.4 GiB of GPU memory. Sixteen concurrent conversations of that length need about 39 GiB, which no longer fits beside the weights on two 80 GB cards.
That is the real capacity constraint. Memory, not floating-point throughput, decides how many requests can be in flight, and requests in flight decide throughput, and throughput decides what a token costs to serve.
Why batching, not discounting, sets your effective rate
Continuous batching is the technique that made inference dramatically cheaper. Instead of running a batch of complete requests to completion, the server packs decode steps from many requests into one forward pass, admitting and retiring requests as slots free up. The model's weights are read once per step and shared across the whole batch, so cost per token falls as batch size rises, until the workload becomes compute-bound or the cache fills memory.
vLLM's PagedAttention paper from SOSP 2023 described the memory-management side of this: store the key-value cache in fixed-size blocks with an indirection table, the way an operating system pages virtual memory. Fragmentation drops, longer batch sizes fit, and identical prompt prefixes can share cache blocks. The provider captures that saving and competes it away through lower prices, which is why list prices fall while your bill does not.
Two of those savings are available to buyers directly and are routinely left on the table. Prompt caching bills a reused prefix at a fraction of the normal input rate. Anthropic documents cache reads at 0.1 times the base input price, cache writes with a five-minute lifetime at 1.25 times, and one-hour writes at 2 times. Batch processing trades latency for roughly half price: OpenAI's Batch API documents a 50% discount with a 24-hour turnaround. A team that sends a fixed instruction block on every request and generates nightly summaries in real time is paying twice for the same work.
Quantization and routing have ceilings
Quantization reduces the bytes each weight occupies. Moving a 70B model from 16-bit to 8-bit halves the weight footprint to about 70 GB, which fits on one 80 GB card and roughly doubles decode speed, since each step reads half as many bytes. Four-bit methods such as GPTQ and AWQ go further. The catch is that quality loss is uneven: long-context recall, code generation, multilingual output, and strict tool-call formatting tend to degrade before chat quality does, so the only defensible test is an evaluation set built from your own traffic.
Routing sends easy requests to a small model and hard ones to a large model. The RouteLLM research from 2024 reported cost reductions of more than two times in some settings without quality loss, by training routers on preference data. That is a real lever, with three real risks: a router calibrated on the wrong distribution misroutes silently, cascade designs add latency because the cheap attempt must fail before the expensive one starts, and every model pair needs its own calibration data. Routing is not free money. It is engineering work with a measurable payoff if you measure.
A 2027 budget model you can defend
The model below is a worked example. Every price is an illustrative placeholder chosen to resemble mid-tier list rates, not a quote from any vendor. Substitute the rates actually on your contract.
| Assumption | Value | Why it moves the answer |
|---|---|---|
| Completed tasks per month | 2,000,000 | Bottom-up volume, not seat count |
| Input tokens per task | 6,000 | Retrieved context, history, tool schemas |
| Output tokens per task | 700 | Drives the expensive meter |
| Retry and regeneration rate | 15% | Paid twice for the same task |
| Routing mix | 80% small, 20% large | Largest single lever after caching |
| Cacheable share of input | 60% at 0.1x price | Usually 40-70% for support and code work |
| Small model rate | $0.40 in, $1.60 out per million | Placeholder |
| Large model rate | $3.00 in, $12.00 out per million | Placeholder |
Monthly input is 2,000,000 tasks times 6,000 tokens, or 12 billion tokens, and 13.8 billion after retries. Output is 1.4 billion tokens, or 1.61 billion after retries. The blended input rate is 0.8 times $0.40 plus 0.2 times $3.00, or $0.92 per million. With 60% of input billed at one tenth, the effective multiplier is 0.46, so the input rate lands at about $0.42 per million and costs roughly $5,800. Blended output is $3.68 per million and costs about $5,900. Call it $11,700 a month, or roughly $141,000 a year.
The three assumptions that decide the number
Sensitivity matters more than the point estimate. Remove prompt caching and input spend nearly doubles to $12,700, pushing the total to $18,600, up 58%. Route every task to the large model and the total climbs to roughly $38,400, more than three times the mixed design. Double output tokens per task, which is what happens when an assistant becomes an agent, and the total rises by about half even with input flat. Meanwhile the difference between two mid-tier models' input rates is noise next to those three variables.
Commitments and batch share sit just behind them. A team that moves nightly enrichment off the interactive path, keeps stable instruction blocks at the front of every prompt so they cache, and reserves the large model for cases that need it is not optimizing list prices. It is optimizing the shape of demand, which is where the money is.
What to do next
- Instrument cost per resolved task before changing models. Tokens per ticket, turns per session, and retry rate are the inputs to every future forecast.
- Reorder prompts so the stable prefix comes first and the volatile user content last. Cache hit rate follows directly from prompt order.
- Split traffic into interactive, asynchronous, and background lanes, then move every lane that tolerates hours onto a batch endpoint.
- Build a 200-task evaluation set from real failures before routing or quantizing anything. Without it, every saving is an unverified quality risk.
- Put a token budget in each product spec the way you put a latency budget in it. "Maximum 6,000 input tokens per turn" is a design constraint, and a cheaper one than re-architecture later.