Investors do not underwrite an AI company's revenue. They underwrite the margin that revenue will carry once the product stops being subsidized by growth capital. That is why the gross margin question arrives earlier in AI than it did in SaaS, and why it is usually asked badly: what is your gross margin, as if there were one number and one definition.
There are several numbers, and the difference between them is the argument. A board that understands which costs sit in cost of goods sold, which levers move them, and how pricing architecture shifts the risk can test a plan in an afternoon. A board that does not will underwrite a projection that quietly assumes inference gets cheaper forever.
Gross margin is a definition before it is a number
Gross margin is revenue minus cost of revenue, divided by revenue. Everything depends on what you put in cost of revenue. For an AI product, the honest test is whether a cost scales with usage. If serving one more customer or one more task increases the cost, it belongs in cost of revenue. If it does not, it belongs in operating expense, where investors will treat it as a fixed investment rather than a variable drag.
The pushback is always the same: our engineers are building the product, so their cost is research. Sometimes true, often not. The team that tunes prompts, evaluates models, and handles escalations is serving customers, and investors will reclassify that labor whether or not management does. A defensible model names the boundary and applies it consistently across cohorts and quarters.
What belongs in AI cost of revenue
- Inference spend with model providers, including retries, failed calls, and evaluation traffic that runs in production.
- Reserved or committed GPU capacity, whether cloud commitments or owned hardware, including the idle share you pay for to hold latency targets.
- Retrieval and vector infrastructure, which scales with documents indexed and queries served.
- Third-party APIs and tooling embedded in the product, from speech to search to document parsing.
- Human review, labeling, and correction work that keeps output quality acceptable, including escalation handling.
- Application hosting, observability, and support costs that scale with accounts or usage.
Payment processing belongs there too, though it is usually already counted. What should not be there is sales and marketing, general and administrative cost, or the platform team that builds shared internal tooling. Move any of those in and the margin looks worse but the business is no different. Move inference out and the margin looks better but the business is misrepresented.
Three levers that actually move cost of revenue
The foundation under all three is batching. Serving engines schedule many requests through one accelerator at once and manage the key-value cache so that memory is reused rather than reallocated; the vLLM project documents the paging approach that made this practical for large models. Higher batch size spreads fixed accelerator cost across more tokens, which is why utilization, not list price, is the first number to fix.
The first is routing. Most traffic does not need the most capable model. A router that sends simple requests to a smaller, cheaper model and reserves the frontier model for hard cases cuts the blended cost per request without changing the product for most users. The engineering cost is in evaluation: you need a way to measure quality per route, or the savings arrive as a quality regression that shows up in churn a quarter later.
The second is quantization. Reducing weight precision lowers memory footprint and memory bandwidth per token, which matters because generation is bandwidth bound. A four-bit quantized model can serve more concurrent requests on the same accelerator, and post-training quantization methods have made the quality loss small enough for many production tasks. The trade-off is that quality loss is uneven: it shows up in reasoning-heavy tasks and long outputs before it shows up in short answers, so test on your own evaluation set rather than a public benchmark.
The third is caching. Providers price cached input far below fresh input because the expensive part, the prefill over the prompt, is already computed. Anthropic's documentation prices cache reads at roughly a tenth of the base input rate and charges a premium on cache writes. Batch endpoints offer a similar structural discount in exchange for latency. Either way, the savings depend on prompt layout: a stable prefix that stays identical across calls is cacheable, while timestamps and per-user identifiers at the top of the prompt defeat the cache.
These three levers are complementary and they are not infinite. Routing is limited by how much of your traffic is easy. Quantization is limited by your quality floor. Caching is limited by prompt repetition and cache lifetime. A plan that claims all three compound to an 80% reduction is a plan that has not been tested against a production traffic mix.
Pricing architecture decides who absorbs the volatility
Seat pricing gives the buyer a fixed bill and gives the vendor all the demand risk. If usage per seat doubles, the vendor pays for it. That works when usage is predictable and the product is used in short sessions. It fails when a customer points an agent at a large repository and the variable cost per seat becomes unbounded.
Usage pricing passes the risk to the buyer, which is fair but sells badly to enterprises that cannot forecast spend. Most AI companies land on a hybrid: a platform fee, a pool of included usage, and metered overage. The architecture matters to gross margin because it determines which customers are unprofitable. A seat contract with one heavy user and nine light ones can carry a blended margin that hides a negative-margin account, and at renewal that account negotiates from your own usage data.
Whatever the structure, the board should see margin by cohort and by customer decile, not just the blended number. Blended gross margin is an average of a profitable core and a tail that may be subsidized.
A worked margin model
All figures below are assumptions for illustration, expressed as a percentage of revenue for a mid-sized AI application. They are not benchmarks, and the point is the arithmetic rather than the values.
| Cost of revenue line | Current (assumed) | After routing, quantization, caching (assumed) | What moves it |
|---|---|---|---|
| Inference and model calls | 34 | 20 | Routing mix, cache hit rate, output length |
| Reserved GPU and committed capacity | 6 | 6 | Fixed by contract, visible in the commitment |
| Retrieval and vector infrastructure | 4 | 4 | Index size and query volume |
| Third-party APIs and tooling | 5 | 4 | Volume tiers and vendor consolidation |
| Human review and labeling | 5 | 4 | Automation and escalation thresholds |
| Hosting, observability, support | 4 | 4 | Mostly fixed at current scale |
| Payment processing | 3 | 3 | Revenue mix and processor terms |
| Total cost of revenue | 61 | 45 | |
| Gross margin | 39% | 55% |
The starting point is deliberately unflattering. A 39% gross margin is below the range that a16z described in its analysis of AI business models, where gross margins often fell between 50% and 60% against 60% to 80% or more for comparable software companies. That analysis predates the current generation of model APIs, but the direction still holds, and the gap is the reason investors ask the question.
Two sensitivities matter more than the headline. If inference cost doubles from the assumed base, the margin falls by roughly 34 points, which is the whole argument for routing and caching before growth spending. If the optimization program delivers only half of the assumed reduction, margin lands closer to 47%, which may still be a good business but is a different story to tell a board.
The useful per-unit version of the model is cost per successful task: tokens in and out at their respective rates, times the average number of turns, plus retries and the expected human escalation cost. Track that number weekly. It moves before the blended margin does, and it is the number a technical investor will ask about.
What a board will test
Expect five questions. Is the margin improving because of engineering or because of price increases? What is the margin on the newest cohort, which is the best proxy for where the business lands? How much of the improvement depends on provider price cuts you do not control? What happens to margin if usage per customer doubles? And which contract structures create unbounded cost exposure?
A company that can answer those five with cohort data and a stated boundary for cost of revenue is underwritable at a higher multiple than one with a better headline margin and no answer. The number investors underwrite is not the current gross margin. It is the demonstrated rate of change, with the mechanism named.
What to do next
- Write down the cost-of-revenue boundary in one paragraph and apply it to the last four quarters. If the restated margin surprises you, that is the finding.
- Instrument cost per successful task, including retries and escalations, and report it weekly next to revenue.
- Measure your routing, quantization, and cache levers on real traffic before putting them in the plan. Each has a ceiling.
- Show margin by cohort and customer decile, and flag any account with negative contribution before the renewal, not after.
- Stress the model at double the inference cost and half the optimization benefit, then decide whether the business still clears the bar you have set.
Gross margin in AI is an engineering output as much as a pricing decision. The companies that defend a number in front of a board are the ones that can explain, line by line, which parts of the cost base they control and which parts they are renting.
