The first AI cloud bill is a shock, and the second one is worse because nobody can explain it. Compute appears under a line item that also carries ordinary application servers. The team that built the feature cannot see which workload drove the increase, and finance cannot tell whether the money bought revenue or idle capacity. FinOps for AI starts with that gap between the invoice and the decision.
The discipline is the same one that tamed general cloud spend: allocate cost to the thing that created it, show the owners what they are spending, and then reduce the waste. What changes with AI is the shape of the cost. Accelerators are expensive per hour, shared by many workloads, and often idle while still billing. Tokens are metered by providers that give you a number but not a unit cost. That combination breaks tagging habits built for virtual machines.
Why AI spend is invisible in the cloud bill
Three things hide it. First, GPU capacity is usually provisioned in pools and shared across teams, so a tag on the node does not identify the workload that consumed it. Second, a large share of the spend is not compute at all: it is managed model APIs, vector storage, evaluation runs, logging, and observability, each with its own meter. Third, the most useful unit of cost, a request or a token, does not appear on the invoice. You get instance hours and API line items, not a price per answer.
The consequence is that cost conversations happen at the wrong altitude. Someone points at a number that grew, nobody can attribute it, and the default response is a hiring freeze on experiments. Attribution is what turns that into a targeted decision.
Tagging: agree on a small schema and enforce it
Most organizations already have a tagging standard. It usually has more keys than anyone can remember, and a compliance rate that nobody wants to measure. For AI workloads, pick seven keys and make them mandatory at provisioning time, not at review time: business unit, product, environment, workload or model, owner, cost center, and data classification. Those seven answer who, what, where, and which system, and they map cleanly onto provider-native cost allocation tags.
Then handle the part that tagging cannot solve: shared capacity. A tag on a GPU node tells you which team owns the node. It cannot tell you which of the four workloads on that node consumed most of the hours. You need a second mechanism, usually a cost allocation model that consumes telemetry rather than tags. Decide the rule in advance and write it down, because the argument about the rule is always cheaper than the argument about the invoice.
Enforce with automation. Reject untagged resources at creation through policy, and add a monthly exception report for the ones that slip through. A tag that is optional is a tag that will be missing on exactly the workloads that matter.
Allocate shared GPU capacity to workloads
The allocation model needs three layers. At the bottom, the invoice: hourly accelerator cost per instance type, including commitments and discounts. In the middle, the cluster: which nodes are shared, which are dedicated, and how much headroom is held for failover. At the top, the workload: requests served, tokens processed, batch jobs completed, training hours consumed.
Convert the middle layer to dollars per GPU-hour, then distribute by measured consumption. For inference, a defensible weight is tokens processed per workload, adjusted for the model and context length, because a long-context request costs far more than a short one. For training, weight by GPU-seconds. For shared services such as embeddings or reranking, weight by requests. Where telemetry is missing, use reserved fractions and flag the workload as approximated rather than pretending the number is exact.
Two allocation choices matter more than the rest. First, whether idle capacity is spread across all consumers or charged to the team holding the reservation. Spreading it hides the problem; charging it changes behavior. Second, whether discounts are redistributed evenly or passed through to the workload that earned them. Pass them through, and the team that commits to a two-year reservation sees the benefit.
Showback earns the right to chargeback
Showback means reporting cost to owners without moving money. Chargeback means putting it on their budget. Most organizations should spend two quarters on showback before attempting chargeback, because allocation models are always wrong in the first draft and a wrong chargeback creates a political fight that sets the program back a year.
Use showback to test the model. Publish a monthly view that reconciles to the finance ledger within a stated tolerance, and ask each owner whether the number looks like their world. When three consecutive months pass without a serious dispute, move the largest and most predictable workloads to chargeback first. Keep a documented dispute process with a named owner and a two-week resolution window. The process matters more than the precision.
Idle detection is the fastest savings
Accelerators are the most expensive idle asset most companies own. Four kinds of idle are common. Provisioned capacity with low utilization, where a pool runs at a fraction of its capability around the clock. Reservation mismatch, where committed capacity sits unused while on-demand instances run beside it. Over-replication, where each service holds a full replica for peak load that arrives twice a week. And abandoned experiments, where notebooks and fine-tuning jobs keep a GPU allocated after the work stopped.
Measure utilization two ways: the hardware gauge and the business gauge. The hardware gauge comes from the accelerator exporter your monitoring stack already has, typically a utilization percentage per device, sampled every few seconds. The business gauge is tokens or requests per GPU-hour. A pool can show high hardware utilization while producing very few tokens, which usually means small batches or an inefficient serving configuration, and that is a cost problem the hardware gauge cannot see.
Report idle capacity as a dollar figure per week, not a percentage. Then set a target that reflects reality: some headroom is required for failover and burst, so the aim is not 100% utilization. A realistic first target is to cut idle capacity in half within a quarter, through scheduling, right-sizing replicas, and moving batch work onto the same pool as online traffic.
Unit economics: cost per request, not cost per token
Blended cost per token is a useful supplier metric and a poor management metric. It hides the tail. Report at least four units: cost per 1,000 requests, cost per 1 million input and output tokens, cost per active user per month, and cost per successful outcome where you can define one. Then look at the distribution rather than the average, because a small share of long-context or agentic requests usually drives a large share of spend.
Segment by product, by model, and by customer tier. A single customer running agent loops can consume more than a hundred small accounts, and the average will never show it. Once you have cost per request by segment, pricing and product decisions get easier: you can see which tier is unprofitable, which feature is expensive relative to its value, and which optimization is worth the engineering time.
A cost model with labeled assumptions
The table below is an illustrative monthly model, not a benchmark. Assumptions: 32 accelerators at an effective $2.40 per GPU-hour including commitments, 730 hours in the month, 45% average utilization, 9 million requests, 1,500 tokens per request on average, 20,000 monthly active users, and 18 TB of egress at $0.09 per GB. All figures are rounded.
| Line item | Monthly cost | Share | Note |
|---|---|---|---|
| Accelerator hours | $56,064 | 75% | 32 devices, 730 hours, assumed rate |
| of which attributable | $25,229 | 34% | At assumed 45% utilization |
| of which idle | $30,835 | 41% | Paid for and not used |
| CPU and memory | $9,400 | 13% | Retrieval, orchestration, gateways |
| Storage | $4,200 | 6% | Models, embeddings, logs |
| Egress | $1,620 | 2% | 18 TB at assumed $0.09 per GB |
| Observability and evals | $3,300 | 4% | Tracing, evaluation runs, dashboards |
| Total | $74,584 | 100% | Illustrative only |
The derived units are the point of the exercise. Under these assumptions the total works out to about $8.29 per 1,000 requests, $0.0083 per request, $5.52 per million tokens, and $3.73 per active user per month. The idle line is 41% of total spend, which is where an optimization program should start. If you cannot produce these four numbers for your own system this month, that is the first deliverable, ahead of any discount negotiation.
What to do next
- Turn on native cost allocation tags in your cloud provider and add the seven mandatory keys through policy, not documentation.
- Pick one shared GPU pool, compute dollars per GPU-hour, and publish a first allocation by tokens processed. Accept that the first version is approximate.
- Report idle accelerator spend weekly as a dollar figure with a named owner for the largest pool.
- Publish monthly showback for three months, reconcile to the finance ledger, and only then move the two largest workloads to chargeback.
- Put cost per 1,000 requests and its 95th percentile on the same dashboard as latency, reviewed in the same meeting.
Cost per request is the unit that survives contact with finance, product, and engineering. Build the model around it, label your assumptions, and improve the assumptions every month.
