The question arrives as a request for a recommendation and is usually a request for permission. A CTO wants to know whether to keep paying a closed API or run open weights, and the honest answer is that both options fail in predictable ways. Closed APIs fail on control: price changes, deprecations, region limits, and terms that can move under you. Open weights fail on operations: GPUs you now own, a serving stack you now maintain, and a model that nobody upgrades for you.
What follows is a framework rather than a verdict, because the right answer depends on four constraints that differ by company: where the data may go, how much customization the task needs, what the compliance file must contain, and whether your volume is large enough to amortize a fixed cost.
Two failure modes, not two camps
The closed-API failure mode is dependency that compounds. A prompt tuned for one model does not transfer cleanly to another, so switching costs rise with every month of tuning work. A deprecation notice with a few months of runway becomes an unplanned migration. A price change lands as a margin event you did not model. Every one of those is survivable, and none of them is on the roadmap.
The open-weights failure mode is a capability gap that nobody owns. The model is genuinely yours, which means the evaluation suite, the serving stack, the autoscaling policy, the security review, and the upgrade path are also yours. Teams underestimate the last one most: a better open model appears every few months, and each one requires re-tuning prompts, re-running evals, and re-certifying output quality. Running weights is a product commitment, not a procurement decision.
The law and the license are not the same thing
"Open" describes weights, not rights. The Open Source Initiative published version 1.0 of its Open Source AI Definition in late 2024 precisely to draw that line, requiring the freedoms to use, study, modify, and share the system, including data information sufficient to rebuild it. Most downloadable model weights do not meet that bar, because the training data remains undisclosed and the license restricts some uses.
The differences matter commercially. Mistral's 7B models ship under Apache 2.0, a permissive license that allows commercial use, modification, and redistribution with attribution and no field-of-use restriction. Meta's Llama 3.1 license is a custom community license: it adds an acceptable use policy, requires attribution and naming rules for derivatives, and sets a threshold at 700 million monthly active users above which a separate license from Meta is required.
For a US enterprise, the practical difference is indemnity and audit. Commercial API contracts typically include some intellectual-property indemnity for the service's output. A weight license provides no indemnity, and the compliance team will ask who is responsible when the model reproduces something it should not. Have an answer before the review, not during it.
Self-hosting TCO, line by line
Most self-hosting business cases compare GPU hours to API spend and stop. The full cost has six lines, and the first two are fixed regardless of traffic.
| Cost line | What it looks like | Common mistake |
|---|---|---|
| GPU capacity | Committed or purchased accelerator nodes, sized for peak plus N+1 | Sizing for average load, then paying for peak anyway |
| Depreciation | Three-year life for GPUs, faster if generation jumps | Assuming resale value |
| Platform labor | One to two engineers on serving, evals, upgrades, on-call | Borrowing time from the product team permanently |
| Utilization gap | Idle capacity at night and on weekends | Forgetting that idle GPUs still bill |
| Model refresh | Re-evaluation and prompt migration every few months | Treating the first deployment as the last |
| Security review | Weight storage, access control, data governance for tuning | Treating weights as code artifacts |
Here is the arithmetic that changes minds. Assume an eight-GPU node that costs $25,000 a month to own or reserve, and that serves a 70B-class model at 8-bit precision at 2,000 output tokens per second sustained. That is about 5.2 billion output tokens a month at full utilization, so the GPU cost alone is roughly $4.80 per million output tokens. At 40% utilization the same node produces 2.1 billion tokens, and the cost doubles to about $12 per million. Add one and a half engineers at $220,000 fully loaded, or $27,500 a month, and the fixed floor is $52,500 a month before a single token ships.
The conclusion is uncomfortable for both camps. At moderate volume with a single workload, self-hosting a large model usually loses to API pricing on cost alone. It wins when the GPUs are filled with several workloads, when a small model can do the job, when data cannot leave your network, or when the API price is high enough that the crossover sits inside your forecast horizon. Run that arithmetic with your own numbers before choosing a side.
The four constraints that decide it
Data. Not "is the data sensitive" but which data may cross a boundary. Health records and payment data usually cannot. Internal documentation often can, under a contract with retention limits and no training on your data. Sovereignty arguments get stronger when the workload includes third-party data whose terms you did not write.
Customization. Prompting, retrieval, and tool design solve most quality problems. Fine-tuning matters when the task needs a format, dialect, or classification boundary that prompts cannot reliably hold. Fine-tuning a closed model is possible at most vendors and often sufficient, which narrows the open-weights case considerably.
Compliance. The EU AI Act imposes documentation and transparency duties on general-purpose model providers, with extra obligations for models judged to pose systemic risk. A vendor's compliance package is a document you can hand to auditors. Running your own weights converts those duties into your responsibilities, and the NIST AI Risk Management Framework gives you the vocabulary to document them but not the staff to do it.
Volume and shape. Steady, high-volume, latency-tolerant traffic favors owned capacity. Spiky, low-volume, latency-sensitive traffic favors APIs, because burst capacity is the thing you cannot buy efficiently.
Ordered this way, the constraints simplify the argument. Data can disqualify an option outright. Customization is the most common reason teams overestimate their need for ownership. Compliance adds documentable work to whichever side you choose. Volume decides cost rather than feasibility. Teams that start with volume spend months negotiating price when the real obstacle was a data boundary that nobody had written down.
A decision matrix for six situations
| Situation | Default choice | Reason |
|---|---|---|
| Regulated payer or hospital, PHI in prompts | Open weights or private deployment | Data boundary is non-negotiable; pay the operations premium |
| Consumer app, 50,000 users, unpredictable spikes | Closed API | Burst capacity at zero fixed cost; tuning risk is low |
| Support automation at millions of tickets a year | Hybrid: closed API for hard cases, small open model for the bulk | Volume amortizes the small model; escalation protects quality |
| Air-gapped or defense environment | Open weights, on-premises | No network path exists |
| Domain classifier with a fixed label set | Fine-tuned small open model | Cheap, fast, and evaluable against a labeled test set |
| Product experiment with unclear retention | Closed API, month to month | Do not buy infrastructure for a hypothesis |
A 90-day test that settles the argument
- Instrument the current workload: tokens per task, turns per task, peak requests per second, and the share of tasks that need the frontier model.
- Build a 200-task evaluation set from real failures with a pass criterion agreed by product and engineering.
- Deploy the best open candidate behind the same interface for two weeks of shadow traffic, and score both models on the same set.
- Fill in the six-line TCO with your actual utilization, including the engineers you would have to hire or reassign.
- Compare against a negotiated API contract, not the list price, and include the exit terms you would need if the vendor changes course.
What to do next
- Write down which data absolutely cannot leave, then check whether the rest needs the same treatment. Blanket sovereignty rules are expensive and rarely justified.
- Price the operations line honestly: one engineer-year is usually larger than the GPU bill for a modest deployment.
- Keep an abstraction layer at the model boundary so that prompts, evals, and routing survive a change of vendor or a change of weights.
- Re-run the decision annually. Both sides of the comparison move fast enough that a two-year-old decision is stale.