Open vs Closed Models: The Inference Economics That Actually Matter
Closed models bill per token with zero infrastructure work, while open models trade a fixed GPU cost for control and lower marginal pricing at scale. This guide breaks down where each wins.
The choice between an open-weight model you host and a closed model you call through an API is often framed as a question of capability or philosophy. In practice, for most production teams it comes down to economics, specifically who pays for the GPUs and when. Closed models fold all infrastructure cost into a clean per-token price. Open models hand you the weights and the bill for running them. Neither is universally cheaper. The right answer depends on your volume, your utilization, and how much operational work you are willing to own.
Two cost structures
A closed model has a simple variable cost: you pay per token of input and output, and the provider handles every GPU, scaling decision, and upgrade behind the scenes. Your cost rises and falls exactly with usage, and at zero usage you pay nothing. There is no idle waste, but there is a markup baked into every token to cover the provider's infrastructure and margin.
An open model that you host has a largely fixed cost: you rent or own GPUs by the hour or the term, and you pay for them whether they are busy or idle. The marginal cost of an extra token is close to zero once the hardware is running, but a lightly used cluster is expensive per useful token. You also absorb the operational work of deployment, scaling, monitoring, and updates.
| Dimension | Closed API model | Open self-hosted model |
|---|---|---|
| Cost shape | Variable, per token | Mostly fixed, per GPU hour |
| Cost at low volume | Low | High per useful token |
| Cost at high steady volume | Higher marginal | Lower marginal |
| Operational burden | Minimal | Significant |
| Control and customization | Limited | Full |
The breakeven volume
The decision pivots on a breakeven point. Below a certain steady volume, the per-token API is cheaper because you avoid paying for idle GPUs and operational overhead. Above it, self-hosting wins because the fixed GPU cost, spread across enough tokens, undercuts the API's marginal price. To estimate your breakeven, divide the monthly cost of a GPU deployment sized for your peak by the per-token price difference. If your projected steady volume clears that line with margin, hosting can pay off. If your volume is low, spiky, or uncertain, the API almost always wins.
Utilization is the hidden variable
A self-hosted deployment is only cheap if the GPUs stay busy. A cluster running at twenty percent utilization can cost more per useful token than the API it was meant to replace. The teams that win with open models keep utilization high through batching, autoscaling, and consolidating workloads onto shared endpoints. If your traffic cannot sustain high utilization, the fixed cost works against you.
Beyond the token price
Token cost is the headline, but total cost of ownership includes more.
- Engineering time: deploying, tuning, and maintaining inference infrastructure is real, recurring work.
- Reliability: the API provider owns uptime; self-hosting makes it your problem.
- Upgrades: closed providers ship new model versions for you, while self-hosting means you manage migrations.
- Egress and storage: moving data and storing weights and caches add line items.
- Customization value: open models allow fine-tuning, quantization, and full control of versioning, which can be worth more than the raw token savings.
When each one wins
Closed models are the right default when volume is low or unpredictable, when you want zero operational overhead, when you need frontier capability you cannot host, or when time to market matters more than marginal cost. Open models become compelling when volume is high and steady, when you can keep utilization high, when data residency or customization requirements rule out external APIs, or when you have the engineering capacity to run inference well.
- Estimate your steady monthly token volume and how predictable it is.
- Price the API path at that volume.
- Price a self-hosted path including GPUs, engineering, reliability, and egress.
- Compare at realistic utilization, not peak.
- Add the value of control and customization to the open side if it applies.
The middle path: managed open models
The choice is not strictly build versus buy. A growing option is a managed endpoint for an open-weight model, where a provider hosts the open model for you and bills per token, much like a closed API. This blends the economics of both worlds: you get the lower licensing and customization profile of an open model without owning the GPUs or the operational burden. For teams that want an open model's flexibility but lack the volume or the staff to host it efficiently, a managed open endpoint can be the cheapest option at moderate scale, and it leaves an easy path to self-hosting later if volume grows enough to justify it.
How the decision evolves over time
Most products do not pick once and stay forever. Early on, when volume is low and the roadmap is uncertain, a closed or managed API minimizes risk and time to market. As a product matures and usage becomes large and predictable, the fixed-cost economics of self-hosting an open model start to win, and the savings can be substantial at scale. The smart approach is to design your application so the model behind it can be swapped without a rewrite, abstracting the inference call behind a clean interface. That way you can ride the API while small, move to managed open models as you grow, and self-host when the numbers clearly favor it, all without re-architecting each time.
Common miscalculations to avoid
Several recurring errors distort open-versus-closed comparisons. The first is comparing the API token price against only the GPU rental cost of self-hosting, ignoring the engineering, reliability, and upgrade work that self-hosting demands. The second is assuming peak utilization rather than realistic average utilization, which makes self-hosting look far cheaper than it is in practice. The third is overlooking egress and storage, which are small per request but add up at scale. The fourth is undervaluing the API provider's continuous model improvements, since a closed provider effectively upgrades your product for free whenever it ships a better model, whereas self-hosting means you own every migration. Listing all of these line items before deciding is the only way to compare like with like, and doing so frequently reverses a conclusion that looked obvious from token price alone.
Conclusion
The open versus closed question is really a question about cost structure: variable per-token pricing with no operational burden, or a mostly fixed GPU bill with low marginal cost and full control. Low or spiky volume favors closed APIs, because you never pay for idle hardware. High, steady, well-utilized volume favors open self-hosting, because the fixed cost spread across many tokens beats the API markup. The honest comparison is total cost of ownership at realistic utilization, not the sticker token price, and the breakeven volume is the number that should drive the decision.