Hosted Open Models vs Your GPUs: The Cost-Per-Token Test

You are deciding between hosted open-weight inference and self-hosted GPUs. The trade-off is all-in cost per million tokens: hosted list price plus egress, retries, and SLO costs, versus GPU-hours times GPU price plus ops overhead, divided by tokens actually served.

If utilization is low, the denominator shrinks and the per-token number rises. Egress, rate limits, and the GPU index price for the equivalent model decide which number wins.

The spread is the entry fee

A hosted price looks cheap when your fleet is idle. Your GPUs still draw power, need patching, and require on-call coverage when traffic is flat. The vendor's price includes that idle capacity, but only if you pay for it. The spread between list price and your all-in self-hosted cost is the real entry fee.

Start with the model you actually serve. A smaller open-weight model with short context can run on fewer GPUs, but a long-context or high-throughput workload changes the hardware mix. Match the hosted model card to your own serving stack: same weights, same quantization, same maximum context, and the same batching assumptions. If the hosted model is faster because it uses a different pipeline, note that as a performance benefit, not a price match.

Then model the idle share. A GPU that serves tokens only during peak hours still needs a reserved or on-demand allocation, monitoring, and a path to scale. A hosted endpoint can absorb that variability, but the price may include a premium for burst. Compare your idle cost to the vendor's price before you commit.

The checklist is the decision

Run this checklist before you sign a contract or buy another GPU. Each item should take a minute; the last line settles the argument.

• List price per million tokens for prompt and completion, including batch or priority tiers.
• Equivalent GPU index price for the model size and context length, at current on-demand or reserved rates.
• Utilization: expected tokens per GPU-hour and the idle share you would still pay for.
• Reliability/SLO: error rate, p95 latency, uptime, and credit terms.
• Egress and data transfer: per-GB outbound cost, free tier, and private-link options.
• Rate limits: tokens per minute, requests per minute, and burst behavior.
• Total cost per million tokens: add retries, monitoring, and on-call, then compare to hosted price.

If your all-in self-hosted number is lower, buy GPUs. If the hosted number is lower, buy tokens. If they are close, choose the option with better SLOs or faster model swaps.

List price is the easiest item and the least useful by itself. A vendor can quote a low per-token number and still lose to your fleet if your utilization is high and your hardware is already committed. The GPU index price is the counterweight. It should reflect the same model class, not a generic accelerator. A reserved price can look better than on-demand, but it assumes you can keep the fleet busy.

Utilization decides whether self-hosting works. Steady traffic lets your team keep the fleet near capacity, and the per-token cost falls. If traffic is spiky, seasonal, or dominated by long prompts that occupy GPUs for longer, the same hardware produces fewer billable tokens. Hosted inference can be cheaper when you would otherwise pay for capacity you cannot use.

Reliability changes the arithmetic too. A hosted SLO with credits can reduce the cost of a bad month. A self-hosted stack needs fallback, retries, shadow traffic, and on-call coverage. If your product cannot tolerate a slow p95, the reliability overhead may be larger than the GPU savings. Tolerant workloads can accept a cheaper, less polished endpoint.

Egress and rate limits are the overlooked line items. Egress can matter when responses are large, when you ship logs, or when you move data between regions. Rate limits matter when a single customer can consume a large share of tokens. A hosted endpoint with a hard token-per-minute ceiling can force you to buy a higher tier or add a second provider. Plan headroom before you rely on burst.

Market size is context, not a cost model

Fireworks is the useful test case because its revenue and token volume make the arithmetic visible. It announced a $1.505 billion Series D at a $17.5 billion valuation on July 16, 2026. The round was led by Atreides Management, Index Ventures, and TCV, with participation from Evantic, Lightspeed Venture Partners, and NVIDIA. Fireworks has crossed the $1 billion mark in annualized revenue run rate, a fivefold jump from its prior funding round, while daily token volume climbed from 15 trillion to above 40 trillion during the same period. 95% of the tokens it serves come from specialized models.

Together AI raised an $800 million Series C on July 1, 2026 at an $8.3 billion valuation, with reported annual bookings around $1.15 billion. Fireworks plans to use the new funding to deepen partnerships with cloud partners such as Microsoft and NVIDIA.

The market is large enough that hosted open-model serving can be a serious alternative. Put the lower all-in number in the budget.