Recalculate your 70B $/token: what the B200 price rise means for self-hosted inference

A $/token sheet built on last quarter's GPU-hour price is stale. Self-hosting beats rented inference only when sustained throughput and utilization are high enough to cover idle capacity, reliability overhead, and ops. The arithmetic is current GPU-hour price divided by sustained tokens per hour per GPU, adjusted for utilization. Start from sustained tokens per second, convert it to an hourly token rate, then apply utilization.

The B200 and H100 lines moved up

The OCPI-B200 index for NVIDIA B200 rental compute settled at $6.82 per GPU-hour on September 9, 2026, equivalent to $163.79 per GPU-day and about $4,913.70 per GPU-month. The index was up 5.6% over the seven days and 20.2% over the thirty days before September 9, 2026, from $5.68 per GPU-hour. The Ornn Compute Price Index for H100 SXM rental compute settled at $2.96 per GPU-hour on September 9, 2026, equivalent to $71.02 per GPU-day and about $2,130.60 per GPU-month. SemiAnalysis's H100 rental composite bottomed near $2.79 to $2.80 per GPU-hour in mid-to-late 2025 before reversing upward. By April 2026, the H100 spot-contract composite was $2.82 per GPU-hour.

Utilization is the hidden multiplier

A GPU-hour is a fixed cost. High sustained output per GPU at high utilization makes the denominator large and the $/token fall. Low throughput or idle fleet makes the denominator shrink and the $/token climb. The formula does not care about the model name; it cares about sustained output under your prompt mix, batch size, quantization, and routing.

Add capacity and reliability overhead before comparing to rented inference. Peak headroom, failover, maintenance, and on-call work can consume GPU-hours that do not produce billable tokens. That overhead raises the effective GPU-hour price, so a self-hosted stack needs a larger utilization cushion than a bare throughput test suggests.

The operating point matters. A larger batch can raise tokens/sec per GPU, but it can also push latency past your service level. A more aggressive quantization can raise throughput, but it can change output quality. The right point is the one that meets your service level while keeping the denominator high.

• Current GPU-hour price for the exact SKU and contract term.
• Sustained tokens/sec per GPU under your prompt mix, batch size, and quantization.
• Utilization after subtracting idle time, maintenance, and failover.
• Capacity overhead for peak demand and reliability.
• Quality constraints that limit quantization or batch size.
• Latency service levels that cap batch size and raise per-token cost.

Spiky traffic lowers utilization. Steady traffic raises utilization. The same hardware can be a bargain or a trap.

Supply and demand explain the direction

The upward price move was linked to Blackwell supply constraints from CoWoS packaging and HBM capacity, together with inference demand exceeding training demand. TrendForce projected the CoWoS supply-demand gap to narrow from about 20% to 10% by the end of 2026, with TSMC's monthly CoWoS capacity approaching 120,000 to 140,000 wafers. Gartner's August 10, 2026 forecast assigned 55% of 2026 AI-optimized IaaS spending to inference and 45% to training, with inference cloud spend at $23.3 billion and training at $19 billion inside a category expected to grow 96% to $42 billion.

The direction matters more than the level. Rising prices make a last-quarter GPU-hour price an old input. Easing supply may let that input pay. Demand outpacing supply may not.

The decision is a current-price test

Rebuild the sheet with the current GPU-hour price, not the last benchmark. Use the exact SKU and contract term. Measure sustained tokens per second per GPU under production traffic, not a clean benchmark. Convert that rate to an hourly token rate. Subtract idle time, maintenance, and failover to get utilization. Add capacity overhead for peaks and reliability.

Check the rented price against the same operating point. Compare rented prices that include batching and autoscaling against your self-hosted cost after those overheads. Convert per-token rented prices to a GPU-hour equivalent using your expected traffic.

With those inputs, you can judge whether self-hosting beats rented inference. Self-host when the current GPU-hour price divided by your sustained throughput and utilization is lower than the rented price after overheads. Rent when it is not.