Add GPU utilization next to $/token before your next GPU term

Your GPU bill has two lines: $/token and GPU utilization. The trade-off is between token price and idle silicon: $/token can look cheap while utilization sits low, and utilization can look high while token price rises. The arithmetic is the gate. Nvidia GB300 NVL72 cut cost per million tokens by up to 35 times versus Hopper at low latency, with the advantage narrowing at higher interactivity and smaller batch sizes. Before you sign the next GPU term, put utilization next to $/token and test the mix that will actually run.

Price idle time before you price tokens

A serving cost model needs a pair, not a single line. The token line is $/token. The utilization line is useful token time over billable GPU time. The cost multiplier is billable GPU time divided by useful token time. A fleet billed for the full clock but serving tokens only part of the time raises the token price even when the vendor quote is unchanged. Idle decode and idle prefill both add to that price. Upper90's financing rationale for the General Compute deal treated continuous inference utilization as more bankable than hardware utilization tied to episodic training schedules. If the vendor quote is fixed, utilization is the only lever you control before the invoice.

The mixed-workload recovery test finds the idle gap

• Define the production mix: prompt lengths, context sizes, concurrency, and latency targets. Keep a table with request class, expected share, and SLO.
• Measure utilization by stage: prefill busy time, decode busy time, and idle time. A dashboard shows which stage is saturated during a token burst.
• Test recovery after a burst: send a mixed workload, spike it, then return to baseline. Tail latency and $/token return to baseline without a manual restart; a vendor demo that runs one request class can hide the idle gap.
• Compare the disaggregated option: when prefill and decode are split across different hardware, check whether the split improves utilization or adds transfer overhead. A side-by-side run uses the same mix and the same SLO.

Nvidia's Dynamo inference framework uses disaggregated serving to split prefill and decode across multiple GPUs, aiming to improve resource utilization and lower token cost. General Compute pairs AMD MI300X cards for compute-intensive prefill with SambaNova SN50 RDUs for decode. In July 2026, a heterogeneous platform with four Nvidia H200 GPUs and 16 SambaNova SN50 RDUs hit 763 tokens per second on MiniMax M2.7 at a 10,000-token context and outperformed GPU-only providers in the same test by several times. Reproduce it.

Benchmark gates keep megakernel claims honest

Flapping Airplanes raised a $1 billion Series A co-led by Index and Lux at a reported $6 billion valuation. The round was announced on October 21, 2026, nine months after launch. Its founders say the megakernel framework can run full training runs, serve inference, and play video games, including all three at once in one kernel. The claim is the test target.

A benchmark gate should compare the claimed workload against your production mix. AMD's performance was competitive when only a subset of inference optimizations was enabled, but not competitive with Nvidia when all three major optimizations were enabled. SambaNova's reported sustained performance above 450 tokens per second at longer context lengths remained company-claimed rather than independently validated until more benchmarks accumulated. Run the same gate on any megakernel claim: same model, same context, same latency target, same mix.

The gate should include cold starts, cache misses, and context growth. A single clean run is not enough. A production mix is messier. For a GPU-focused cloud provider, the gate protects margin; for a platform team, it protects the SLO. Route prod traffic only after the mixed-workload recovery test passes and the benchmark gate matches your production mix.