Turn MLPerf v6.1 into a cost-per-token model for your serving stack

MLPerf v6.1 gives the procurement arithmetic

You are choosing GPUs, serving software, and capacity for LLM, RAG, and agent workloads. The trade-off is raw tokens per second versus cost per useful token: idle GPUs, failed requests, and weak serving software all land on the bill. Use cost per useful token = (GPU-hours x $/hour + outage cost) / useful tokens. MLPerf Inference v6.1 had 30 submitting organizations and 486 datacenter and edge results. The best per-accelerator DeepSeek R1 server result was 5.7 times the v5.1 result from a year earlier. The result set gives a common profile and separates software progress from silicon progress.

v6.1 added End-to-End RAG and Agentic Edge Inference, so RAG and agent traffic now have measured profiles. GPT-OSS-120B was the most-submitted workload, with 112 submissions. Use it as the baseline when your traffic is close to that profile; otherwise, find the nearest scenario and note the gap.

Five steps turn throughput into cost per useful token

• Match the workload profile. Pick the MLPerf scenario that mirrors your traffic: server for interactive requests, offline for batch, retrieval-plus-generation for RAG, or agent calls for agent workloads. Write down prompt length, completion length, concurrency, and timeout. Record the request mix, not just average tokens. Stop when you can name the scenario and the token budget per request.
• Normalize tokens per second. Put every result on a single accelerator, a single model, a single scenario, and a single software version. Convert batch throughput to per-request latency, then to tokens per second per accelerator. Keep the software version visible in the table. Leave one table with no hidden batch size.
• Apply utilization and outage cost. Multiply GPU-hours by your dollar rate, then add the cost of failed or delayed requests. Include idle time between requests, cold starts, and retries. Treat retries as paid work, not free noise. Do not use peak offline throughput for interactive capacity; it overstates usable tokens. Use your real SLO, not the benchmark's target. The number should be cost per useful token, not cost per published token.
• Compare software-only gains. Check whether the same hardware gets faster with a newer serving stack. Intel's Arc Pro B70 four-GPU node improved GPT-OSS-120B by 36 percent in the server scenario and 27 percent in the offline scenario versus v6.0 on identical hardware, driven by software optimizations. Run the same model and scenario on your current stack, then on the candidate stack. Record a dollar delta before any new purchase. If the delta is small, keep the current stack.
• Set capacity. Size from the largest credible system, not from a single node. Crusoe's 512-GPU AMD Instinct MI355X cluster was the largest system submitted, and the cluster reached 5.75 million tokens per second for GPT-OSS-120B offline. Apply that scale to check whether your target is a node problem or a fleet problem. Headroom should cover growth and failure, not just peak. End with a GPU count, a dollar budget, and a headroom number.

Use the hardware delta to choose stack or silicon

NVIDIA's Vera Rubin NVL72 reached up to 3.7 times the Qwen3-VL throughput of GB300 NVL72 using vLLM with Dynamo. Put that number next to the dollar delta from the software-gain step. Migration effort includes drivers, serving config, and rollback. If the hardware gain does not survive utilization, outage cost, and migration effort, the cheaper move is a stack upgrade. Otherwise, the hardware line item becomes the main cost driver.

Keep one row per candidate: model, scenario, accelerator, serving stack, tokens per second per accelerator, GPU-hours, dollar rate, utilization, outage cost, and cost per useful token. Add a row for the current stack so the comparison is explicit. The table should answer three questions: how many GPUs, what the monthly cost is, and what happens when traffic grows. When stacks land within a small band, choose the one with the lower operational risk; drop a clearly worse stack early.