TPUv7 vs Blackwell: Decide LLM Serving Capacity Numerically

A five-step test decides the purchase

You are choosing LLM serving capacity and TPUv7 Ironwood looks cheaper on paper. The trade-off is cost versus latency, concurrency, procurement, and model assumptions. At 100 tokens/s/user, TPUv7 Ironwood is modeled at about $0.181 per million total tokens, B200 at $0.222, and B300 at $0.276. At concurrency 256, its mean time-to-first-token is 5.41 seconds, versus 3.75 seconds for B200 and 2.40 seconds for B300. The decision is a checklist: target tokens/s/user, cost per million tokens, TTFT at concurrency, externalization/procurement risk, and open-source model assumptions. Use it before you commit capacity.

• Target tokens/s/user. Pick the sustained rate, peak rate, and degradation rule your product must hold. Start from the user experience, then back into the serving budget. A chat surface, a code assistant, and a batch summarizer need different tolerances for slow output. The target survives when you can name the high-percentile request length, the expected burst, and the time window where the product must not feel broken. If the target only works in a demo, it is not a serving target.
• Cost per million tokens. SemiAnalysis released initial third-party inference benchmarks for TPUv7 Ironwood through InferenceX Official Preview. The benchmark set reports TPUv7 Ironwood at up to 50% better performance per dollar than B200 and B300 in comparable tests. Do not compare a modeled token price to a list price. What matters is the cost of your workload at your target rate. Convert the modeled price to your model mix, context length, batch size, idle capacity, queueing, and the model version you will actually serve before you treat it as a purchase price. If traffic is spiky, the average token price can hide a peak that forces extra capacity.
• TTFT at concurrency. MLPerf data, cited by EDN, shows roughly 138 tokens per second per accelerator for Llama 3.1 405B on Nvidia's GB200 NVL72. A token price only matters if the first token arrives before the user abandons the request.
• Externalization/procurement risk. Inference externalization can move the decision from a chip comparison to a vendor, contract, and data-flow comparison. Ask who owns the serving stack, where prompts and completions go, and what happens when capacity is reserved, overcommitted, or renegotiated. Put each risk in a table and assign a dollar penalty to the failure mode. A dollar penalty makes the risk comparable to the token price. Keep an exit path that lets you move workloads, change models, or split traffic without a rewrite.
• Open-source model assumptions. The cheapest token may come from a model change, not a hardware change. Four inference providers reported 4x to 10x cost-per-token reductions using Blackwell with open-source models. Sully.ai reduced healthcare AI inference costs by 90% and improved response times by 65% after moving to open-source models on Baseten's Blackwell platform. Do not borrow that discount if your quality bar depends on a proprietary model, a fine-tuned model, or a tool-calling behavior that has not been validated on open weights.

Latency can kill the cheaper token

Turn the checklist into a capacity plan. Use the target user load, not the demo load, for the accelerator count, then check whether the first-token budget still holds when the queue is deep. A system can look cheap on average and still fail the moment traffic spikes. Deliver a latency budget and an accelerator count that both hold at peak load. If TPUv7 misses latency or concurrency, the cheaper token is not a production win.

Procurement and model assumptions decide the exit

If TPUv7 holds the latency and concurrency targets, the remaining question is whether the serving stack can survive a model change, a provider change, or a traffic shift. A cheap token is less attractive when the exit path depends on a single contract, a single region, or a single model format. Run the hardware comparison only after you have validated the model change on your own workload.