Size inference GPUs by tokens per second per dollar, not by model size;

The trade-off is simple: when capacity exceeds what a workload needs, utilization drops and cost per token rises.

Treat GPU sizing as mapping the use case to a GPU footprint based on real workload behavior rather than guesswork. The device choice follows that distribution, not the opposite. The footprint should be compared with the real token-length and cache-hit profile that the workload actually uses.

Costs

A higher cache hit rate can reduce TTFT, cost per request, and the GPU capacity needed for the same traffic. This keeps the capacity decision a cost decision, not guesswork.

Start with the five-number worksheet. The inputs are input/output tokens, cache hit rate, batch size, quantization level, and owned or rented cost. If your traffic reuses repeated system prompts, document prefixes, or multi-turn context, put cache hit rate in the worksheet before you choose a GPU, not after the first latency incident. The outputs are tokens per second per GPU, dollars per token, memory headroom, and breakeven utilization. Memory headroom is the gap between available memory and the memory estimate. Keep the worksheet in the same units as your billing system, because the number that matters to finance is not raw throughput; it is the inference cost of a user-visible token. Compute tokens per second per GPU from your actual batch shape. A useful form is batch size multiplied by output tokens per request, divided by the time to complete that batch. Do not use a single request at a time if production will batch. Do not use a synthetic prompt if production prompts are longer. Longer input and output token lengths increase GPU memory and compute demand. Dollars per token is GPU cost per hour divided by tokens per GPU-hour. If you rent, use the hourly rate for the exact instance class. If you own, use the owned cost of the device. The worksheet should show the break point where reserved or owned capacity beats on-demand, and the utilization level where the extra capacity starts paying for itself. Breakeven utilization is the share of capacity that must be busy for owned or reserved cost to beat a smaller or on-demand footprint. Once those formulas are in place, the right inference GPU is the cheapest device that keeps latency, memory headroom, and cost per token inside your token-length and cache-hit profile. That keeps the device choice tied to the arithmetic instead of to model size. The worksheet should be rerun when the token-length and cache-hit profile changes, and the breakeven utilization should be rechecked.

Models

The model choice is a cost lever, not just a quality lever. Quantization, pruning, and distillation can improve performance while lowering TCO. The trade is worth considering when the performance and cost effects are favorable. If the smaller model keeps the product requirement, a distilled model may be enough. If the quality loss is unacceptable, the trade is not worth it.

Do not size the GPU from the model parameter count. Include weights, KV cache, activations, and batch overhead in the memory estimate. Long context and high concurrency can change the memory estimate enough to flip the device choice. The worksheet forces you to compare the actual serving state, not the model card.

Quantization also changes the memory arithmetic. If you are choosing between a higher-precision model and a lower-precision model, run both through the same token-length and cache-hit profile. The winner is the one that keeps latency and memory headroom inside the product requirement at the lowest dollars per token.

Serving

For steady-state traffic, the capacity rule is boring and correct. Establish a baseline of on-prem or reserved cloud GPU capacity for steady-state workloads. That can help align capacity with steady-state workloads. If your traffic has peaks, do not buy the peak as a permanent baseline. The baseline should be tied to the real token-length and cache-hit profile, and the utilization target should be rechecked.

Utilization is the number that tells you whether the serving design is honest. If the GPU is mostly idle, the cost per token is inflated by the fixed cost of the device. If the GPU is pinned at the edge, latency will degrade before you notice the cost problem. The target is not maximum utilization. The target is the utilization that keeps latency inside the product requirement while leaving enough memory headroom for cache and batch growth.

Run the worksheet in three states: normal, peak, and degraded. Normal is the traffic you expect most of the time. Peak is the traffic you must survive without dropping quality. Degraded is the state where available capacity is reduced or model performance is slower. The output should be a range, not a single SKU. If the range is wide, your workload is not stable enough for a fixed footprint, and you need a serving policy that can move requests between device classes.

The final check is simple: the worksheet output is the decision rule. If the worksheet cannot serve the real token-length and cache-hit profile at the target latency with enough headroom to absorb growth, the GPU is too small and will show latency and memory headroom problems. If the worksheet shows you are paying for capacity the workload does not use, the GPU is too large and will show idle silicon and a higher cost per token. The right size is the one where the worksheet says you can serve the real token-length and cache-hit profile at the target latency, with enough headroom to absorb growth, and without paying for capacity the workload does not use.