Pick the Cheapest LLM Stack That Meets Your p95 Latency

The trade-off is arithmetic, not taste: cost per token falls as tokens per GPU-second rise, while p95 latency rises as queueing, batch contention, and parallelism overhead grow. Write the constraint as cost = GPU-seconds per token × $/GPU-second, and the SLA as p95 = 95th percentile of request completion time. The cheapest compliant LLM inference stack is the point where your quality floor, p95 latency SLA, and $/token ceiling intersect. Inference efficient frontiers are most often expressed as a latency-throughput tradeoff, with cost determined by throughput.

That is why cheapest model is the wrong question. A smaller model can look cheaper on paper and still fail your latency or quality floor. A larger model can be more expensive per token at low batch size and become cheaper at high throughput if serving can be tuned correctly. The goal is not to minimize model parameters. The goal is to minimize $/token while staying above the quality floor and inside the latency SLA.

Costs

Start with cost optimization, not the model card. Define three numbers before you benchmark: the quality floor, the p95 latency SLA, and the $/token ceiling. The quality floor should be a measurable gate, not a vibe: task accuracy, judge score, regression suite, or business KPI. The latency SLA should be p95, not average, because users feel the tail. The cost ceiling should be expressed per token, not per request, because prompt length and output length vary.

Then make the cost equation explicit. If a serving configuration generates fewer tokens per GPU-second, its $/token rises even if the model is small. If it generates more tokens per GPU-second, its $/token falls even if the model is larger. Small batch sizes give excellent per-user latency but high cost per token, while larger batch sizes worsen latency but improve throughput and lower cost. That is the core cost lever: batch size moves you along the frontier. If your p95 SLA has slack, increase batch size until latency hits the limit. If your cost ceiling is tight and latency is loose, let the system batch harder. If both are tight, you need a different point on the frontier, not a harder squeeze on one knob.

Models

Model choice is a quality-latency-cost point, not a brand preference. Sweep model size, but do not stop there. The same model can occupy several points on the frontier depending on precision, context length, and serving configuration. A frontier plot should have three axes: quality, p95 latency, and $/token. If you only plot quality versus model size, you will miss the cheapest compliant stack. If you only plot latency versus cost, you may choose a model that fails your quality floor.

Quantization is the first model-side lever to test. Quantization improves both latency and throughput, and can yield large serving-efficiency gains with little-to-no quality reduction, especially with MXFP4 and NVFP4. In practice, test the quality floor after quantization. The frontier can be jagged: one precision may be nearly free, while another may collapse a task you care about. Do not assume a lower bit count is automatically better. Run the same evaluation set at each precision, then mark the lowest $/token point that still clears the floor. If the quality drop is small and the latency or cost gain is large, quantization can move the whole stack left and down on the plot.

Serving

Serving is where the frontier becomes operational. The model is a point; the serving stack is the path around it. You need to sweep batch size, parallelism, and traffic shape. The same model can be too slow at low batch, too expensive at high batch, or too tail-heavy under mixed traffic. The serving configuration should be chosen after the quality floor is fixed, because serving changes latency and cost more than it changes raw model quality.

Batch size is the first serving knob. It trades per-user latency for throughput. If your p95 SLA is loose, batch harder. If your p95 SLA is tight, batch less or change the parallelism strategy. For latency-sensitive deployments, focus on increasing Tensor Parallelism (TP). Use TP when the tail is the problem and you can afford the footprint. Use batch size when the budget is the problem and the tail has slack.

Throughput-oriented parallelism is the opposite trade. Attention Data Parallelism (ADP) boosts throughput by replicating attention layers, at the expense of per-request speed. Use ADP when total tokens per second is the bottleneck and your p95 SLA can absorb the penalty. Use TP when the tail is the problem.

High-volume workloads add another axis. P/D disaggregation, or separating prefill and decode onto dedicated workers, is a strategy for optimizing high-volume deployments of LLMs. Test it only when volume is high, batch size and parallelism have hit their limits, and the cost ceiling still fails.

Inference Frontier Decision Sheet

1. Define the quality floor, p95 latency SLA, and $/token ceiling before benchmarking.
2. Sweep model size, quantization, batch size, TP, ADP, and P/D.
3. Plot quality versus p95 latency versus $/token, not quality versus model size alone.
4. Choose the lowest $/token point above the quality floor and inside the latency SLA.
5. If latency fails, increase TP; if cost fails, increase batch size or ADP, or quantize; if volume is high, test P/D.

The output is not a favorite model. It is a point on the frontier: the cheapest compliant stack for your traffic, quality bar, and latency tail. If any constraint changes, move the point. The method stays the same: measure the frontier, enforce the constraints, and buy the cheapest token that still passes.