Budget TTFT: prefill, batching, quantization, and routing for streaming LLM products

The trade-off: first-token latency is a prefill tax

The trade-off is simple: first-token latency is bought with prefill capacity, and every knob that lowers it can raise spend or reduce throughput. A useful budget is TTFT ~= prompt_tokens / prefill_tokens_per_s + queue_wait + network. Time to first token measures the delay from prompt submission to the first output token. For streaming LLM products, that delay is the first thing users feel, so it should be treated as a cost line, not a mystery.

First-token latency is largely driven by the prefill phase, where the model processes the full prompt before generating the first token. That means a long system prompt, retrieved context, conversation history, or tool output is not just a quality input; it is a latency input. If the prompt grows, the prefill term grows. If the serving node is busy, the queue term grows. If the request crosses a slow network path, the network term grows. The opinion comes after the arithmetic: do not chase a lower TTFT by adding capacity you cannot afford, and do not hide a prompt-size problem behind a bigger GPU.

In production, TTFT matters because it shapes perceived responsiveness. A user who sees no first token quickly may assume the product is broken, even if the rest of the stream is fast. Inter-token latency is the other half of the experience, but it does not rescue a slow start. If the first token arrives late, the stream feels dead before it begins.

Set a TTFT budget before you tune

Start with a budget, not a benchmark. Define the prompt-token ranges you serve: short chat, long context, agent traces, RAG-heavy answers, and system-prompt-heavy workflows. For each range, set p50 and p95 TTFT targets you can defend. Then write it as an equation you can inspect:

• Prefill term: prompt tokens divided by the prefill tokens per second your serving path can sustain.
• Queue term: time waiting for a worker, slot, or batch to accept the request.
• Network term: client-to-gateway, gateway-to-worker, and any cold-start or routing overhead.

The budget should be split by ownership. The model team owns prompt-token efficiency and model choice. The serving team owns batching, cache reuse, and worker placement. The product team owns the prompt architecture and the user-facing tolerance for delay. If the p95 target is missed, the first question is which term grew, not which team is at fault.

Buy latency without paying twice

The first lever is prompt-token budget. Trim what does not need to be in the prompt. Compress history, summarize older turns, retrieve only the context that changes the answer, and move stable instructions into a cached prefix when your stack supports it. This is the cheapest latency control because it reduces the prefill term directly. It also reduces spend per request, which makes the rest of the stack easier to defend.

The second lever is batching. Batching can improve throughput and sometimes latency, but larger batches can trade throughput for per-request latency. That trade-off is why a fixed maximum batch size is not a detail; it is a product decision. A small batch may protect TTFT but leave GPU capacity idle. A large batch may raise tokens per second and lower cost per token, but it can make the first token wait. The right setting depends on the prompt-token distribution and the p95 target, not on a generic maximum.

The third lever is model and precision choice. Quantized or smaller models can reduce memory and compute per token, which can lower TTFT and end-to-end latency. This is a quality-versus-latency decision, not a free win. Measure the quantization delta on the tasks that matter: instruction following, tool use, long-context recall, and safety. If the smaller or quantized model keeps the product useful, it can be the fastest way to buy first-token latency. If it does not, the latency savings are not real, because you will pay in support load, retries, or user churn.

The fourth lever is routing. Cache-aware routing can avoid re-running prefill by routing requests to nodes that already hold relevant KV cache, which is why production guides recommend it over round-robin for multi-replica inference. A request that lands on a cold worker repeats work that another worker already did. Cache-aware routing is more stateful and harder to debug, but it can turn repeated prompts into a cheaper, faster path.

Every lever should be scored against $/1k tokens, not just latency. A configuration that lowers TTFT by spending more on idle capacity can be worse than a configuration that accepts a slightly slower first token and keeps the unit economics sane. The goal is not the fastest possible first token; it is the first token that fits the product's latency budget and cost budget.

What to log and how to decide

Log the inference metrics that let you reconstruct the budget. Record TTFT, inter-token latency, tokens per second, prompt tokens, output tokens, model, batch size, cache hit or miss, and $/1k tokens. If you cannot separate prefill time from queue time, you cannot tell whether a slow first token is a model, scheduling, or routing problem.

1. Set a prompt-token budget for each product surface and enforce it in the request path.
2. Set p50 and p95 TTFT targets that match the user experience, not the marketing page.
3. Choose a maximum batch size that protects the p95 target while keeping throughput high enough to control cost.
4. Test quantized or smaller models on real product tasks before treating them as a latency fix.
5. Use cache-aware routing when repeated context is common, and measure the prefill savings it produces.
6. Review the cost per 1k tokens alongside latency, because a faster first token that doubles spend is not a win.

The practical rule is to treat TTFT as a prefill-phase cost. If the prompt is too long, fix the prompt. If the queue is too deep, fix scheduling. If the model is too heavy, fix the model. If the cache is cold, fix routing. When the budget is explicit, choices stop being vibes and become arithmetic.