Ship Reliable LLM Inference: A Production Architecture for Batching, KV-Cache Reuse, Autoscaling, Routing, and SLO Observability

The trade-off in LLM serving is latency versus cost: each request is a queue of tokens, and the bill is requests times tokens times price per token in dollars. If you raise batch size, you can lower cost per token, but you may raise tail latency; if you add GPUs, you can lower latency, but you may raise idle cost. The architecture has to make that arithmetic visible before it makes the model feel smart.

Capability is not a control. A model can answer well and still fail a user-facing SLO because the serving path hides queueing, cache misses, routing errors, and capacity gaps. Production inference is an engineering control problem: you need batching, cache reuse, autoscaling, routing, rate limiting, and observability to turn model output into a service with a cost ceiling.

Costs

Start with the cost model, not the model card. For every request, estimate prompt tokens, completion tokens, cache hit rate, batch occupancy, and GPU idle time. The unit you care about is dollars per accepted request, not dollars per GPU-hour in isolation. If a request can be served from a smaller model, a cached prefix, or a batched queue, the cost should drop before the user notices.

Vendor claims can help bound the opportunity. Nutanix Enterprise AI 2.8 adds an MCP-based Agent Gateway and enhanced Private Inference with multi-GPU tensor parallelism, speculative decoding, and LoRA fine-tuning for sub-8B-parameter models. Nutanix reports up to 2.5x inference acceleration. Treat that as a ceiling, not a promise: the figure is an upper bound.

Use the number to set a test: if the same request set moves from one serving configuration to another, measure tokens per second, queue wait, and dollars per request. If the speedup does not reduce cost per accepted request, it is a latency feature, not a cost control.

Models

Model routing is the cheapest way to keep quality high and cost bounded. Route by task, not by habit: short classification, extraction, and summarization can often use smaller models; long-context, tool-heavy, or high-risk tasks can use larger models. The routing decision should be explicit, logged, and reversible.

Low-rank fine-tuning is useful when a narrow task needs stable behavior without paying for a larger base model. Nutanix says NAI 2.8 is generally available for multi-GPU, low-latency LLM serving and LoRA fine-tuning on sub-8B-parameter models.

Do not route on vibes. Define a task taxonomy, set a quality bar for each class, and log the model chosen. If a smaller model passes the bar, use it. If it fails, escalate. The goal is not to hide the model behind a facade; it is to make the model choice a measurable control.

Serving

The serving stack is where SLOs are born or broken. A reliable LLM API architecture needs six controls: request batching, KV-cache reuse, GPU autoscaling, model routing, rate limiting, and observability. Each one changes the arithmetic.

1. Request batching. Continuous batching lets new requests join an active batch without waiting for the whole batch to finish. It improves GPU utilization, but it can increase tail latency. Set a maximum batch age and a maximum queue length, then measure both.
2. KV-cache reuse. Reusing cached prefixes avoids recomputing attention state for repeated system prompts, tool schemas, or document chunks. Track cache hit rate and cache eviction rate. If hit rate is low, your prompts are not shaped for reuse.
3. GPU autoscaling. Scale on queue depth, token throughput, and tail latency, not only on GPU utilization. A GPU can look busy while users wait. Add scale-up rules for burst and scale-down rules for idle cost.
4. Model routing. Put routing in front of the model, not inside the prompt. Log the route, the model, the token count, and the outcome. A route that cannot be audited is a hidden cost center.
5. Rate limiting. Limit by user, tenant, feature, and model class. Rate limiting is not just abuse control; it is capacity protection. If one tenant can starve the batch queue, the SLO is not real.
6. Observability. Expose request latency, queue wait, batch size, cache hit rate, token throughput, error rate, and cost per request. SLOs need a budget, a burn rate, and an alert that fires before users notice.

Infrastructure choices are constraints on the six controls: the platform's expected bare-metal Kubernetes, full-stack AHV, AI catalog, expanded GPU support, and CNCF AI conformance set the platform boundary; Unified Storage's NVIDIA-Certified Storage validation and low-latency, high-throughput GPU data path set the storage path. Nutanix Unified Storage provides a low-latency, high-throughput data path to GPUs for large-scale production AI workloads.

Agentic workloads make the control plane more important, not less. Without it, every agent is a new source of unbounded load.

The point is simple: model capability is the input, not the service. Ship the controls first. Batch the requests, reuse the cache, route the model, limit the load, scale the GPUs, and observe the cost. When the arithmetic is visible, the product is reliable; when it is not, the demo is just a demo.