When a Kubernetes inference control plane pays: llm-d v0.8

You are choosing between a Kubernetes inference control plane and bare engines. The trade-off is arithmetic: subtract control-plane outage cost from the glue code it deletes; a positive result means adopt the control plane. Per-token cost follows from the deleted work: fewer components, shorter upgrades, lower incident time. Stay on bare engines when a stable engine and self-owned upgrades are cheaper than a control-plane dependency. Name the failure mode your team can own. A control plane earns its place only when the deleted code is recurring, not one-time.

Production status changes the outage cost

llm-d v0.8 is the reference point because its Flow Control, Batch Gateway, and multi-modal paths are production, not experimental. Production status makes the dependency a budget line. A control plane that only routes traffic is easier to bypass. One that owns batch jobs, admission control, and multi-modal paths is harder to replace without breaking SLOs.

Engine variety is the first cost to measure. llm-d v0.8 treats SGLang as a first-class engine next to vLLM across most supported paths. Multiple engines multiply the routing surface: model-specific endpoints, scaling policies, and image tags all need one owner. The multi-engine case becomes a routing layer, not a fork of the engine.

Run the five-question adoption test

Run the five-question adoption test before you add the dependency. Each item is a cost you can verify in a minute; keep the list with the runbook. Treat it as a pre-deployment cost check.

• Engine count: do you run multiple inference engines? The control plane absorbs routing and scaling differences.
• Batch size near or above 50,000 requests: the llm-d v0.8 Batch Gateway exposes OpenAI-style /v1/batches and /v1/files endpoints, with a job ceiling at that size.
• Flow-control SLOs: admission and backpressure spread across endpoints; they belong in a shared place.
• Image-and-text P/D need: do you need disaggregated prefill/decode for image and text models?
• Image-upgrade ownership: do you own engine image upgrades instead of tracking a control-plane image?

Adopt when at least two items are true and the control-plane outage cost is below the glue-code cost. Fewer items weaken the arithmetic. The two-item threshold keeps the decision from turning on a single convenience.

Price the batch and image work

Batch work is the easiest cost to price: file upload, job tracking, retry policy, and completion handling are stateful. When that state lives in a control plane, the API server stays thin. A sidecar that builds job state, retries, and file handling is the work the control plane deletes.

llm-d v0.8 sets the default GPU image to the unmodified upstream vLLM build vllm/vllm-openai:v0.23.0. It removes the custom llm-d-rocm, llm-d-xpu, and llm-d-cuda-gb200 images. PR #1791 switches llm-d platforms to direct upstream vLLM image use, not llm-d-built custom builds. The image change shifts release risk from the control-plane team to the engine release stream. Track upstream engine releases without waiting for a control-plane image, and your upgrade path is shorter. Otherwise, the control plane adds a release dependency to every engine change.

llm-d v0.8 now documents production multi-modal serving, with split prefill/decode paths for vision and text models like LLaVA and Qwen-VL. Prefill and decode have different resource shapes, so a split path changes node sizing, scheduling, and failure isolation. Roadmap models of that kind make the control plane shape serving topology, not just route text tokens.

Make the outage cost decide

A control plane is a shared dependency. Its death can remove routing, admission, or batch state from every engine behind it. The outage cost includes lost admission, stalled batch jobs, and rerouting work during the incident. The question is whether that failure is cheaper than the failure modes it replaces.

Price the alternative first. A bare deployment is cheaper when the workload is one engine, one image, and a short upgrade path. A deployment with multiple engines, batch jobs, and image-and-text paths has large custom code, and the control plane is doing real work.

Run the reliability test in staging before production: bypass the control plane during an incident, canary an engine image without a control-plane release, restore batch state without rebuilding job metadata, and measure recovery time. Any failed step means plan the incident path before deployment. The output is a known incident path, not a hope.