Make agent traffic faster and cheaper: three serving configs behind vLLM's AgentX results

Agent serving trades cache reuse against rank locality. If each turn repeats the prior context, prefill work grows with turn count; if the cache survives, it grows only with new tokens. That arithmetic decides latency and cost before model choice. On AgentX, vLLM reported up to 130,000 total tokens per GPU-second on DeepSeek V4 Pro and up to 376 tokens per second of interactivity on MiniMax M3. vLLM reported a 14.6x to 106x serving-cost advantage over Opus 5 API pricing across DeepSeek V4 Pro, MiniMax M3, and Kimi K3.

Apply three serving changes:

• Enable interval KV/linear-state retention at turn boundaries.
• Use session or subagent identifiers to direct requests to the rank that already holds their cache.
• Scale prefill and decode separately against a cost-per-token target.

Turn-level state reduces repeated prefill

Enable interval retention at turn boundaries. vLLM's interval-based retention policy preserves prompt-end caches or linear states at each turn for later agentic reuse. A multi-turn agent can then reuse the previous prompt tail instead of rebuilding it, which shrinks prefill batches and memory churn. Default eviction is the failure mode: under load, the last turn's state disappears before the next subagent call arrives, and the next turn pays the full prefill price again.

Save state where turns actually end. A tool call, a user reply, or a subagent handoff can all end a turn. Treating each call as a new prompt throws away the reuse opportunity. Mark the prompt end, not the whole conversation, so the next call starts from the last stable point.

Session pinning reduces cache disruption

Route by session or subagent ID to the rank that already holds the cache. In data-parallel attention deployments, traffic can be steered to the rank holding the relevant cache by session or subagent identifier, reducing repeated cache disruption. One session should hit one rank across turns, with fewer evictions between calls. Pinning by network address or user leaves the cache on the wrong rank and restarts prefix cache thrashing.

Subagents are separate cache keys, so a planner, a tool caller, and a verifier should each carry their own routing ID. That keeps the cache where the next request needs it.

Treat session-aware routing as a load-balancing decision. The router needs a stable key that survives tool calls and retries. When the key changes, the rank changes, and the cache benefit disappears. Keep the key tied to the logical agent run, not to the transport connection.

Prefill-decode split controls cost

Scale prefill and decode separately against a cost-per-token target. The AgentX numbers in the opening are the frame: prefill nodes absorb long prompts while decode nodes keep token streams steady, and the fleet is tuned to a target cost per token. The split matters because prefill and decode stress different parts of the stack. Prefill is parallel work; decode is a long sequence of small outputs. Scaling them together wastes capacity on the side that is not the bottleneck.

Set the cost-per-token target before you scale. The target should come from your traffic mix, not from a single benchmark run. A long-prompt agent run and a short chat run will hit different prefill and decode ratios. Miss the target, and change the serving config before you change the model. Keep the decision tied to the serving stack.

Track upstream changes that move the numbers

Verify the upstream change list before you copy a config. SemiAnalysis linked more than 50 upstream pull requests to AgentX-related optimization work. A serving review should list which cache, routing, and kernel changes moved your latency and cost numbers. Agent workloads expose serving bugs that chat traffic hides. Configs that work for one-turn requests can fail when a session returns after a tool call. Tie each change to the workload it fixes, then re-run the same agent trace after the change.

Keep the same agent trace across changes. The trace should include the turn order, tool calls, and subagent boundaries. Changing the trace changes the comparison too. Use the trace to see whether a routing or cache fix moved the number, not whether a new model changed the workload.