Stop re-prefilling long hybrid contexts: when one cache tree pays off

You are serving a hybrid-attention model and the same long prefix is being prefilled again. SGLang's unified radix cache trades extra tier state for fewer repeated prefill passes. In SGLang's HiCache results, DeepSeek-V4-Flash with an L3 Mooncake tier delivered 145.5K effective input tokens/s at a 98% hit rate, while GPU-only cache delivered 9.4K. The decision is whether the prefix is worth preserving. Enable the tree only when the prefix is reusable; otherwise it adds state without saving work.

Hybrid attention is the first gate

SGLang's Unified Radix Cache models DeepSeek-V4 as FULL plus SWA, Kimi-K3 as FULL plus MAMBA, and Inkling as all three components on one tree. The gate is architectural: the tree is built for mixed attention styles. An all-attention model does not need that hybrid path. A model with sliding-window or recurrent state can keep different component states in one tree instead of a separate cache per style.

Flip the switch only after four checks

Operators can enable SGLang's Unified Radix Cache by setting the environment variable SGLANG_ENABLE_UNIFIED_RADIX_TREE to 1. The flag is cheap; the workload test is the cost. The decision is workload-specific. SGLang's Unified Radix Cache is designed to keep cache hit rate high and TTFT growth slow in multi-turn workloads when lower tiers preserve the reusable prefix after GPU capacity is exhausted. A failed check is a stop, not a reason to tune harder.

• Confirm the model has a non-FULL component, such as SWA or MAMBA.
• Check that traffic has long reusable prefixes or multi-turn sessions.
• Name the bottleneck as TTFT or input-token cost.
• Prove you can measure hit rate, TTFT, and effective input tokens/s against L1, L2, and L3 baselines.

A model that already fits a simpler path should not get a generic cache. The traffic gate targets prefixes that survive across turns. The bottleneck must be named before you add state. The win must be auditable before you commit capacity and before you scale it.

Benchmark the tiers before you trust the tree

In SGLang's multi-turn benchmark, Inkling-Small with L3 delivered 67.1K effective input tokens/s, a 96.8% hit rate, and 1.23-second TTFT, while L1 alone delivered 15.5K and L1 plus L2 delivered 21.1K. The lower tiers change the size of the win; the top tier is not a replacement for them.

SGLang's experimental Rust tree core cut TTFT by up to 42% on long sliding-window attention workloads. Treat it as an opt-in experiment after the main tree has a stable baseline; keep it if long-window latency improves, drop it if it adds variance. Do not make it the default.

On SWE-bench agent trajectories, session-aware eviction in Unified Radix Cache lowered TTFT by 2.9% to 16.6% relative to the HiRadixCache baseline for DeepSeek-V4-Pro and Qwen3.5-397B-A17B. The smaller deltas still point to the same operating condition: session ownership keeps first-token latency flatter.

Start with the baseline you already run. Record cache throughput, hit rate, TTFT, and effective input tokens/s for each tier, then add the next tier and compare side by side. Keep the same prompt mix, batch size, and session length for each run. Do not mix traffic shapes between runs. Keep the run window fixed. A comparison is only useful if the traffic is the same; change one tier at a time so the delta is attributable. First-token latency is the user-facing symptom; input throughput is the cost signal. If the tree slows the hot path, the cache is not helping.

Every missed prefix is paid again in prefill compute, and every hit shifts the cost toward memory and network. Do not enable the tree because the model is new or a demo looked fast. If traffic is short, stateless, or dominated by unique prompts, the cache has little to preserve. Keep separate caches or a GPU-only cache and re-test when the prefix mix changes.