SGLang v0.5.17: cut restart and eviction costs

Operational cost buys restart and eviction behavior

Before enabling SGLang v0.5.17's session cache or weight daemon, count the operational cost. For Qwen3-235B FP8 on 4 GPUs, the measured large-model restart is about 6.5 minutes. SGLang v0.5.17 adds a session-reference-aware Unified Radix Cache and a per-GPU weight-cache daemon. The trade-off is operational: a flag, a persistent per-GPU process, and memory that must be watched. At that baseline, each additional restart adds about 6.5 minutes of downtime. Enable neither by default.

Each lever pays only for a matching workload

Run the checklist in order before changing production flags. For agentic serving, the session cache helps for repeated context already in the radix cache under eviction pressure. The daemon helps for model restart latency that dominates cost. Neither is a general reliability fix.

• Profile restart frequency and session shape. Track restarts per day, median context reuse, and eviction pressure; stop for rare restarts and short sessions. Keep the profile in the runbook.
• Enable the session cache only for stable sessions. Start SGLang with --enable-session-radix-cache so agentic or reinforcement-learning rollout requests can carry a stable session_id. Call /close_session at session end.
• Do not treat a session reference as a pin. It changes KV cache eviction order, but it does not keep the referenced KV or recurrent state resident. Verify that eviction order reflects active sessions without assuming a memory ceiling.
• Start the weight-cache daemon only for frequent large tensor-parallel model restarts. It keeps a persistent process on each GPU that stores the post-quantized tensor-parallel shard, and a new engine reuses that physical GPU memory through CUDA IPC, a GPU memory-sharing mechanism, with Unix-domain socket coordination. Confirm the replacement engine reuses the cached shard before widening the rollout.
• Measure p95 restart time, KV hit rate, GPU memory, and storage-backed L3 misses. Record the baseline before enabling either lever. Compare before and after, and treat either lever as a reliability fix only if tail latency and memory stay inside budget.

In the shipped v0.5.17 change, storage-backed L3 cache is not included in the session lifecycle path. Check whether that path is covered before relying on session references for a storage-backed radix cache.

The sub-second number is a phase, not a recovery time

The weight-mapping phase has a measured sub-second duration. Sub-10-second cold restart and sub-one-second warm standby are still roadmap goals, not shipped whole-engine recovery times. Plan around the measured restart baseline for SLOs that depend on whole-engine recovery.

A canary rollout keeps the rollback path short

Put the session flag on a small pool first, not on the full fleet. Keep the canary small enough that a bad eviction or daemon failure is visible before fleet impact. Watch whether active session references affect eviction as expected and whether memory stays inside the budget. Back the flag off when radix cache eviction harms other traffic. For the daemon, start with a limited GPU set and confirm the replacement engine reuses the cached shard before you widen the rollout. The engine should fall back to the normal restart path on daemon failure without leaving orphaned GPU memory.

The session cache is useful for repeated context already resident, but it can add little value for a mostly cold radix cache. The daemon holds a per-GPU shard in physical GPU memory, so a later restart can reuse it. A memory ceiling makes the persistent shard a capacity risk. In both cases, the decision is a budget question, not a feature question.

Assign one owner to the restart path and one to the cache path. The restart owner should know the measured baseline, the daemon state, and the fallback for daemon failure. The cache owner should know which sessions are active, which prefixes are referenced, and what happens at session end. Put the handoff in the runbook for cross-team ownership. A restart should not wait for a cache decision. A cache eviction should not be mistaken for session closure.