Budget speculative decoding as a two-model system

The trade-off is two artifacts, not one

Draft-model speculative decoding turns one serving artifact and one KV pool into two of each. The latency win pays only if acceptance length covers the extra weights, extra KV, and extra staging. The arithmetic is target weights plus drafter weights, target KV plus drafter KV, and an acceptance length that must clear the draft block. SGLang v0.5.18 was released on 2026-08-21 and included 710 pull requests from 212 contributors. In that release, DSpark, DFlash, MegaMoE, and MTP became generally available.

Count the two downloads before traffic: target model, drafter model. Size the two KV pools before traffic: target attention, drafter attention. Set the acceptance-length target before traffic: the latency input that decides whether the second pass pays. A weak drafter makes you pay for two models and still run the target. An over-strong drafter can make you overpay in memory for a latency gain you do not need. Put the two-model budget in the runbook.

The budget has five testable lines

Keep this checklist in the runbook. Each item is a quick check before you flip the draft-model switch. The budget you keep is a target/drafter serving budget: artifact count, KV-cache geometry, drafter acceptance length, startup staging, and co-design hook. The first two lines are the memory bill, the third is the latency dial, the fourth is the startup tax, and the fifth is the co-design hook.

• Add the drafter to the image and storage budget.
• Size the drafter KV pool from its own attention geometry.
• Set an acceptance-length target before rollout.
• Stage checkpoints overlapped so startup does not eat the latency win.
• Check that the drafter's feature taps and block size match the target.

Run the checklist before rollout, not after an incident. The five lines are the pre-rollout gate. The first two lines show up in the memory graph, the third in the latency graph, the fourth in the deploy log, and the fifth when the target changes.

The KV pool and acceptance length are the two dials

The drafter is small, but it still owns memory. The Ling-3.0-flash-dspark checkpoint is a five-layer BF16 transformer: 2,560 hidden dimensions, 1,363,707,905 parameters, 32 query heads, 32 KV heads, auxiliary target features from depths 1, 11, 23, 29, and 35, a rank-256 Markov head, and an 8-token DSpark block.

The DSpark drafter maintains its own KV cache, sized from its 8-token block, 32-head, and five-layer attention geometry rather than sharing the target model's cache. Use the drafter's geometry as the sizing input, not the target's context length.

Do not borrow the target's sequence budget when sizing the drafter pool. The drafter's block size, head count, and layer count define its own footprint. A long-context target does not make the drafter pool long-context; it still needs only the geometry it uses. A short-context target can make the drafter pool small, but it remains a separate allocation. Track it as a separate line in the memory budget.

inclusionAI, an Ant Group unit, published Ling-3.0-flash-dspark as the first non-Liquid DSpark drafter, with 1.36B parameters and a 5.29 macro mean acceptance length over nine benchmarks. The drafter is deployed as a separate model artifact alongside the target model in a single SGLang serve invocation. Set the acceptance target from your traffic, not from a single benchmark. A macro mean is a starting point. Your production mix can be easier or harder than the benchmark mix. If your workload accepts fewer tokens per draft pass, the drafter's extra compute and KV traffic can outweigh the saved target steps. If it accepts more, the same two-artifact budget buys lower latency. Make the acceptance target a serving SLO, checked per request or per batch rather than as a global average. Show the acceptance target in the runbook, the deploy check, and the SLO dashboard.

Startup staging is part of the budget

Startup time is where a latency win can disappear before the first request. In SGLang v0.5.18, overlapped checkpoint staging made Qwen3-32B on H100 start 8.6% to 11.7% faster than serial prefetching and 2.38 times faster than the plain default, with 35.6 seconds versus 84.8 seconds.

Treat that as the staging line item and make it part of the latency budget. The drafter adds a second artifact to stage, so the overlap budget must cover both. Startup staging is the time from cold start to first request. Serial loading makes the cold-start penalty additive; parallel staging keeps it closer to the slower artifact. Keep the staging line item visible in the deploy plan, not hidden behind model load. Name the staging mode, the artifact order, and the warmup check in the budget.

The co-design hook is the reason the drafter is not a drop-in. A target change requires a new acceptance check, a drafter change requires a new KV budget, and a change to both requires the whole two-model plan.