890 bytes/token: KV-cache math for 1M-token DeepSeek V4.1 Flash

At 890 bytes per token—about one quarter of V4-Flash and 437 times smaller than V1—a 1,000,000-token DeepSeek V4.1-Flash session holds 890,000,000 bytes of KV state before prefix hits. The trade-off is memory per token versus sessions per GPU: that byte count decides how much context stays in HBM, how much spills to host RAM or SSD, and what a cache miss costs. If the pool cannot hold the uncached tail, the scheduler must evict, offload, or reject work, and each choice changes latency and cost.

Bytes per token set the HBM ceiling

DeepSeek shipped V4.1-Flash on September 10, 2026, as the third entry in the V4 line. The model has 552B total parameters, 8B active in prefill, and 16B active in decode. Those active counts shape compute, but the serving ceiling comes from KV bytes per token. The memory budget is the KV state. A parameter-efficient model can still fail a capacity test if the per-token cache leaves too little room for residency.

Weights occupy a fixed slice of the pool, while KV state grows with every token you keep. The causal encoder-decoder design splits 40 layers into 20 encoder layers and 20 decoder layers, with decoder KV projected from encoder outputs. That split changes where state is stored, not the token count you must budget. The CSA2 hierarchical sparse indexer caps the candidate pool at 16,384 positions, so per-decode-token FLOPs rise by only one quarter as context grows from 4K to 1M. Decode cost stays bounded, but the memory bill still scales with the tokens you keep.

Prefix hits turn bytes into sessions

The uncached session cost is KV bytes per token times context length. A prefix hit removes the bytes for the shared prefix, so the real session cost is the uncached tail plus any replayed state. A smaller per-token KV cache lets more context stay cached, raising prefix hit rates, cutting time-to-first-token, and lifting throughput per GPU. Lower bytes per token increases sessions per GPU, but only if the prefix actually repeats. If the prefix hit rate is low, the uncached tail grows and the session count falls.

Agent workloads are prefix-heavy. System prompts, tool schemas, repository context, and earlier turns often repeat across calls. When those prefixes stay cached, the scheduler skips attention over them and serves the next token faster. When they do not, the same user-facing session can cost more than a shorter one that reuses a stable prefix. For a request-rate estimate, divide expected decode throughput by average tokens per request, then cap it by the session count the KV pool can hold. That cap is the sessions-per-GPU number you can use before choosing hardware.

Replay and offload set the miss penalty

The main KV cache uses E2M1 with one E4M3 scale per 16 channels; 128-token sliding-window states sit in host RAM with a minute-level TTL, and a miss triggers SWA Bounded Replay to recompute the last 128 tokens. A short TTL can turn a cached prefix into a recompute if the agent pauses. Budget the replay as extra prefill work, not as a free cache hit.

Host RAM sits between HBM and SSD in the usual serving hierarchy. Treat it as the first recovery tier. A sliding-window state that remains in host RAM can be recovered quickly, while a state that has moved to slower storage can force a larger recompute. The TTL matters because agent sessions pause: a tool call, a human approval, or a background job can leave the prefix idle long enough to expire. If the uncached tail plus replayed state exceeds HBM, the offload path changes latency and tail cost, not the byte count.

Judge sessions by bytes, not vibes

• Session cache = bytes/token × tokens; plug in the model's KV bytes and context length.
• Subtract prefix hits: remove bytes for tokens already cached, then add replayed state for any miss.
• Compare HBM residency with host-RAM or SSD offload; the offload path changes latency, not the byte count.
• Compute sessions per GPU by dividing usable KV memory by the per-session byte total, then apply the replay penalty.