Pricing a 770B MoE: Hy4 preview per-token cost

You are choosing GPUs and token prices for Hy4 preview, a 770B-total, 49B-active mixture-of-experts model. The trade-off is memory versus arithmetic: weight memory follows the stored parameter set, and decode compute follows the active path. The first price lever is the active share: 49B/770B. Use that ratio before you compare Hy4 preview to a dense model, because the headline size is not the compute size. Record that ratio in the worksheet first.

Active parameters set the compute baseline

Inference compute scales with the 49B active parameters rather than the 770B total set. Hy4 preview is a 78-layer MoE with 256 routed experts and one shared expert, and it activates 8 routed experts plus the shared expert for each token. Its attention mechanism is Gated DeepSeek Sparse Attention with IndexCache, and its native multi-token prediction layer adds a 10B-parameter speculative decoder with 0.7B active parameters. If you enable that layer, add its active path to the per-token compute estimate. The active total is the number you multiply by your compute model, not the total parameter count.

The total count is the storage and loading number; the active count is the token-latency number. Hold those two numbers separate in the worksheet: one for memory budgeting, one for decode throughput. That separation prevents a dense-model intuition from inflating the price.

FP8 and eight-GPU tensor parallelism set the memory baseline

The release ships BF16 and FP8 checkpoints, and the official vLLM and SGLang recipes run the FP8 variant with eight-GPU tensor parallelism. Make FP8 the default because the official recipes run it, and take the recipe's parallelism size as the starting point. Tensor parallelism changes communication cost, not just weight placement, so a different GPU count changes both memory headroom and decode latency. If you change the GPU count, rerun the benchmark before changing the price.

Treat the context window above 1M tokens as a capacity constraint when your workload uses long context. Reserve memory for cache and sparse attention state, not only weights. Long context can dominate the memory budget even when the active compute path is small. A reproducible benchmark config should let you rerun after changing context length, batch size, or cache policy, and the config should record the checkpoint, parallelism size, and serving stack.

Borrow the launch throughput range before you benchmark

At launch, OpenRouter listed Tencent Cloud as the sole Hy4 preview provider with roughly 26 to 38 tokens per second and a sub-4-second P50. Use that range as a sanity check for your own decode throughput, and report tokens per second per GPU rather than only tokens per second per request. For the official eight-GPU recipe, divide request-level tokens per second by eight to get a per-GPU number.

Hy4 preview's API costs 0.834 USD per million input tokens, 2.501 USD per million output tokens, and 0.042 USD per million cache-hit tokens. For a self-hosted estimate, divide your GPU-hour cost by the tokens you can serve per hour, then compare the result to those API rates. Use the same context length and batch size in the benchmark and the cost model. If your traffic has repeated prompts, count cache-hit tokens separately and compute a blended cost as the weighted average of the three rates by your token mix.

Rerun the worksheet at your context and cache mix

• Compute the active parameter total: base active count plus speculative active count when multi-token prediction is enabled.
• Choose the FP8 checkpoint and the recipe's eight-GPU tensor parallelism size.
• Reserve memory for long context and sparse attention state.
• Benchmark decode throughput in tokens per second per GPU at your target context.
• Compare your per-token cost to the API rates.

Run the worksheet at short and long context lengths and at low and high cache-hit rates. The gap between those runs tells you whether your price is driven by compute, memory, or repeated prompts. Record the measured context length, batch size, cache-hit rate, and GPU count next to the price so the number can be reproduced.