API vs self-hosting a large open MoE: Kimi K3 serving math
Compare API and self-hosted serving on memory, active compute, cache, and license before buying GPUs.
Kimi K3 is an open-weight 2.8-trillion-parameter model and set the open-model parameter record. Your decision is whether to serve it on your GPUs or call Moonshot's hosted API. Self-hosting keeps prompt inference on your infrastructure instead of sending prompts to Moonshot's hosted API servers. The trade-off is memory against compute: divide weights memory, context memory, active compute, and cache-adjusted prompt cost by tokens, then compare the result to the API price. License terms are a separate check.
Cost, isolation, and load decide the trade-off
Measure both options on three terms: cost, isolation, and operational load. Cost is price per token after cache. Isolation asks whether prompts leave your network. Operational load is the memory, scheduling, and failure surface you own. Keep the terms separate; they do not move together.
For the API, cost is the provider's input and output price. Isolation is weaker because prompts leave your network. Operational load is low, but you inherit provider incidents. In April 2026, Moonshot's hosted Kimi service had a cross-user data isolation failure in which a user's resume details were shown to another user.
Self-hosting costs are your cluster amortized over tokens. Isolation is stronger: prompts stay on your infrastructure. The operational burden is high: weights memory is the first constraint, and you also own routing, batching, and context state.
Only cost can favor the API when isolation favors self-hosting. Bursty traffic lets the API absorb peaks without idle GPUs. Steady, prompt-heavy traffic lets cache change the result. Compare your marginal cost to the API invoice before you buy hardware.
MoE inference cost is active parameters, not total parameters
Kimi K3's Stable LatentMoE routes each input token through 16 active experts from a pool of 896. Token-level compute load is more comparable to a 50-billion-parameter dense model than to the full model. The model's quantization-aware training uses MXFP4 weights and MXFP8 activations. MXFP4 is the format behind the weights memory estimate below.
The context window is another memory term. Kimi Delta Attention gives Kimi K3 a one-million-token context window by pairing standard quadratic-cost attention with a linear-cost alternative. A maximum-context serving stack must hold KV state, routing state, and batch metadata for long prompts. Short-context workloads make the maximum-context capability a spec, not a cost.
Batching changes the arithmetic. Higher batch size can lower cost per token, but it raises tail latency. Prompt caching changes the input side. Repeated prefixes lower your effective prompt cost. Novel requests leave the cache discount small. Track both levers in your cost model.
Run the self-host test before buying GPUs
Keep the test to the inputs below. Missing inputs make the API the safer default.
- Weights size: about 1.4 terabytes of fast memory in MXFP4.
- Active parameters: 104 billion per token.
- Context memory: budget for maximum-context requests.
- Cache-hit discount: multiply prompt cost by the complement of your hit rate.
- License terms: the Kimi K3 License governs the model's code and weights.
The weights number is the deciding input. The complete weights were scheduled for release by July 27, 2026. Do not include it in the cost model until the weights are available.
The inputs decide the winner
For platform builders, self-hosting wins when your cluster is already paid for, your cache-hit rate is high, and your license terms are clear. The API wins when you need capacity now, cannot hold the weights memory, or cannot satisfy the license terms. Use the API as the baseline. Put your actual price in the formula. If self-hosting's marginal cost is lower, deploy it. If not, keep the API and revisit when your cache-hit discount or cluster utilization changes.
Run a side-by-side comparison before committing. Send the same traffic to the API and to your self-hosted stack. Compare price per token, tail latency, and error rate. Do not buy GPUs to make the self-hosted stack cheaper at your real mix. Leave the API as overflow and as a fallback while you resolve license terms.