Estimate AI Serving Cost in Five Numbers: Z.AI's 100,000-Accelerator Run
A serving audit turns AI bills into arithmetic: active parameters, tokens per day, accelerators, cost per token, and prefill/decode scaling.
Models
Choose the model your serving stack can run
Pick foundation models by pretraining scale, modality, and inference behavior, then match them to API, self-hosted, or hybrid serving.
Turn a frontier model update into a serving budget: the Gemini 4 example
A frontier model becomes a product only when post-training, quantization, latency, and price per million tokens fit a serving budget.
Open-Weights-First Model Routing Pipeline
Route most production tokens to open weights, and reserve closed APIs for narrow exceptions that pass quality, cost, and latency gates.
Pick a 2.4B SLM when your workflow clears the cost, latency, and privacy gates; otherwise pay for frontier
Use a small language model when the task is bounded, evals pass, and deployment fits your latency and privacy budget; otherwise buy frontier.
Advertisement
Pick a Production LLM With a 6-Point Eval
A production LLM choice is a trade-off: compare fit, eval score, p95 latency, token cost, license risk, and fallback success before committing.