The stack under your AI product — models, serving, and what it costs
Chess has about 10⁴⁷ positions. No-Limit Hold'em has about 10¹⁶⁰. That ratio is the whole story of why the machine beat chess in 1997 and poker in 2019 — and why the machine now answers either one in seconds on a cloud invoice.
Re-run the GPU-hour rate, tokens per GPU-hour, and cost per token before signing reservations, then compare to the index and decide.
When B200 residual values and rental rates rise, re-run depreciation, $/token, break-even, inflation, and renewal leverage before signing capacity.
A checklist maps pinned model IDs to endpoints, cache state, and peak windows before you trust the bill.
Audit H100, H200, and B200 $/token, utilization, and migration cost after the H100 rent jump.
Audit benchmark provenance, cost basis, availability, and task-specific gaps before routing production coding traffic to a cheaper model.
Route by cost per completed task, not sticker price: a flagship wins only when token efficiency, cache reads, and context length offset the price gap.
The sticker price hides the output-token multiplier and the future price step that decide whether to migrate.
Use only non-promotional, plan-wide, schedule-stable rates as token-budget floors; treat expiring or coverage-limited cuts as temporary assumptions.
A checklist to verify whether a cheaper, stronger model lowers effective cost per token after cache, peak hours, KV memory, concurrency, quality gates, and rollback.
Convert Sonnet 5's per-token price into cost per task before you change the model pin, fallback, budget alerts, or evals.
Fable 5.1's cheaper cache reads shift the cost lever: cut write churn, protect stable prefixes, and make read reuse the main line item.
Rental prices for B200 and H100 have moved, so self-hosted inference cost must be rebuilt from current GPU-hour rates, sustained throughput, and utilization.
When frontier API rates drop, rebuild the cost-optimal route and hosting choice from current token prices, task mix, and index.
Fixed hardware plus unified memory can undercut cloud per-token pricing once token volume is steady enough to amortize the machine.
A four-step cost framework: classify the task, route by model cost, cap context, and measure token savings against quality and latency.
A lower token price shifts inference spend toward cache misses, routing, and serving overhead. Redesign the request path before chasing tokens.
A practical model for pricing AI features when token prices fall, usage rises, and inference becomes a controllable line item.
Set a monthly AI agent inference budget from turns, tool calls, context growth, model routing, retries, and a variance reserve.
Define quality, p95 latency, and $/token limits, then sweep model size, quantization, batching, and parallelism to find the cheapest compliant point.
Treat VCF as a private inference platform only when measured vLLM latency, GPU scheduling, and token economics beat a managed model API.
A six-line worksheet turns accelerator economics into a defensible token price floor before you price inference.
A local server can beat API tokens when disk, decode, prefill, compatibility, and cost all clear the bar.
Price redundancy, standby capacity, and failover latency against the revenue or SLA credits lost when inference goes down.