Build a Sub-$1 Inference Cost Model

Token price is the wrong number

The trade-off is between list price and effective cost. Cache hit rate, batching, and routing decide the effective price. The arithmetic is: effective price = list price × (1 − cache discount) × batch discount × routing mix. That line decides whether inference is controllable or a margin leak.

The volume-weighted token price closed near $0.97 per million in August, down more than half from a May peak near $2.05. On OpenRouter, a model-routing platform, token usage rose about 47% month-over-month in August, while dollar spending rose only about 7%. Usage is growing faster than spend. That is the condition that makes a cost model necessary.

Goldman Sachs estimated that five rated hyperscale cloud providers would have combined capital expenditures of about $737 billion in 2026, roughly 38% of their revenue. By the end of August, 78 of the 91 bonds from hyperscale cloud providers in 2026 traded below issue price. The buildout is large enough that token pricing pressure is not a temporary dip.

The model starts with effective price

Measure the price you actually pay

Track input, output, cache, and batch rates for every model you serve. Include prompt caching, response caching, and any committed-use discount. Done looks like a rate card that maps each request class to a price before the request leaves the queue.

Start with the vendor's list price, then subtract only discounts you actually receive. A committed-use discount applies only to the volume you expect to hit. Pay-as-you-go users should not borrow a discount they have not earned. You are done when the rate card shows list, discounted, and effective columns.

Treat cache as a discount

Cache hit rate changes the price of the same token. A repeated system prompt, tool schema, or document chunk served from cache lowers the blended price. The common mistake is treating a lower list price as a lower cost when cache misses and small batches raise token consumption.

Record cache hit rate by feature, not by model. A chat feature may cache system prompts well, while a search feature may miss because queries are novel. A falling hit rate raises the effective price even when the list price is unchanged. The output is a blended-price dashboard by feature and model.

Price batching by queue depth

Batching changes cost per token, but it also changes latency. Model the minimum batch size that keeps your latency target inside the product experience. You are done when the queue-depth curve shows a cost line and a latency line.

Do not assume continuous batching is free. It can reduce cost per token, but it can also increase queue time. Users who retry when latency rises can grow token consumption. Model the batch size that keeps tail latency inside your target. Batch size, cost per token, and latency belong in the table.

Route by task cost

Model tiers move. Meta's Muse Spark 1.3 reached the same benchmark tier as GPT-5.6 Sol and Claude Opus 5, frontier models, after its September 2 launch. OpenAI's Astra met cybersecurity thresholds under its Preparedness Framework, a safety review process, on September 1, becoming the first model in that category. Use those changes to test whether a cheaper model can serve a task at acceptable quality.

Build a task map before you change models. List the jobs each model does: classification, extraction, summarization, code, safety review, and final answer. Assign each job a quality bar and a price tolerance. Test the cheaper model against that bar. The routing table holds task, model, quality gate, and effective price.

Convert to cost per paid session

Multiply requests per session by tokens per request, then apply the effective price. Separate free sessions from paid sessions, because the guardrail only protects revenue. The number should compare cleanly against subscription price without a spreadsheet of raw tokens.

Keep the session definition stable. A paid session that includes onboarding, support, and usage needs a clear rule for which calls count toward the guardrail. A long session can hide a bad unit. Split by user tier when needed. The result is a cost per paid session by tier, not just by model.

Guardrails keep growth from hiding the leak

Set the ARPU guardrail

Set the ARPU, average revenue per user, guardrail before the next pricing review. Say you cap effective inference cost at 20% of ARPU for a paid session and allow volume growth to outpace price decline by no more than 10%. Above that line, the feature is no longer self-funding. That rule forces a choice: raise price, cut tokens, improve cache, or stop the feature.

Put the guardrail in product review

Make the guardrail part of the product review. When a feature request asks for more tokens, require the effective cost impact. A request that raises cost per paid session above the cap needs a price change, a cache improvement, or a scope cut. Launch should wait until the number is inside the cap.

Re-run the model when inputs move

Review the model whenever an input moves: list price, cache hit rate, batch size, routing mix, or session length. A token price cut is good news only when the effective cost per paid session falls. A new model or cache provider change requires a rerun of the effective price. Use the same formula and updated inputs in the review.