Open-Weights-First Model Routing Pipeline

Your model router decides which tokens pay the closed-API premium. Open weights can carry most production token volume at a lower unit cost, while closed APIs remain useful for narrow capability gaps. Open models carry about 80% to 90% of enterprise token volume while accounting for only 10% to 20% of AI budgets. At the midpoint, the remaining 15% of tokens consumes 85% of budget. Use three gates: quality parity, cost per million tokens, and p95 latency. Make open weights the default lane, and route to closed APIs only when a gate justifies the exception.

Default to open weights before buying API capacity

Open-model serving is now a mainstream path. Ollama's platform includes 9 million developers, 178,000 GitHub stars, and adoption spanning 85% of the Fortune 500. AT&T now routes 40% of its token consumption through open-weight models. Ollama Cloud token usage increased 150-fold from the beginning of 2026. Coding agents in early 2026 and the OpenClaw/Hermes agent wave in April 2026 were the two inflection points in Ollama Cloud token usage.

Long agent work changed the calculus. Open-model context windows expanded from 128K to more than 1 million tokens, a key enabler for long, multi-step agent work. That expansion moves long-context work from a closed-API assumption to a measurable routing decision. If your feature sends a long transcript, a codebase, or a multi-step plan, the open route may now handle it without a closed fallback.

• Inventory model calls. Tag each call with feature, token volume, context length, and current provider. Exit: a table where every endpoint has an owner and a measured cost per million tokens.
• Baseline open-weight serving. Stand up a serving path for the open model that will carry the default route. Ship when the endpoint has token counters, error rates, and a latency histogram.
• Shortlist open models. Start from models that already carry real traffic. On Ollama's cloud platform, Chinese-origin models dominate consumption, with DeepSeek leading and GLM close behind. Output: candidates that fit your context length and tooling.
• Measure quality parity. Run the same evaluation set for the open model and the closed model. Pass when the open model meets the feature's quality bar on the tasks that matter.
• Compare cost per million tokens. Put both endpoints on the same input and output mix. Output: a dollar figure per million tokens, not a vendor list price.
• Check p95 latency. Measure under production-like load, not a single warm request. Pass when p95 fits the user-facing deadline.
• Route by feature. Bind each product feature to a model route, not to a vendor name. Ready when a feature can switch routes without a code change.
• Log the decision. Record which gate allowed or blocked each model. The audit trail explains why a request went to the open route or the closed route.

Keep closed APIs as gated exceptions

A failed gate keeps a model out of the default route. A passed gate moves traffic to the open route. Closed APIs remain exceptions. Route by measured quality, cost, and p95 latency, not model name or vendor habit. Use the closed route only when a task needs a capability the open model cannot match, a proprietary data constraint applies, or a latency requirement exceeds the open endpoint. Record the failing gate, the owner, and the review date.

Make the default route boring

Keep the default route predictable. Version the model, pin the serving stack, and test upgrades against the same evaluation set. A rollout should promote, pause, or roll back a new open model without touching product code. Expose the model name, prompt version, token count, and route decision in the same log line. The trace answers why a request took the route it took.

Measure the fallback as a budget line

Report gate results for every model call. Track the share of tokens on open weights, the share on closed APIs, and the cost per million tokens for each route. The dashboard keeps the exception route small enough to justify. Review the exception list on a regular cadence. If an open model now passes the gate, move the traffic back. If the closed route keeps growing, document the product reason and review the gates again.