Pick a Production LLM With a 6-Point Eval

The trade-off in LLM selection is quality against cost, tail latency, and failure tail. Before you call a model better, write the arithmetic: quality gain ÷ (cost increase + latency penalty + downside risk). If the numerator is small and the denominator is large, the model is not production-ready. A useful production model-selection question is which model and reasoning setting meets quality, cost, speed, consistency, and failure-rate tolerances.

Costs

Cost is not the sticker price. It is the expected spend per useful output. Record price in US dollars per thousand tokens, expected input length, expected output length, retry rate, and fallback rate. A model that is cheaper per token but produces longer outputs, needs more retries, or triggers more fallbacks can be more expensive per completed task.

Do not treat higher reasoning effort as a quality dial. Increasing reasoning effort is not a reliable quality dial and can increase cost or reduce scores.

Build a cost model with three columns: input cost, output cost, and failure cost. Failure cost includes retries, human review, support tickets, and lost user time. When finance asks why the AI feature is expensive, this table is the answer.

When presenting to finance, separate variable cost from fixed cost. Variable cost changes with traffic, output length, and retries. Fixed cost includes platform fees, monitoring, and maintenance. A model can look cheap in a demo and become expensive when the feature is used by many users or when the answer length grows. The scorecard should force that comparison before the model is selected.

Models

The best LLM for production is the one that survives a task-specific scorecard. Use six fields: task_fit, eval_score, p95_latency_ms, per-thousand-token cost, license_risk, and fallback_success_rate. A selection method can compare score, cost, time, and downside consistency using a frontier.

Task fit is whether the model's output shape matches the product: classification, extraction, summarization, code, tool calls, or structured JSON. A model can score well on a generic benchmark and still fail your schema. Eval score should come from a small eval set built from production-like examples, not only public LLM benchmarks. Score each candidate with the same rubric, same prompts, same temperature, same max tokens, and same timeout.

Consistency and downside risk should be part of model evaluation because a strong median can hide occasional weak outputs.

License risk is a policy gate, not a benchmark. Record whether the model allows commercial use, data retention, fine-tuning, redistribution, and audit. If security or legal says no, the model is out regardless of score.

Make the eval reproducible. Store the prompt, the response, the score, the latency, the token count, and the failure label. If a candidate improves after prompt tuning, record the tuned prompt as part of the candidate. A model plus prompt plus settings is the production unit, not the model name alone.

Serving

Serving determines whether the model is reachable, observable, and recoverable. Gateway platforms are enterprise-focused options for centralizing LLM access, enforcing security policies, managing compliance, and monitoring usage.

Latency and token output are measurable constraints when evaluating LLM gateways. Record first-token latency and total latency, but also the length of the returned answer. A fast first token is not a fast answer if the model streams a long, low-value response. For user-facing features, p95 latency is the number that matters.

Fallback behavior is the last line of defense. Define what happens on timeout, provider error, policy block, schema failure, or low confidence. A fallback can be a smaller model, a cached answer, a rule-based response, or a human handoff. Measure fallback_success_rate as the share of degraded requests that still return a usable response.

Reliability teams need a runbook, not a hope. For each failure mode, define the detection signal, the fallback action, the user-visible message, and the escalation path. If a provider outage is common, the fallback should be tested in staging with realistic traffic. If the fallback is a human handoff, define the queue and the response window. The goal is to keep the feature useful when the primary model is not.

Run the selection in four steps. First, list the workflow tolerances for quality, cost, speed, consistency, and failure rate. Second, score each candidate on the six fields. Third, compute downside risk and remove candidates that fail the tolerance. Fourth, keep only the non-dominated candidates and choose the one with the lowest operational risk. The model you ship should be the one that is hardest to defend against, not the one that looks best in a demo.