How to Decide When Reasoning Can Replace Retrieval — and What It Costs

Your router is choosing between a direct model answer and a retrieval pass. GPT-5 and Gemini-3 store 95-98% of tested facts in parameters. Direct recall still misses 26-34% of encoded facts, and thinking recovers 40-65% of those misses. The trade-off is extra tokens and latency for a smaller recovered-fact share. Multiply the miss rate by the recovery rate to size the routing subset.

Run a recall-cost eval on your own fact set before you change the router. It turns a vibes call into a routing rule with a break-even number.

Direct recall sets the floor

Start with the facts your product actually answers. Pull a labeled set from support tickets, product docs, and known user questions. Keep the labels simple: correct, wrong, or not answerable. Make the set reflect production traffic, not a toy benchmark. Include the questions that caused user complaints, not just the cleanest docs.

• Measure direct recall. Ask the model for each fact without thinking. Record a per-fact pass/fail table and a miss rate you can trust.
• Recover the misses with reasoning. Send only the failed facts to a reasoning pass. The follow-up table must show pass/fail, token counts, and latency for each recovered fact.
• Count the facts that need thinking. Divide the facts that reasoning recovers after a direct miss by the total fact set. Only 10-20% of evaluated facts need thinking, so always-on reasoning wastes compute. That share is a routing share, not a global model setting.
• Price the recovered fact. Multiply the routed subset's thinking tokens by your per-million-token price, add any latency penalty, and divide by recovered facts. Compare that number to retrieval cost per answer. The result is a break-even threshold you can enforce in the router.

Turning thinking on for every fact pays for facts that direct recall already handled. Route only the miss set, then only the subset where the answer is worth the extra spend.

Reasoning earns its place on recovered value

Reasoning is strongest when the answer requires chaining, not a stored datum. Use it for the facts that direct recall misses and that still matter to the user. If the missed fact is a low-value detail, retrieval or a fallback message may be cheaper.

Stratify the fact set by user value. A pricing error, a safety constraint, or a workflow-critical lookup deserves a higher recovery threshold than a trivia miss. Give the router a value tier for each fact before it spends tokens.

Re-run the eval when you change the model, the fact set, or the retrieval index. A fact that is cheap to retrieve today may be expensive to reason about tomorrow.

Costs decide the router

Put the cost on the answer, not the model. For each recovered fact, record thinking tokens, prompt tokens, output tokens, and wall-clock latency. Convert latency into the cost your product actually pays: user wait, queue depth, or timeout risk. Use the same units across models and retrieval paths.

Compare the recovered-fact cost against retrieval. Retrieval has its own price: index maintenance, embedding calls, reranking, and the latency of a search pass. If reasoning costs less than that bundle for the routed subset, it can replace retrieval for those facts.

Set a threshold per tier. A high-value fact can absorb more tokens and latency than a low-value one. Store the threshold as a number in the router, not a feeling in the architecture doc.

Build the router around a decision, not a model preference. Return direct recall when it is correct. Route reasoning only after a direct miss on a high-value fact. Fall back to retrieval or a clear uncertainty message when reasoning fails or the cost exceeds the threshold. Log each path so the next eval starts from production evidence, not memory or habit.