Re-baseline coding-agent routing on long-horizon scores

Short-horizon boards hide the spread that sets cost

When a coding agent runs for hours, the model choice shows up as attempts per completed task, not leaderboard rank. The production question is which model keeps a long-running agent inside budget. Claude Fable 5.1 led the published FrontierSWE v2 snapshot with a 56.3% mean@5 score. GPT-5.6 recorded a 32.2% score on FrontierSWE v2, 24.09 percentage points below Claude Fable 5.1's 56.29%. Those scores imply expected attempts per success of about 1.78 versus 3.11.

If per-attempt compute is similar, the lower-scoring model can cost far more to finish the same job. That gap is invisible on saturated short-horizon boards, where a model can look interchangeable on easy tasks and then diverge once a task needs sustained debugging, long context, and repeated tool calls. The spread matters because long-running tasks are where retries, wall-clock time, and human escalation become the real cost. The score gap is the input; the budget is the constraint.

The arithmetic turns a score gap into a routing threshold

FrontierSWE v2 contains 34 tasks after adding 21 new ultra-long-horizon challenges. It evaluates each model with five trials per task under a 20-hour budget in the Proximus harness. GLM-5.3 recorded a 30.2% score on FrontierSWE v2. That fixed setup exposes the spread that short-horizon coding model comparison often compresses, because the tasks require sustained work rather than one-shot completion.

Convert the score gap into failure rates: 43.7% versus 67.8%. The failure rate is the probability that a trial does not finish the task. With similar per-attempt cost, the lower-scoring model needs more attempts and more budget to finish the same task. Treat those numbers as a cost model. A router that trusts saturated scores will keep sending long tasks to the wrong model, so the threshold should come from the expected cost delta.

Routing thresholds should come from long-horizon reliability

• Pick a long-horizon agent benchmark with unsaturated scores and a fixed harness.
• Fix the trial count, task budget, and evaluation environment before scoring.
• Score mean@k so the number reflects reliability.
• Convert the score gap into failure-rate deltas and expected cost per completed task.
• Set routing or fallback thresholds from the cost delta.

Start with the task class. Long-running coding and research agents need a different baseline than short completions. If the work involves multi-file changes, test loops, dependency fixes, or repeated verification, use the long-horizon score as the primary input to model routing. If the work is short and bounded, a saturated board may still be useful, but it should not set the fallback threshold.

Set the threshold from expected cost. Route to the primary model when its expected cost per completed task stays below budget. Route to a cheaper model only when the cost delta is smaller than the escalation risk, including retry cost, wall-clock cost, and the chance that a failed attempt still produces a partial artifact worth keeping. The fallback decision should be explicit enough that an engineer can explain why traffic moved and when to move it back.

Make the constraint set explicit in the routing config: benchmark version, harness version, task list, budget, trial count, environment, and scoring rule. If any of those change, re-run the comparison and the cost model. A score is comparable only when the trial count, task budget, and environment are the same, so the next engineer should be able to reproduce the number from the config.

Monitor completion rate, attempts per completed task, wall-clock per completed task, and human escalation rate by model. When the production gap moves away from the benchmark gap, the benchmark is no longer the right proxy. Re-baseline then. Keep the threshold review tied to the same task class, because a routing rule that mixes short and long tasks will drift. Use the model whose expected long-horizon cost per completed task is lowest under your current budget; if the cost delta is small, keep the cheaper model, and if it is large, move traffic before the next budget review.