Changing a model changes software people already depend on. It deserves the same scrutiny as any other production change.
The announcements make a reasonable case for another look. Anthropic’s Sonnet 5.5 release reports lower cost per task through reduced token use. OpenAI’s GPT-6.1 Sol announcement reports performance approaching Astra on several evaluations at lower token prices. For a team already running agents, the useful question is how those gains carry over to its own workload.
At Riften, we think a model should earn more responsibility by handling specific work well. A decision to change the default should be explainable in terms of the tasks that improved, the checks they passed, and the cost of completing them.
Keep the difficult case
Consider a database migration that stops halfway through. Some records have been copied into the new schema. Others remain in the old one. The API still expects to read both. An engineer needs a repair that can finish the migration without losing records or creating duplicates.
Give a model an empty database and this problem largely disappears. It can produce a patch that looks convincing and passes every test, while the condition that made the work difficult was never present.
A useful environment preserves that condition. Start each attempt from the same partially migrated database. Ask for the repair. Check the records afterward, run the migration again, and call the API. The requirements should be in the task; the tests should allow different implementations that satisfy them.
That last detail matters. In its July audit of SWE-Bench Pro, OpenAI estimated that roughly 30% of tasks were broken. Some tests demanded implementation details the instructions never specified. Others let incomplete fixes pass. The estimate applies to that benchmark, but the failure modes are useful questions to ask of any test we write.
Read the run
For the migration example, the evidence includes the patch, the database state, and the commands used to get there. Anthropic’s evaluation guide distinguishes an agent’s transcript from the outcome left in its environment. Both are worth inspecting: one shows what the agent did; the other shows whether the work is complete.
A correct answer can also arrive by a route the evaluator never intended. In Anthropic’s BrowseComp investigation, Opus 4.6 identified the benchmark and decrypted its answer key in two cases within a 1,266-problem multi-agent evaluation. Those runs could not establish the research ability the benchmark was meant to measure.
For a coding environment, we would keep the grader outside the agent’s editable workspace and inspect suspicious successes as closely as failures. A passing check deserves an explanation when the patch appears unrelated to the task.
Repeat the task from a fresh copy of the starting state. The τ-bench researchers made consistency an explicit part of evaluation with passk, which measures success across all k trials. For an unattended workflow, repeated success matters because a person will not be there to select the best attempt.
Change what the evidence supports
A model comparison should account for the whole attempt: tool calls, retries, timeouts, and any human repair. A lower token price is useful, but the relevant bill is the cost of getting the job done.
The decision can be narrow. Suppose a candidate handles document extraction well but still needs intervention on interrupted migrations. There is a case for assigning it the extraction work and continuing to test the migration workflow. Neither result establishes how it will perform on every other task.
We would start with a limited rollout, retain a way back to the previous configuration, and compare live outcomes with the test results. When production exposes a new failure, preserve enough of the case to reproduce it safely.
The next model should face that case too. Over time, this gives a team a record of what its software must handle and a concrete basis for deciding when to change it.