On this page4 sections
A coding-agent leaderboard has a winner. You can try the winning product and see whether it helps your team. But suppose you want only its model, plugged into the tool you already use. Will the advantage come with it? The score alone cannot tell you which parts of the setup earned it.
A discussion of scientific-code parallelization points to an evaluation project that runs models through tools including Codex CLI and Claude Code. Its methodology records those differences. Reading them alongside the model name changes the question from “Which model won?” to “What was actually doing the work?”
Who put the failing assertion in front of the model?
When a model asks to inspect a file, run a command or edit code, surrounding software handles that work and returns the result. That software is often called the harness. It decides how to present failures and long outputs, whether an edit gets another attempt and when the task stops.
Imagine giving the same model and repository to two harnesses. The job is to repair a function failing a unit test. One returns the failed assertion and the nearby stack trace. The other returns a long command transcript, cut short before the useful failure reaches the model.
The second setup may spend another turn searching for an assertion the first already supplied. The model’s reasoning ability has not changed; what it can see has. File selection, summaries of earlier work and stopping rules can create similar differences before the next edit is made.
The testing policy adds another difference. Running the full suite after every edit adds checks and waiting. Running no tests may make generation look quick while leaving someone else to validate the result. Both affect how long the team waits for useful code, and neither is explained by the model name alone.
Decide whether you are choosing a product or a model
If the decision is which complete product to use, compare the products as the team would use them. Give them comparable tasks and starting repositories with reasonable settings. Better file handling and smoother test integration are legitimate product advantages, not distractions to remove from the result.
If the decision is which model to put inside an existing tool, hold the surrounding harness, tools, instructions, retry allowance and checks as steady as the APIs allow. Provider differences may make a perfectly identical experiment impossible. Record those differences so the comparison says what it actually measures.
In the diagram, read down a column to compare models in the same harness, or across a row to compare harnesses using the same model. Moving diagonally changes both. That diagonal can identify a better complete setup without telling us how much of its advantage came from the model.
You do not have to turn every purchase into a research program. A complete-product comparison can be enough to choose that product. Just do not expect the model to carry all the surrounding software’s advantages into a different home.
Nine minutes to code is not always nine minutes to a result
Consider another hypothetical task. Agent A parallelizes a calculation in nine minutes; Agent B takes fourteen. Both compile and pass the small tests. On a larger input, A’s changed calculation order moves the answer outside the permitted numerical tolerance. B stays within it.
At fourteen minutes, B has produced acceptable code for that input. A has not. If A needs seven more minutes of repair, its accepted result takes sixteen minutes before review. If it never passes, there is no accepted-result time to report. Keep the generation time, but do not let it answer a question about completed work.
The scientific-code project separates generation, validation and performance testing. Programs must pass output comparison before qualifying for performance measurement. Those stages explain what a time chart includes, and which programs never reached the measurement being shown.
The project’s October 19 Luna 6 Medium comparison in Codex and OpenCode makes the distinction concrete. Each setup produced 220 programs. Codex had 171 validation passes and OpenCode 181, but 150 and 149 respectively completed performance benchmarking. Some programs that validated failed under the benchmark configuration. The campaigns also ran on different dates with provider aliases, so this describes complete setups rather than isolating a causal harness effect.
Keep final acceptance outside the agent’s authority to redefine. It can run tests while working, but an independent clean run should judge the assigned requirement. The live-inference testing guide explores that separation for serving changes. A candidate should not get to turn its own changed expectations into the definition of success.
Bring a few real jobs to the shortlist
Choose representative test repairs, dependency updates and performance changes. Look at those kinds of work separately before combining them into one score. Start from the same clean commit, state what must pass and how much time or repair is allowed, and repeat where practical. An inconsistent result is part of what the team needs to learn.
| Keep with the trial | The question it answers |
|---|---|
| Model, harness version and relevant settings | Which complete setup did this work? |
| Starting commit and assigned task | Did the candidates begin with comparable work? |
| Final test command and its result | Did the result meet the shared requirement? |
| Elapsed time, reported usage or cost, and retries | How much did the repairs add to the first attempt? |
| Human intervention and unsuccessful attempts | What still needed an engineer, or never finished? |
The project calculates API cost estimates from token volumes and pricing profiles; these are not observed bills. Cached input, uncached input and output may have different rates. Keep estimates distinct from actual charges, and keep the engineer’s time visible rather than treating it as free because it is not in a token report.
The leaderboard has done a useful job if it narrows the choices. Your own trials show whether the advantage survives the tasks, waiting and repairs your team faces. You can choose a setup that repeatedly finishes well without pretending to know precisely how much credit each part deserves. That answers the purchasing question honestly, even when the causal question remains open.
Source context
Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.
Report an error or outdated detail