Opens in a new tab
Explainer

Which part of the coding agent won the benchmark?

In brief

The winning setup may be exactly the product you want. Replacing only its model is a different experiment, because the software around it helps decide how the work gets done.

5 min read

Is this the right starting point?
Before you read
Basic coding-agent use, repositories and test-based code review.
What you will take away
Read model/harness benchmarks correctly and compare complete accepted coding tasks under controlled conditions.
Sources
Two runners approach the same finish with visibly different equipment: one wears a tool belt and the other carries a large loaded backpack.
On this page4 sections

A coding-agent leaderboard has a winner. You can try the winning product and see whether it helps your team. But suppose you want only its model, plugged into the tool you already use. Will the advantage come with it? The score alone cannot tell you which parts of the setup earned it.

A discussion of scientific-code parallelization points to an evaluation project that runs models through tools including Codex CLI and Claude Code. Its methodology records those differences. Reading them alongside the model name changes the question from “Which model won?” to “What was actually doing the work?”

Who put the failing assertion in front of the model?

When a model asks to inspect a file, run a command or edit code, surrounding software handles that work and returns the result. That software is often called the harness. It decides how to present failures and long outputs, whether an edit gets another attempt and when the task stops.

Imagine giving the same model and repository to two harnesses. The job is to repair a function failing a unit test. One returns the failed assertion and the nearby stack trace. The other returns a long command transcript, cut short before the useful failure reaches the model.

The second setup may spend another turn searching for an assertion the first already supplied. The model’s reasoning ability has not changed; what it can see has. File selection, summaries of earlier work and stopping rules can create similar differences before the next edit is made.

The testing policy adds another difference. Running the full suite after every edit adds checks and waiting. Running no tests may make generation look quick while leaving someone else to validate the result. Both affect how long the team waits for useful code, and neither is explained by the model name alone.

Decide whether you are choosing a product or a model

If the decision is which complete product to use, compare the products as the team would use them. Give them comparable tasks and starting repositories with reasonable settings. Better file handling and smoother test integration are legitimate product advantages, not distractions to remove from the result.

If the decision is which model to put inside an existing tool, hold the surrounding harness, tools, instructions, retry allowance and checks as steady as the APIs allow. Provider differences may make a perfectly identical experiment impossible. Record those differences so the comparison says what it actually measures.

A two-by-two matrix crosses two models with two harnesses; diagonal cells are observed system bundles, while same-harness comparisons help isolate model effects before independent tests.
A conceptual two-by-two comparison. Changing both model and harness compares complete systems; changing one factor offers a controlled contrast. Every candidate still needs the same independent acceptance tests.

In the diagram, read down a column to compare models in the same harness, or across a row to compare harnesses using the same model. Moving diagonally changes both. That diagonal can identify a better complete setup without telling us how much of its advantage came from the model.

You do not have to turn every purchase into a research program. A complete-product comparison can be enough to choose that product. Just do not expect the model to carry all the surrounding software’s advantages into a different home.

Nine minutes to code is not always nine minutes to a result

Consider another hypothetical task. Agent A parallelizes a calculation in nine minutes; Agent B takes fourteen. Both compile and pass the small tests. On a larger input, A’s changed calculation order moves the answer outside the permitted numerical tolerance. B stays within it.

At fourteen minutes, B has produced acceptable code for that input. A has not. If A needs seven more minutes of repair, its accepted result takes sixteen minutes before review. If it never passes, there is no accepted-result time to report. Keep the generation time, but do not let it answer a question about completed work.

The scientific-code project separates generation, validation and performance testing. Programs must pass output comparison before qualifying for performance measurement. Those stages explain what a time chart includes, and which programs never reached the measurement being shown.

The project’s October 19 Luna 6 Medium comparison in Codex and OpenCode makes the distinction concrete. Each setup produced 220 programs. Codex had 171 validation passes and OpenCode 181, but 150 and 149 respectively completed performance benchmarking. Some programs that validated failed under the benchmark configuration. The campaigns also ran on different dates with provider aliases, so this describes complete setups rather than isolating a causal harness effect.

Keep final acceptance outside the agent’s authority to redefine. It can run tests while working, but an independent clean run should judge the assigned requirement. The live-inference testing guide explores that separation for serving changes. A candidate should not get to turn its own changed expectations into the definition of success.

Bring a few real jobs to the shortlist

Choose representative test repairs, dependency updates and performance changes. Look at those kinds of work separately before combining them into one score. Start from the same clean commit, state what must pass and how much time or repair is allowed, and repeat where practical. An inconsistent result is part of what the team needs to learn.

Keep with the trialThe question it answers
Model, harness version and relevant settingsWhich complete setup did this work?
Starting commit and assigned taskDid the candidates begin with comparable work?
Final test command and its resultDid the result meet the shared requirement?
Elapsed time, reported usage or cost, and retriesHow much did the repairs add to the first attempt?
Human intervention and unsuccessful attemptsWhat still needed an engineer, or never finished?

The project calculates API cost estimates from token volumes and pricing profiles; these are not observed bills. Cached input, uncached input and output may have different rates. Keep estimates distinct from actual charges, and keep the engineer’s time visible rather than treating it as free because it is not in a token report.

The leaderboard has done a useful job if it narrows the choices. Your own trials show whether the advantage survives the tasks, waiting and repairs your team faces. You can choose a setup that repeatedly finishes well without pretending to know precisely how much credit each part deserves. That answers the purchasing question honestly, even when the causal question remains open.

Source context

Source links appear within this article. Read them alongside the author’s analysis and evaluate the guidance against your environment.

Report an error or outdated detail

More on OpenAI

All OpenAI coverage

Keep reading

Where to go next