A benchmark cannot separate the model from its harness

A coding-agent benchmark that gives each model its own agent harness measures two systems, not two isolated models. That comparison can still be useful. It answers which bundle completed the work under the reported conditions. It does not show how much of the difference came from the model, tool loop, prompts, context management, retry policy, or test strategy.

A recent r/codex discussion highlighted this problem around a scientific-code autoparallelization benchmark. The reported summary compared GPT-6.1 Sol using one agent harness with Claude Opus 5.5 using another. The author explicitly noted that the harnesses differ. Reported mean time and API cost therefore describe those tested bundles, not a controlled model-only effect.

Delivering the patch and demonstrating the intended service behavior therefore need separate acceptance decisions. A fast agent result matters only after an independent build and workload test establishes that the parallelized program still computes the intended result.

Name the question before reading the rank

Decide what quantity the benchmark is meant to estimate. A system-level question asks, “Which available agent setup gets this repository task to a verified result with less time or cost?” For that decision, letting each product use its native harness may be realistic. The result belongs to the whole setup.

A model-level question asks, “How does the model change the outcome when the surrounding agent is held constant?” That requires the same tool interface, prompt policy, context budget, stopping rule, retry allowance, and verifier. Even then, provider-specific APIs and reasoning controls can make perfect equivalence impossible. The limitation should be part of the result rather than hidden in a model leaderboard.

The published local LLM evaluation project documents a broader research program and links the Euro-Par 2026 paper “Evaluating the Parallelization Capabilities of State-of-the-Art Agentic Large Language Models.” Its artifacts are more useful than a screenshot because they give readers a method and a citation to inspect. The Reddit summary remains a discovery source, not independent replication.

Treat the harness as measured software

An agent harness decides what the model sees and what it can do. It may summarize files before the first call, choose which tools are available, truncate command output, retry a failed edit, run tests automatically, or stop after a token budget. Each choice can change time, cost, and success.

Capture the harness version and configuration with the model identifier. Record the initial prompt, tool schemas, maximum turns, reasoning setting, context policy, retry policy, timeout, repository state, and exact verifier command. Keep per-run traces sufficient to classify failures without exposing credentials or private source.

Separate model calls from harness work. Report wall time and API cost for the complete system, then also report model-call count, input and output usage, tool time, test time, retries, and human interventions. A harness that runs a long test suite after every small edit may make a capable model look slow. A harness that skips tests may look fast by moving verification cost outside the benchmark.

Project-level work such as ParBench illustrates another design choice: provide common build, execution, and verification infrastructure while asking models to transform the kernels. That does not make it the one correct benchmark. It makes the controlled object clearer.

Use paired tasks and independent verification

Run every candidate from the same clean repository state and task specification. Randomize execution order when shared infrastructure or time-of-day load could bias the result. Repeat tasks enough to show variance, not merely a mean. Preserve failures, timeouts, and invalid patches in the denominator.

Verification should be outside the agent's control. Compile from a clean environment, run correctness cases, compare numerical tolerances, check performance on the intended hardware, and inspect unsupported changes. The agent may run the same tests while working, but its declaration of success is not the acceptance record.

Consider a hypothetical kernel task where Agent A produces a patch in nine minutes and Agent B takes fourteen. Agent A changes the reduction order and passes the small examples but exceeds the accepted numerical error on a large input. Agent B passes both the correctness envelope and the throughput threshold. The generation-time rank favors A; time to verified solution favors B. The example is illustrative, not a claim about the reported benchmark.

Define timeout handling before the run. If a candidate fails a build, decide whether the harness may repair it and how that retry is charged. If infrastructure fails, classify it separately but do not silently rerun only the losing candidates. The benchmark should make its second chances visible.

Design two comparisons when the decision matters

For a local procurement or workflow decision, run a native-system track and a controlled track. The native track uses each product as a practitioner would reasonably configure it. It answers which available bundle works better for the team. The controlled track uses one harness where possible and varies the model. It helps explain whether the rank persists when orchestration is held steadier.

Do not combine the tracks into one score. Label every table by model, harness, configuration, hardware, task set, and verifier. If only the native track is feasible, say so plainly and keep conclusions at the system level. “This setup was faster in these tasks” is defensible. “This model is faster” is not established when the harness changed too.

The site's live-inference agent test guide uses a related acceptance boundary for serving changes. Patch delivery and demonstration of intended service behavior are separate decisions. The incident-response evaluation shows why the observer that accepts recovery must remain independent of the agent.

Publish a decision record, not a winner

A useful benchmark report ends with the decision it supports. State the task population, what remained fixed, what changed, which costs were included, how failures were counted, and what the verifier proved. Include enough run-level data to show spread without claiming that one small task set represents all coding work.

Stop short of a model attribution when a harness difference could plausibly change the result. That boundary changes the recommendation from selecting a model to selecting or retesting a complete agent setup. The restraint does not weaken the benchmark. It tells the reader what evidence to gather next.

Agent benchmarks are operational experiments. Models, harnesses, repositories, tools, and verifiers all participate in the outcome. Treat each as a versioned factor, keep acceptance independent, and report time to verified solution. Then a fast result can become a defensible engineering decision instead of a fragile leaderboard claim.