SWE-Serve asks whether the patch works through the server

Two callers ask different questions. The server produces two answers and sends each to the wrong caller. A test that checks only the number of outputs can pass while both people receive the wrong result.

That is a hypothetical failure, but it illustrates the question behind NVIDIA’s SWE-Serve benchmark: what happens when an AI coding agent’s patch must work through the running inference service? In NVIDIA’s September 23 report, the same 627 patches on 19 tasks passed 69.4% of the time without live-serving checks and 45.9% with complete verification.

The difference is 23.5 percentage points in that experiment. It is not a general failure rate for AI-written code. Using the same patches makes the point more specific: adding checks for serving behavior changed which patches counted as working.

Follow each question to its own answer

A component check might establish that configuration was accepted or that a function returned two outputs. A request through the running server also encounters initialization, routing, shared state and delivery. More assertions about the output count would still miss the swapped callers.

For the fictional batching fault, send distinguishable requests through the serving path and verify which result each receives. The count remains useful; now it sits beside the identity check it cannot replace.

The postmortem follow-through guidance treats delivering a correction and demonstrating the desired outcome as separate decisions. Write down the behavior the patch must establish before choosing the evidence. Otherwise the available test can quietly become the definition of success.

Keep the benchmark conditions beside the score

SWE-Serve’s first release contains 53 tasks derived from SGLang changes; 19 use live-serving checks. The public repository supplies tasks and verifiers, with Linux, Docker and Python 3.12 or later in its setup and an H100 configuration for GPU tasks.

It distinguishes closed-book runs from runs allowed to retrieve upstream solutions. Those conditions matter when comparing agents. A score can change because the agent changed, because it received different information, or because the required behavior changed.

This release covers SGLang on CPU or one H100. Other engines and multi-GPU or multi-node serving are outside the reported scope, and NVIDIA separates benchmark success from merge or deployment readiness. A larger service can learn from the missed-check mechanism without calling the published score validation of its own topology.

Choose the check that crosses the changed behavior

In the two-caller example, the missing evidence is the route from each request to its answer. That route is why a live check helps. If a patch changes behavior entirely outside serving, a focused test may already reach the relevant failure and a GPU benchmark may add cost without useful coverage.

Keep the claim and the exercised path together in review. The point of SWE-Serve’s two verdicts is not that one type of test is always virtuous. It is that a green result answers only the question the test actually asked, and sometimes the customer is waiting on the question it left out.