SWE-Serve tests coding agents through live inference

A patch can satisfy the tests around a component while failing the request that reaches it through a running server. NVIDIA’s new SWE-Serve benchmark puts that gap directly into the evaluation of AI coding agents.

In its September 23 report, NVIDIA describes 53 tasks derived from SGLang changes. Nineteen include live-serving checks. Across the same 627 patches for those tasks, 69.4% passed with live-serving checks omitted, compared with 45.9% under complete verification. That is a 23.5 percentage-point difference in this experiment, not an estimate of failure rates across all AI-written software.

Choose the claim before choosing the test

Keep delivering a patch separate from demonstrating that the intended operational risk has changed. Our postmortem follow-through guidance makes that distinction for corrective work. Applied here, it asks the reviewer to name the behavior the patch must establish before accepting the test result as evidence.

A test that recognizes a model configuration does not, by itself, show that a request using that model returns the expected output. The missing evidence depends on the change: initialization, request routing, concurrent state or an output contract may be decisive. Adding every conceivable test would obscure that judgment. Select the path that exercises the claim.

A reproducible benchmark with a bounded scope

The public repository includes executable tasks and verifiers. It specifies Linux, Docker and Python 3.12 or later, with an H100 configuration for GPU tasks. It also distinguishes closed-book runs from runs that can retrieve upstream solutions. Compare results within the same release and access conditions; otherwise a score can reflect a different experiment.

The first release covers SGLang on CPU or one H100. NVIDIA explicitly excludes other engines and multi-GPU or multi-node serving from its reported scope. It also says a benchmark pass does not establish merge or deployment readiness. Those limits matter for teams operating a different inference stack.

Give a green result an explicit meaning

In a hypothetical batching change, a unit test confirms that two outputs exist, while a live request reveals that they are returned in the wrong order. The delivered patch and the customer-visible contract now have different evidence. Retain the failed request as the acceptance case, fix the behavior and repeat that case alongside relevant regression checks.

Write the expected service behavior into the review, then identify the running-system check that could disprove it. Record both the patch’s status and what remains unverified. If the affected behavior does not cross the serving path, choose a focused test that actually covers it instead of requiring a GPU benchmark by default.

For an inference-engineering team, SWE-Serve offers concrete examples of where that boundary belongs. The useful question for the next review is which customer-visible claim would still be untested if every current check turned green.