SWE-Serve tests coding agents through live inference

Imagine asking a server for two model outputs and receiving both, attached to the wrong requests. A test that counts the outputs would be satisfied. The people who asked the questions would have a rather different assessment. That hypothetical mismatch is a useful way into SWE-Serve, NVIDIA’s benchmark for testing whether an AI coding agent’s patch works through the running inference service.

In its September 23 report, NVIDIA describes 53 tasks drawn from SGLang changes, including 19 with live-serving checks. For the same 627 patches on those tasks, omitting the live checks produced a 69.4% pass rate. Complete verification produced 45.9%. The 23.5 percentage-point gap belongs to this experiment; it is not a failure-rate estimate for all AI-written code. What makes it interesting is that the patches did not change. The question asked of them did.

What did the green test actually establish?

A component check can establish that a configuration is recognized or that a function returns the expected number of results. Serving adds the path that puts those pieces to work: initialization, request routing, concurrent state and the final output contract. A change can satisfy the local assertion while failing somewhere along that path. More checks help when they reach the missing behavior, rather than merely asking the existing question several more ways.

The two-output example makes the distinction concrete. Counting results tests one property. Sending requests through the server and checking their corresponding outputs tests another. Delivering the patch and demonstrating the intended service behavior therefore need separate acceptance decisions. Our postmortem follow-through guidance uses that distinction for corrective work: first state what the correction should change, then choose evidence capable of showing it.

How far can this result travel?

The benchmark’s public repository provides executable tasks and verifiers, so the reported distinction is more specific than a general warning to test carefully. Its setup calls for Linux, Docker and Python 3.12 or later, with an H100 configuration for GPU tasks. It also separates closed-book runs from runs allowed to retrieve upstream solutions. Those conditions belong beside a score because changing the available hardware or access to solutions changes the experiment being compared.

There is a corresponding limit to what a pass buys. This first release covers SGLang on CPU or one H100. NVIDIA excludes other engines and multi-GPU or multi-node serving from the reported scope, and explicitly separates benchmark success from merge or deployment readiness. A team running a larger serving topology can learn from the missing-test mechanism without treating the published result as validation of its own stack.

The batching test must preserve request identity

In the hypothetical batching change, the corrected implementation must preserve each output’s request identity as well as pass its relevant regression checks. The live request supplies a reviewable case for that requirement: the original local check counted outputs but did not establish that they reached the right requests.

If a change’s claimed behavior never crosses the serving path, a focused test can be sufficient; requiring a GPU benchmark would add cost without necessarily adding relevant evidence. For a change that does cross it, record the running-system behavior exercised and the parts still unverified. SWE-Serve’s different verdicts on the same patches show why that choice of evidence matters.