NVIDIA measures confidential inference, but replacement workers need their own test

An operator brings up a replacement worker, but the protected service will not send sensitive requests to it yet. The worker must first provide evidence that its hardware and software match the state the service trusts. Until that check succeeds, an idle GPU can sit beside busy existing workers without adding protected serving capacity.

That hypothetical replacement is a useful companion to NVIDIA’s September 22 confidential-inference benchmark. On eight B200 GPUs, enabling confidential computing retained 96.1% to 98.2% of baseline output-token throughput and increased mean time per output token by 1.2% to 4.3%. These are NVIDIA’s measurements, not an independent AIOpsSRE test.

Put the measured workload beside the percentages

The experiment used DeepSeek-R1-0528-NVFP4 with TensorRT LLM’s PyTorch backend, 32K input and 1K output lengths, and concurrency from one to sixteen. It kept the workload and configuration fixed while changing confidential-computing state. Its software included a TensorRT LLM release candidate, so the build belongs beside the hardware in any reproduction.

This is a comparison of a particular model representation, runtime and request shape, not a clock on our waiting replacement. Mean time per output token describes generation once underway; it does not tell the operator when a user first sees a token or how bad unusually slow requests become. Those need their own measurements. Even excellent generation numbers leave unanswered how long the new worker takes to become ready.

The AI infrastructure evaluation applies the same rule: identify the tested workload before asking a published result to support a local capacity decision.

The runtime helps produce the result

While a worker serves requests, protected execution also changes work around inference, including memory movement, timing and communication between GPUs. NVIDIA’s asynchronous sampler change moves readback work away from the main execution path. Rather than leaving that work on the central path that drives inference, the adaptation lets it proceed asynchronously. The published result therefore belongs to the runtime as well as the GPU; installing the same GPU name does not reproduce the same sequence of work.

For the waiting replacement, the first question is not token speed but whether its configuration is allowed. NVIDIA’s Trusted Computing documentation connects deployment and attestation guidance with a matrix of GPUs, firmware, drivers and operating modes. That evidence-producing process is attestation. It gives the service something to compare with its trusted hardware and software state, instead of accepting a worker merely because it has started.

It does not decide whether a particular application may process a request, or whether the model’s answer is good. For the replacement worker, it answers the earlier question: is this the environment we agreed to accept sensitive work into?

Test the return of protected capacity

In the hypothetical pool, existing workers may continue serving while the replacement waits. The team now has both an admission issue and less available capacity. Steady-state throughput from already-admitted workers cannot establish how that replacement should recover.

Suppose the replacement’s required trusted-state evidence is unavailable. The operator needs to distinguish that admission failure from a worker that has been admitted but generates tokens slowly. A replacement or upgrade trial can follow both: the allowed configuration and evidence before sensitive requests enter, then the request mix, load range, completed work and relevant latency limits while serving. The same missing worker can create a capacity problem without invalidating the existing workers’ steady-state performance result.

If policy requires confidential computing for customer material, use approved synthetic data or another permitted substitute on the unprotected side of a comparison. Disabling the protection on that material merely to make a symmetrical chart changes the authorization conditions of the test.

The operator can now read “fast enough” without mistaking it for “ready to receive protected work.” The existing configuration may serve the tested workload well while the replacement’s admission remains unresolved. NVIDIA’s percentages help estimate the serving cost; the replacement trial follows the separate wait before a new machine is allowed to contribute. Both matter to the requests still arriving.