On this page3 sections
Imagine bringing a replacement machine online while the rest of an AI service is busy. Its GPU, the chip that will run the model, is idle and ready for something to do. Yet the service sends it no sensitive requests. Before it can help, the machine must show that its hardware and software match a state the service trusts. Running and allowed to receive private work are two different milestones.
That hypothetical wait sits just outside the question answered by NVIDIA's September 22 confidential-inference benchmark. With protection enabled on eight B200 GPUs, NVIDIA measured 96.1% to 98.2% of the unprotected baseline's output-token throughput, the rate of generated pieces of text. Mean time per output token increased by 1.2% to 4.3%. These are NVIDIA's results, not an independent AIOpsSRE test. They describe the cost of serving under confidential computing, which protects data while it is being processed, rather than the wait to admit our replacement.
What was the machine being asked to do?
NVIDIA used DeepSeek-R1-0528-NVFP4 with the PyTorch backend of TensorRT LLM, software that runs the model. Requests had 32K input and 1K output lengths, measured in tokens, and the number running concurrently ranged from one to sixteen. The comparison held the workload and configuration fixed while changing whether confidential computing was enabled. TensorRT LLM was a release candidate, so reproducing the result means keeping track of that software build as well as the eight chips.
A person watching an answer appear has more than one reason to notice a delay. They may wait for the first piece of text, or encounter an unusually slow request after several ordinary ones. Mean time per output token describes generation once it is underway; it does not measure either of those experiences on its own. Nor does it time the replacement machine's journey into service. Those questions need their own measurements, even when the generation numbers look good.
The AI infrastructure evaluation makes the same distinction from the capacity side: a published result becomes useful to a local decision when the tested workload is clear. Here that includes the particular model representation, runtime and request shape, not just the B200 name.
The software has work to rearrange too
Protecting execution changes more than the calculation on a chip. Data still moves through memory, operations still need timing, and GPUs still talk to one another. NVIDIA's asynchronous sampler change shifts repeated readback work, which brings generated-token and sampling data back from the device, to an asynchronous worker. That lets it proceed without holding up the main execution path in the same way. The result belongs partly to this arrangement of work in the runtime. Buying the same GPU does not reproduce the arrangement.
Our waiting replacement has an earlier hurdle. NVIDIA's Trusted Computing documentation brings together deployment guidance, supported combinations of GPUs, firmware, drivers and operating modes, and the process called attestation. Attestation produces information about the machine's hardware and software state that a service can check against what it trusts. It is how the service gets something more useful than the machine announcing that it has started.
That check has a specific job. It does not grant an application permission to process every request, and it does not judge the model's answer. It helps answer whether this is an environment the service has agreed to accept sensitive work into. Our replacement can be quick at generating text and still fail that earlier requirement.
Follow the replacement all the way into service
In the hypothetical pool, the existing workers keep serving while the newcomer waits. The team has less usable capacity than the powered-on hardware count suggests. That does not invalidate the existing workers' steady-state throughput. It means their result cannot tell the team how to get the replacement admitted.
Suppose the required trusted-state information is unavailable. That failure belongs before admission, rather than in a report about slow token generation after admission. A proposed replacement or upgrade trial can follow the two stages separately: first the allowed configuration and required checks before sensitive requests enter, then the request mix, load range, completed work and relevant latency limits while the worker serves. Keeping the stages distinct makes the missing capacity easier to explain and the next action easier to choose.
The comparison also has to respect what the data is allowed to do. If customer material requires confidential computing, the unprotected side needs approved synthetic data or another permitted substitute. Switching off protection around that material for a tidier comparison would change the test's authorization conditions, not merely its performance settings.
When the operator looks back at the busy pool, both observations can be true: the existing configuration serves its tested workload well, and the replacement is not yet allowed to help. NVIDIA's percentages put a number on the serving cost. A replacement trial follows the separate wait before another machine can contribute. The requests arriving during that wait are why the distinction matters.
Sources & context
Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.
- confidential-inference performance resultsdeveloper.nvidia.com
- asynchronous sampler changegithub.com
- Trusted Computing documentationdocs.nvidia.com
