NVIDIA measures the overhead of confidential inference

A private inference service needs a useful answer to two questions: can it serve requests quickly enough, and can it do so in the protected environment those requests require? NVIDIA’s September confidential-computing results speak to the first question for a specific configuration. A replacement worker is a good way to see why the second still needs its own answer.

NVIDIA published results for an eight-B200 system on September 22. With confidential computing enabled, the system retained 96.1% to 98.2% of baseline output-token throughput, while mean time per output token rose by 1.2% to 4.3%. These are the vendor’s measurements, not an independent AIOpsSRE benchmark. They make the performance cost of protected execution concrete without establishing every property of a production service.

Using those percentages starts with matching the tested workload and software to the service under consideration. The AI infrastructure evaluation makes the same evidence-scope question explicit. Here the model, request lengths and concurrency show which performance comparison NVIDIA actually made; the replacement-worker exercise later asks what happens when protected capacity must return.

The model, input length and concurrency in the test

The experiment used DeepSeek-R1-0528-NVFP4 with TensorRT LLM’s PyTorch backend, 32K input and 1K output lengths, and concurrency from one to sixteen. It held the workload and system configuration constant while changing confidential-computing state. The software also included a TensorRT LLM release candidate, making the exact build part of any reproduction.

These details explain what the throughput comparison can help a team predict. They also reveal the next questions. A service whose users care about the first token needs a first-token measure; one constrained by slow requests needs more than a mean per-token time. Neither result establishes how long a replacement worker takes to become usable. Additional measurements are worth collecting when their answer could change the deployment choice.

Why the software build belongs in the result

Confidential computing changes the runtime work surrounding inference as well as the protection of the environment. NVIDIA identifies memory transfers, timing and multi-GPU communication as areas requiring aware behavior. Its asynchronous sampler change, for example, moves readback work away from the main execution path. That is a specific reason not to treat a GPU name as a complete description of the tested system.

The protection side has its own configuration dependencies. NVIDIA’s Trusted Computing documentation links deployment and attestation guidance with a compatibility matrix for GPU, firmware, driver and operating modes. Attestation provides evidence about the environment’s trusted state. It cannot decide whether an application is authorized to process a particular request, or whether the model’s answer is good. Those remain separate checks even when attestation succeeds.

A replacement worker that cannot establish trusted state

In a hypothetical private inference service, the existing pool meets its throughput target, but a replacement worker cannot establish the required trusted state. The original workers may keep serving. The replacement cannot yet accept sensitive work under the service’s protection policy, so the team has both a protection decision and a capacity shortfall. A good steady-state benchmark does not resolve either simply by remaining true.

This is where a local test can add the evidence the deployment actually needs. The performance claim identifies the request mix, load range, latency limits and completed-request count. The protection claim identifies the allowed configuration and the trusted-state evidence required before sensitive requests enter it. Exercising replacement or upgrade then shows what the workflow does when that evidence is unavailable.

If the data policy requires confidential computing, the unprotected side of a comparison must use approved synthetic data or another permitted substitute. That limits the experiment, and the report should say so. Disabling a required protection on customer material to make the chart symmetrical would change the conditions under which the work was authorized.

The result may support a useful, qualified decision: the supported configuration has acceptable steady-state performance, while restoration of protected capacity still needs work. Keeping rejected cases and recovery limits with the measurements lets the service owner see that distinction. NVIDIA’s percentages can then inform the choice they actually address, while the replacement exercise answers whether the service can continue meeting both commitments after a change.