NVIDIA’s confidential inference results need a workload boundary

A benchmark can show that protected inference runs quickly without settling whether a particular service is ready to handle sensitive requests. The distinction matters when one result becomes the justification for a production rollout.

On September 22, NVIDIA published confidential-inference performance results for an eight-B200 system. The reported confidential-computing configuration retained 96.1% to 98.2% of baseline output-token throughput. Mean time per output token increased by 1.2% to 4.3%. These are NVIDIA’s measurements, not an independent AIOpsSRE benchmark.

Write the deployment claim before choosing the test, and keep performance acceptance separate from the claim that sensitive work is protected. This is the same distinction used in our AI infrastructure evaluation: a successful configuration and a demonstrated operational claim are separate pieces of evidence.

What NVIDIA’s experiment actually covers

The reported workload used DeepSeek-R1-0528-NVFP4, TensorRT LLM’s PyTorch backend, 32K input and 1K output lengths, and concurrency from one to sixteen. The comparison held the workload and system configuration constant while changing confidential-computing state. The software included a TensorRT LLM release candidate, so the exact version matters when reproducing it.

The useful takeaway is a scoped comparison. A mean per-token result does not establish the slowest-request behavior, first-token delay or the time a replacement worker takes to become usable. Those are different questions. Choose the ones that could change your service decision, rather than expanding the benchmark into a promise it did not test.

Protected execution changes the runtime

NVIDIA identifies memory transfers, timing and multi-GPU communication as areas needing confidential-computing-aware behavior. For a concrete implementation example, its asynchronous sampler change moves readback work away from the main execution path. That is a reason to record the framework build with the result, not merely the GPU name.

The Trusted Computing documentation separately links deployment instructions, attestation guidance and a compatibility matrix for GPU, firmware, driver and operating modes. Attestation supplies evidence about an environment’s trusted state. It does not replace application authorization or prove the quality of a generated answer.

Keep two acceptance questions visible

In a hypothetical private inference service, average token throughput meets the target, but a replacement worker cannot establish the required trusted state. Existing workers may continue serving while replacement capacity is unavailable. A throughput result from the original pool cannot decide whether the replacement should receive sensitive work. That decision needs the protection policy and its evidence, with a separate capacity response if the check fails.

A bounded evaluation can record two claims. The performance claim names the workload, load range, latency limits and completed-request count. The protection claim names the permitted configuration and the evidence required before work enters it. Ask what happens when either claim becomes unknown during replacement or upgrade.

If policy forbids processing the target data without confidential computing, an unprotected benchmark must use approved synthetic data or another permitted substitute. Treat the permitted comparison as an explicit test limitation. Do not disable a required control on real customer material merely to complete a symmetrical chart.

Accept only the workload and protection claims the evidence actually establishes; keep the remaining assumptions visible to their owners. A small pilot can begin with the supported configuration and a representative request mix. Keep the observed results, rejected cases and recovery limitations together. NVIDIA’s result makes that evaluation worth considering; it does not make the local acceptance decision on the service owner’s behalf.