NVIDIA tests GPU cluster readiness with real workloads

A healthy inventory of GPUs does not establish that a distributed workload will run well across them. NVIDIA's September 23 introduction of Cluster Readiness Engine addresses that gap with an open-source Kubernetes controller that executes workloads across selected node groups.

The controller organizes a requested Certification into category Workflows and execution Jobs. Results return by node and test category. NVIDIA describes communication, hardware-diagnostic and training tests, plus adaptive splitting of failing groups to narrow the suspect population. The project is available as open source; its setup prerequisites include Kubernetes 1.29 or later and NVIDIA GPU Operator, with additional requirements for particular hardware and tests.

A completed workload is only one part of readiness

Write the readiness claim before choosing the test, and keep successful execution separate from acceptance of the risk. That applies the distinction in our risk-verification guidance to a cluster handoff. The useful question is which workload can now be admitted under which conditions.

NVIDIA says the engine evaluates user-defined metric thresholds with Common Expression Language and ships no default thresholds. It records a missed target separately from whether the workload ran successfully. Copying an example threshold from different hardware would therefore substitute somebody else's acceptance decision for the one the cluster owner needs to make.

Carry the tested scope into the handoff

In a hypothetical cluster handoff, a communication test completes on one rack while the incoming job will span two racks. The result may support that rack's tested path, but it has not exercised the inter-rack boundary. If the planned workload or topology lies outside the test, keep that part of the readiness claim open and run a relevant validation before admission.

Retain the tested node group, workload, thresholds and measured result with the capacity decision. Record configuration versions and any nodes removed after a failure. Otherwise, a later operator may inherit a pass label without knowing that it describes a smaller or differently configured cluster.

Workload-driven validation consumes the capacity it exercises. Run an initial evaluation in a reserved window with a named stop owner and a defined load ceiling. It complements continuing health monitoring rather than replacing it: a readiness result describes an observed condition, not a promise that hardware will remain healthy. This article reports NVIDIA's documented capability and proposes an acceptance approach; it does not report an independent cluster benchmark.