NVIDIA’s Cluster Readiness Engine tests GPU workloads before admission

Before a distributed job arrives, a cluster operator needs to know whether its intended nodes and communication paths can support the workload. NVIDIA’s September 23 introduction of Cluster Readiness Engine describes an open-source Kubernetes controller that runs tests on selected node groups. Its results can inform that handoff, provided the tested population and workload match what the incoming job will use.

The controller represents the request as a Certification, divides it into category Workflows and execution Jobs, and reports results by node and category. NVIDIA describes communication, hardware-diagnostic and training tests, plus adaptive splitting of failed groups to narrow suspects. Kubernetes 1.29 or later and NVIDIA GPU Operator are required, with further requirements set by hardware and selected tests.

The incoming workload supplies the readiness criteria

Readiness depends on what the incoming job needs from the tested nodes, including the communication paths it will use. The risk-verification guidance makes that distinction explicit. A completed job becomes admission evidence only to the extent that its conditions match what the next workload needs.

NVIDIA’s engine evaluates user-defined thresholds through Common Expression Language and supplies no default thresholds. It also distinguishes running successfully from meeting the target. Those choices require the cluster owner to set criteria appropriate to the workload and hardware; a copied example threshold cannot make the acceptance decision on their behalf.

The node set is part of the result

Consider a hypothetical handoff in which communication passes within one rack, but the incoming job spans two. The tested path says nothing yet about communication between racks. Keep that requirement open and validate the relevant topology before admission rather than extending an accurate local pass into an unsupported broader claim.

Carry nodes, workload, thresholds, measurements and configuration versions together in the handoff. Include nodes removed after failures. Otherwise a successful retest on a smaller set can be mistaken for acceptance of the original offered cluster.

Testing consumes the capacity under examination, so reserve a window, set a load ceiling and name the person who can stop the run. Continue normal health monitoring afterward: the test records a condition at a particular time. This is an account of documented capability and a proposed evaluation method, not an independent cluster benchmark.

For the two-rack job, the handoff should make the untested between-rack path visible beside the within-rack result. The receiving operator can then request that missing validation without discarding the useful local evidence or admitting the larger job on its strength alone. Keeping the tested nodes and conditions attached to the measurements makes that distinction survive the handoff.