NVIDIA’s Cluster Readiness Engine tests GPU workloads before admission

The operator receives “cluster passed,” then discovers that tomorrow's job crosses a network path the test never used. NVIDIA's September 23 Cluster Readiness Engine announcement offers a way to test selected node groups before work arrives. Its result is useful when the tested nodes and paths match what that work needs.

The operator can turn that missing cross-rack check into a request to the open-source Kubernetes controller. The request is called a Certification. The engine organizes the chosen test categories into Workflows and runs them as Jobs, returning results that identify the nodes and categories. These names describe the route from a requested check to work actually running, not three different meanings of “passed.” NVIDIA describes communication, hardware-diagnostic and training tests, including dividing failed groups into smaller groups to narrow the search.

This is for Kubernetes GPU clusters, not a desktop GPU check. Documented prerequisites include Kubernetes 1.29 or later and NVIDIA GPU Operator, with further requirements depending on hardware and tests. After those are met, decide what the incoming workload actually needs demonstrated.

The incoming workload supplies the readiness criteria

For tomorrow’s two-rack job, a test confined to one rack leaves precisely the connection the operator is worried about unexamined. The selected nodes and communication paths need to include that connection. Our risk-verification guidance makes the scope matter: a completed test supports admission only for the conditions it establishes.

In this hypothetical example, suppose the owner needs a selected communication measurement to reach 10 GB/s, but the test reports 8 GB/s. The Job can finish successfully while the measured result misses the target. The engine expresses user-defined conditions on measurements in Common Expression Language. It supplies no default thresholds: these invented numbers demonstrate the distinction, not a recommended speed or an engine benchmark. The owner must choose criteria suited to the actual workload, test and hardware.

The node set is part of the result

Imagine a communication test that passes within one rack, while the new job will span two. Keep the within-rack result and request the missing cross-rack test. The existing pass has not exercised that additional route.

Both tests may be called communication tests, but they ask different things of the network. Keeping the actual node group with the result makes that difference visible before the incoming job finds it.

Now suppose a failed node is removed and the smaller group passes. Tomorrow’s job cannot inherit that pass as if the original group had passed too. The handoff needs the removed nodes alongside the tested nodes, workload, thresholds, measurements and configuration versions. That record explains why the result changed instead of leaving the next operator to guess.

The cross-rack test also uses capacity the incoming job will need. A reserved time, load ceiling and named stop owner let the operator investigate without treating the test itself as free work. Ordinary health monitoring still matters afterward: a passing result describes conditions at that time, not a promise about tomorrow. This article reports documented capability and a proposed evaluation, not an independent cluster benchmark.

The receiving operator now has a useful pass and a specific unanswered question. Testing the second rack can fill that gap without dismissing the first result or stretching it beyond the work it measured.