NVIDIA Topograph puts GPU placement evidence in focus

Four available GPUs do not necessarily form one useful place to run a tightly coupled job. Their connections matter, and the scheduler needs evidence of those connections before it can make an informed choice.

NVIDIA published new Topograph deployment guidance on September 22. It explains how the existing open-source toolkit feeds physical topology into Kubernetes and Slurm workflows. This is a technical-guidance update, not evidence that Topograph first became available today.

A topology view should be treated as decision evidence with a freshness boundary, not as a permanent description of the cluster. That is the monitoring-freshness principle applied to a consequential consumer: the component choosing where AI work will run.

From physical fabric to scheduler input

The upstream project separates providers, which discover topology, from engines, which publish it. Outputs include Kubernetes node labels and Slurm configuration. Fabric locality and accelerator domains are distinct inputs; a scheduler must consume the relevant output for placement to change.

The practical handoff has several places to inspect. Did discovery see the correct nodes? Did publication preserve their relationships? Does the selected scheduling policy refer to those labels? Finally, did a newly placed workload land where that policy intended? A successful installation settles none of these questions by itself.

A healthy process can carry old evidence

NVIDIA’s guide says Topograph reflects reported topology and refreshes it when generation runs. Which fabric changes become visible depends on the provider and triggering events. Its Kubernetes example requires a supported provider, Kubernetes 1.27 or newer and Helm 3.10+ or 4.x. Check those prerequisites against the version you deploy.

Consider a hypothetical node replacement during a busy inference period: the replacement joins the cluster while its old locality label remains in an operator’s saved inventory. A placement decision made from that inventory could be internally consistent and still wrong for the replacement. The useful test follows the change through discovery and publication, then inspects a newly scheduled test workload. Restarting the discovery service without checking the result would answer a different question.

Record when the source observation was obtained and when the scheduler-visible output was verified, wherever the selected integration exposes that evidence. If freshness cannot be established, retain that uncertainty. An owner should decide whether new tightly coupled jobs wait, use a less restrictive policy or move to a verified pool; the response depends on the workload commitment.

Locality is a policy choice with a cost

The guide distinguishes ordinary preferred Pod affinity from topology-aware gang placement. A preference influences individual placements; it does not establish that all members of a distributed job receive a coordinated allocation. Select the mechanism that matches the job before interpreting a good-looking node map as scheduling success.

For independent jobs that barely communicate, tighter locality may offer little benefit while restricting the capacity available to start work. Conversely, a strict placement boundary can leave a coupled job waiting for enough resources in one domain. Compare queue delay and completed work as well as communication behavior. Do not infer a performance improvement solely from more compact placement.

After a representative cluster change, verify the published topology, the scheduler’s use of it and the resulting placement before expanding the policy. Start with one supported environment and one representative workload. Keep its previous scheduling configuration available, capture the topology evidence with the test result, and agree who can relax or restore the policy if waiting work misses its commitment.

For the surrounding capacity discussion, see GPU capacity, power and cooling. Topograph addresses a different part of that problem: whether placement uses an accurate account of the hardware relationships available now.