NVIDIA Topograph maps GPU connections for workload placement

A scheduler can find four free GPUs and still make a disappointing choice for a job that needs those GPUs to exchange data frequently. Availability says which devices are idle. Topology says how the devices are related, including the connections the job will depend on. NVIDIA’s Topograph guidance is about getting that second kind of information into the placement decision.

The September 22 deployment guide explains how the existing open-source toolkit supplies physical topology to Kubernetes and Slurm workflows. This is updated technical guidance, not an announcement that Topograph first appeared that day. For a workload owner, the interesting question is whether knowing more about the fabric produces a better completed job after the wait for suitable hardware is included.

For that comparison to be useful, the map has to describe the cluster the scheduler can use now. The scheduler needs the current GPU relationships, including any changed by a node replacement. A stale locality label can direct a new job according to connections that no longer exist, even when the discovery service itself is running.

How a physical relationship becomes a placement rule

The upstream project separates providers that discover topology from engines that publish it. The outputs include Kubernetes node labels and Slurm configuration. Fabric locality and accelerator domains are distinct inputs, so the scheduler has to consume the output relevant to the relationship the workload needs.

Following one job through this chain is more revealing than checking that Topograph is installed. The provider has to see the intended nodes; publication has to preserve the relationships; scheduling policy has to refer to the resulting labels or configuration; and a new allocation has to follow that policy. An error in any connection can leave the discovery system working while placement remains unchanged.

A replacement node changes the scheduler’s map

NVIDIA says Topograph reflects reported topology and refreshes it when generation runs. Visibility of fabric changes depends on the provider and the events that trigger it. The Kubernetes example requires a supported provider, Kubernetes 1.27 or newer and Helm 3.10+ or 4.x. Those prerequisites need to match the deployed version before a refresh test can tell much about the integration.

Consider a hypothetical node replacement while inference work is queued. The new node joins, but an operator’s saved inventory retains the old locality label. A scheduler using that description can obey its rule exactly and still place the job poorly. Restarting discovery would only show that a process restarted. Following the changed relationship into published output and then into a newly scheduled test workload shows whether the actual decision changed.

Where the integration exposes them, source-observation time and verification time for the scheduler-visible output help establish that chain. If the map cannot be shown to be current, the workload owner has a choice with real consequences. A coupled job might wait for verified information, move to a checked pool or use a less restrictive policy. Which is preferable depends on the cost of waiting and the job’s commitment.

A shorter communication path can mean a longer queue

The policy mechanism matters after freshness is established. The guide distinguishes preferred Pod affinity, which influences individual placements, from topology-aware gang placement, which coordinates an allocation for a distributed job’s members. Favorable labels on several individual nodes do not establish that a workload needing coordinated allocation received it.

For largely independent jobs, tighter locality may offer little benefit while excluding otherwise usable capacity. Even a tightly coupled job can wait longer when a strict domain boundary requires enough resources to become free together. Queue delay and completed work therefore belong beside communication behavior in the comparison. Compact placement is an input to the result, not a performance result on its own.

A representative workload in one supported environment makes a manageable pilot. Save the topology evidence with its observed placement and completion time, and retain the previous scheduling configuration. Agree who can relax or restore the policy if additional waiting threatens the commitment. This keeps the trial focused on the workload’s outcome rather than on whether the new labels look tidy.

The discussion of GPU capacity, power and cooling covers the physical resources available to the cluster. Topograph addresses how their relationships inform placement. Returning to the four free GPUs, a useful result now explains which four were chosen, whether that choice reflected the current cluster, and whether the communication benefit repaid the extra wait. Those are the grounds for keeping the policy or choosing a different one.