Opens in a new tab
News

Four free GPUs still need a good route between them

In brief

NVIDIA Topograph gives schedulers a map of hardware connections. The useful result is a job that finishes sooner, including the time spent waiting for a better group.

4 min read

Is this the right starting point?
Before you read
Basic Kubernetes or Slurm scheduling and distributed-workload familiarity; topology and locality terms are explained.
What you will take away
Trace topology evidence into placement and compare communication gains against queue delay and completed-job outcomes.
Sources
A paper map has a dashed circular outline beside four navy nodes connected by cords.
On this page3 sections

Imagine choosing four free GPUs, the chips doing an AI job's calculations. The job starts, but the chips keep waiting for results from one another. All four were available. Their connections were the part the availability count did not tell you. NVIDIA Topograph helps scheduling software see that missing detail: how the hardware is connected, or its topology.

The open-source toolkit supplies that map to Kubernetes and Slurm, systems that assign work to computing resources. NVIDIA's September 22 deployment guide walks through an existing project rather than announcing a new one. For the person waiting on the job, the question is pleasantly concrete: will a better-connected group finish sooner, even if it takes longer to get one?

That answer depends on a map of today's machines. Replace a node, one machine in the cluster, and yesterday's connection labels may no longer fit. A discovery process can look healthy while the scheduler is still choosing from an outdated picture.

Give the scheduler the connections it cannot count

In the project, one part discovers the connections and another puts them into a form the scheduler can read. Those parts are called providers and engines. A provider finds the relationships; an engine publishes them as, for example, Kubernetes node labels or Slurm configuration. The label is information for a placement decision, not the decision itself.

Suppose our chips repeatedly pass intermediate results between machines. Their messages travel across the connecting network, often called the fabric. A locality domain groups resources by a connection relationship. Fabric locality and accelerator domains describe distinct relationships, so the scheduler needs the one relevant to this job, rather than treating every kind of closeness as interchangeable.

A tightly coupled job can leave several chips waiting for the slowest exchange. Shortening that path may help all of them get back to work. Four independent jobs have a different bargain: with little to exchange, they may finish sooner on available devices now than on preferred devices later.

You can follow the decision without guessing from a running process. Start with the nodes the provider observed, find their relationships in the published output, then inspect the resources a new test job actually receives. If the allocation does not respond to the map, that chain shows where to investigate next. Discovery being alive cannot answer the placement question on its own.

Topology evidence: Observe fabric, Publish topology, then Verify placement.
Conceptual path from discovered connections to a new test job: the provider reports relationships, the engine publishes labels or scheduler configuration, and the resulting placement is inspected.

The new machine arrives before the map catches up

Topograph refreshes its reported topology when generation runs, according to NVIDIA. That is a discovery-and-publication step, not a promise that every physical change will be noticed immediately. What becomes visible depends on the provider and its triggers. The guide's Kubernetes example calls for a supported provider, Kubernetes 1.27 or newer and Helm 3.10+ or 4.x. Those requirements need to match the version being deployed.

Take a hypothetical node replacement while inference jobs, which use a model to produce answers, are waiting. The replacement joins, but the saved inventory still carries the previous locality label. The scheduler follows its instructions faithfully and makes a poor choice anyway. Following the changed relationship into the published map and a newly scheduled test job tells you more than restarting discovery and watching it come back up.

Where the system exposes them, keep the times of the source observation and the scheduler-output check. If the map's freshness cannot be established, the job's owner has a real choice to make: wait, use a checked pool, or relax the locality requirement. The promised completion time matters alongside the quality of the placement.

Sometimes the free chip now is the better choice

The guide draws another useful distinction. Preferred Pod affinity gives Kubernetes a preference when placing individual units of work, called Pods. Topology-aware gang placement coordinates resources for the members of a distributed job together. Several individually favorable labels do not add up to that coordinated allocation by themselves.

Insist on a particular group and some free resources may sit unused while the missing members become available. Independent work can suffer, and even a tightly coupled job can wait too long. Compare its time in the queue, its communication behavior and its completion time. A neatly grouped allocation is interesting; the person waiting still needs the finished job.

For a proposed comparison, choose a representative job and keep the earlier policy recoverable. Retain the map, the actual allocation and the completion time together. Before waiting threatens the commitment, agree who may relax or restore the policy. That leaves a usable decision when the preferred group is slow to become free.

The power and cooling guide asks how much capacity the site can support. Topograph helps describe the relationships within that capacity. Back at our four GPUs, the answer is not simply that they were close together. It is which four were chosen, whether their map was current, and whether the time they saved talking to one another paid back the time spent waiting to begin.

Sources & context

Sources linked in this article. Read alongside the author’s analysis; a citation does not independently verify a publisher’s claims.

Report an error or outdated detail

More on NVIDIA

All NVIDIA coverage

Keep reading

Where to go next