Network topology levels
Draw and test the physical route from one source GPU allocation through links, switches, and a network interface to one destination GPU allocation
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
A two-node training job reduces a 1 GiB gradient tensor across sixteen GPUs, eight GPUs in each server. A benchmark reports 310 GiB/s for one pair of GPUs inside a server but only 21 GiB/s for one particular cross-node rank pair. The identical GPU model names do not explain the difference. The physical route and shared resources are hypotheses, but clock policy, CPU and network-interface placement, software configuration, and concurrent traffic can also differ. A route claim requires a topology map plus a controlled measurement that changes or isolates the proposed edge.
Run `nvidia-smi topo -m` or the platform's supported topology tool on each server, then run a peer-bandwidth sweep for local GPU pairs and a network test for GPU-to-GPU pairs across the two servers. The topology table names links between GPU, CPU, and network-interface endpoints. The benchmark supplies completed payload bytes divided by elapsed seconds for each tested direction. The two artifacts connect a visible rate difference to a possible physical path.
A communication route begins in memory attached to the source GPU and ends in memory attached to the destination GPU. Depending on the machine, bytes can pass through NVIDIA NVLink, an NVIDIA NVSwitch, one or more Peripheral Component Interconnect Express switches, a CPU root complex, a network interface card, and fabric switches. Direct memory access engines move the payload while software queues and protocols coordinate the transfer.
Engineers call the arrangement of endpoints and connections a topology. Rank numbers identify processes; rank numbers do not encode that topology. A useful topology drawing therefore follows the payload through named endpoints and marks every shared link. The drawing predicts which rank placements should be fast, and pairwise measurements test the prediction before a complete collective obscures the route.
Trace the intra-node GPU path
Begin with source GPU memory and destination GPU memory. A direct GPU fabric can connect the devices, or the transfer can traverse a device switch. Systems without an applicable direct path can use PCIe peer-to-peer routing through one or more PCIe switches and a root complex. If peer access is unavailable, a software stack can stage through host memory, which adds copies and host-memory traffic.
Record direction as well as pair. A route can have different read and write behavior, and several GPU pairs can share the same switch uplink. A pairwise bandwidth matrix exposes these relationships more clearly than one average intra-node number.
Trace the GPU-to-NIC path
For a common RDMA inter-node path, the local route continues from GPU memory to a NIC. Under GPUDirect RDMA, the NIC can access mapped GPU memory through PCIe peer transactions, avoiding payload staging through a CPU buffer. The transfer still requires memory registration or cached mappings, NIC queue work, PCIe transactions, and protocol processing.
NVIDIA's GPUDirect RDMA documentation identifies shared upstream PCIe topology as an important condition and warns that routes crossing CPU interconnects can perform poorly or fail on some platforms. Verify the actual platform instead of assuming that a visible GPU and NIC imply a valid direct path.
The NIC path is common but not universal. Some systems extend an accelerator fabric across chassis or nodes, and collective libraries can select that fabric instead of an RDMA NIC path. Treat node, chassis, and rack as placement labels until topology discovery and route measurements identify the data path actually used.
Trace the inter-node fabric
After the NIC, InfiniBand or RoCE packets traverse one or more switches. A leaf switch can connect servers within a rack or group. Traffic to another group can cross spine links or additional tiers. Routing selects one or several paths, and rail configuration selects NIC ports or independent fabrics.
The useful rate depends on more than endpoint link speed. Packet headers, link encoding, congestion control, switch buffering, path selection, and concurrent flows affect delivered payload bandwidth and tail latency. Shared jobs can change this behavior over time.
Calculate shared cuts and bisection limits
A cut divides the topology into two sets of ranks. Bisection bandwidth is the aggregate capacity that crosses a balanced cut. If sixteen simultaneous flows each have a fast endpoint link but all cross four shared uplinks, those uplinks limit the aggregate transfer.
Collectives create structured simultaneous flows. An all-to-all can demand traffic between every pair and stress a cut more strongly than a ring using a limited set of neighbors. Draw concurrent routes and sum demand on each shared edge instead of multiplying one pairwise benchmark by the rank count.
Include CPU and NUMA placement
The host thread that launches GPU and network work runs on a CPU core. Its memory allocation, NIC queues, interrupt handling, and control traffic can belong to a NUMA node. Binding a rank to a remote CPU socket can add host interconnect traffic and increase launch or networking latency.
Record process CPU affinity, local GPU, selected NIC, and memory policy together. Rank 0 mapped to device 0 is not evidence of locality. Logical numbering often reflects enumeration order rather than the shortest hardware path.
Validate the map in increasing scope
Use topology tools and library logs to obtain the discovered graph. Then measure GPU peer pairs in both directions and at several message sizes. Measure GPU-to-network-interface or RDMA paths when suitable tools are available. Pairwise measurements identify missing peer access, asymmetry, and unintended cross-socket routing.
Finally run the actual collective with representative group membership and concurrency. A pairwise test does not reproduce reduction work, channel selection, or simultaneous flows. Compare the measured collective with the path map and retain both as the baseline for later diagnosis.
Worked example: Diagnosing rank placement
A server has two CPU sockets. GPUs 0–3 and NIC 0 attach near socket 0. GPUs 4–7 and NIC 1 attach near socket 1. Single-node collectives perform as expected, but two-node all-reduce achieves only half the predicted network bandwidth.
- Create a table for each rank containing host, CPU affinity, CUDA device, NVML device, selected NIC, PCIe root complex, and collective group. The table shows that ranks on GPUs 4–7 send through NIC 0.
- Draw the unintended route: GPU 6 → local PCIe hierarchy → socket-to-socket interconnect → NIC 0 → fabric. The payload crosses a shared CPU link before it reaches the network.
- Measure GPU-to-NIC throughput for local and remote pairs at small and large messages. Large remote-pair throughput is substantially lower, while NIC-to-fabric tests remain healthy.
- Bind ranks on GPUs 0–3 to socket 0 and NIC 0, and ranks on GPUs 4–7 to socket 1 and NIC 1. Apply the same rail mapping on the second node.
- Repeat pairwise and all-reduce sweeps. Confirm that the large-message improvement appears after the placement change and that startup-dominated small messages remain within expected variance.
- Run the training step because concurrent GPU kernels and multiple process groups can contend differently from isolated collectives. Verify lower exposed communication and unchanged workload semantics.
Conclusion: The placement change removes the cross-socket edge from each affected route. The conclusion is supported by the route table, pairwise measurements, collective curve, and application timeline.
Common errors
- Assuming identical GPU models imply identical communication. Attachment, direction, route, and sharing determine the path.
- Treating GPUDirect RDMA as a path with no CPU or software involvement. Direct GPU-memory access avoids host payload staging but still requires setup, queues, protocol, and supported PCIe mappings.
- Using endpoint link rate as collective payload bandwidth. Protocol and shared cuts reduce the useful rate.
- Testing only one pair at a time. Concurrent collective traffic can expose oversubscription that isolated paths do not.
Reference summary
A rank identifier does not describe a communication route. Two ranks in one process group can exchange data through a direct GPU link, an intra-node switch, several PCIe switches, a CPU interconnect, a network interface, and multiple fabric switches. Identify every link, switch, memory endpoint, and network interface on the route, plus every other transfer that can share those resources.
- Intra-node GPU fabric: NVLink and NVSwitch can provide higher GPU-to-GPU bandwidth than a PCIe path, but bandwidth and route still depend on system generation and attachment.
- PCIe hierarchy: GPUs and NICs attach through switches and root complexes. A path that crosses a CPU interconnect can be slower or unsupported for peer access.
- Inter-node fabric: InfiniBand, RoCE, other network transports, or a multi-node accelerator fabric can connect hosts or accelerator domains. Trace the actual ports, switch tiers, routes, and shared cuts instead of inferring the path from the node boundary.
- GPUDirect RDMA: a capable peer device can transfer directly to or from GPU memory through PCIe mappings. This avoids staging the payload through host memory but does not remove registration, protocol, or network costs.
- Placement: process CPU affinity, local GPU, NIC selection, rail selection, and rank group membership determine whether the intended route is actually used.
Knowledge check
Why can assigning rank 0 to GPU 0 be poor placement even when every GPU is identical?
Show the answer guide
- Rank 0 and GPU 0 are logical enumeration choices; neither identifier states the GPU's PCIe root, CPU socket, nearest network interface, NVLink neighborhood, or rail.
- A poor mapping can send GPU traffic across a CPU interconnect or oversubscribed switch while a different GPU-to-NIC pair has a shorter supported peer route.
- A defensible placement uses a topology map, CPU affinity and NIC records, directional pair tests, and a collective measurement; identical endpoint models do not prove identical paths.
Map and test one communication route
Use the platform topology tool to choose one predicted-fast and one predicted-slow directed GPU pair or GPU-to-NIC path. Draw every named endpoint and shared edge before running pairwise tests.
Evidence to keep: Topology output, route diagrams, CPU/GPU/NIC placement table, raw directional bandwidth and latency samples, and a statement of whether the measurements support each route hypothesis.
Once the route is known, a latency–bandwidth model estimates how message size and communication rounds translate into time.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.